diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/MESHBAY_DESIGN.md | 2628 | ||||
| -rw-r--r-- | docs/transfers-v1.md | 1159 |
2 files changed, 3787 insertions, 0 deletions
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md new file mode 100644 index 0000000..73f6b42 --- /dev/null +++ b/docs/MESHBAY_DESIGN.md @@ -0,0 +1,2628 @@ +# MeshBay — Design + +> **Status: the specification.** This document states what MeshBay is, how it is +> built, and why each part has the shape it has. It replaces the draft series +> (`meshbay-draft-v5.md`, `meshbay-draft-v6.md`, `old-draft.md`) and the design +> notes that grew around it; §16 maps every reference those documents and the +> code make onto a section here. +> +> **The convention that is not negotiable: a claim in this document names the +> adversary it holds against.** A property that holds against a passive hub and +> not an active one is written that way. A section that says "this buys nothing" +> is as load-bearing as one that says it buys something. +> +> **The second convention: this document states design, not history.** Where a +> label like `C1`, `NS6` or `T3` appears — and hundreds of code comments cite +> them — it names the invariant that holds today, not the incident that produced +> it. §13 is the register of those labels. +> +> Wire versions at the time of writing: **MNP 3.0** (oldest peer accepted 3.0), +> **MHP 0.1**, packages **0.13.0**. The normative source for the wire format is +> `MESHBAY_NODE_PROTOCOL.md`; this document states the design the protocol +> serves, not its byte layout. + +--- + +## 0. How to read this + +### 0.1 Scope + +| Read | For | +|---|---| +| **this document** | the architecture, the trust model, and the reason each decision is what it is | +| `MESHBAY_NODE_PROTOCOL.md` | the MNP wire format, message by message | +| `transfers-v1.md` | the transfer system's failure-mode analysis, kept because a synthesis cannot carry "every way a slot can be lost" | +| `playlists.md` | the playlist design in full — decided, not built (§9.10) | +| `USERGUIDE.md` | how to use and operate a deployment | +| `PACKAGING-GUIDE.md`, `QUICKSTART.md`, `HTTPS.md`, `MAIL-SERVER.md`, `windows-build.md` | installation and server operations | +| `CLAUDE.md` | project conventions and the engineering lessons that govern how changes are made | +| `old-draft.md` | historical archive of drafts v1–v4. Nothing there is authoritative | + +### 0.2 Reference labels + +Code comments, tests and commit messages cite short labels — `C1`, `H3`, `NS6`, +`T3`, `C5b`, `W2`, `E9`, `F1`. **Every one of them is defined in §13**, stated as +the rule it names today. + +Three label families collide, and the register keeps them apart: + +- **`C*` / `H*` / `M*` / `L*` with no qualifier mean the second review's numbering** + (§13.3). That is the numbering the code uses. The first review's `C1`/`C2` and + the third review's `H1`/`H2`/`M1`–`M6`/`L1`–`L11` are always written with the + review named (§13.1, §13.4). +- `M2a`, `M2b`, `M2c` and `C5a`, `C5b` are sub-items and are unique across the set. +- **`D1`–`D4` are the client-architecture decisions** (§14.2); **`D1`–`D12` written + as "Stage D*" are desktop-client build stages** (§15.2). + +### 0.3 The one-sentence version + +> **The hub cannot read your content unless it ships you malicious client code — +> and against a native client it cannot do that undetectably.** + +Everything below either supports that sentence or states precisely where it stops. + +--- + +## 1. What MeshBay is + +### 1.1 Positioning + +MeshBay is a platform for **private, encrypted, self-hosted groups with an +application store**. A group is a set of people, a set of directories on +somebody's machine, and a set of applications over them — chat, a file explorer, +a video library, a music player, a photo album. + +It is **not** a public file-sharing network. Public groups exist as an optional +hub feature and are switched off on the reference deployment (§7.4). The design +optimises for the private case throughout: the hub keeps no file names for +private groups, registers no content hashes for them, and holds no key that +opens them. + +### 1.2 The three parties + +``` + ┌─────────┐ accounts, group registry, signaling relay, ┌─────────┐ + │ hub │ notifications, moderation, instance policy │ peer │ + └────┬────┘ — never content, never a group key │ hubs │ + │ └─────────┘ + MHP 0.1 │ signalling (SDP/ICE, <1 KB), presence, revocation push MHP + │ + ┌────┴────┐ MNP 3.0 ┌──────────┐ + │ node │◄──────── WebRTC DataChannel / QUIC ──────────►│ client │ + └─────────┘ index, file chunks, streams, chat, admin └──────────┘ + holds the files browser SPA or desktop + holds the group key client; both are the + is the content authority same UI source +``` + +- **The hub** is a registrar and a signaling relay. It is in the trusted path by + choice, not by necessity (§14.2 D4), and it is never in the data path. +- **The node** is the daemon that holds a group's directories and its group key. + It is the **sole content authority**: it decides who is served, what is served, + and who may change anything about the group's content. +- **The client** is the browser SPA or the desktop application. Both are built + from one source tree (§8.4). + +### 1.3 The rule that settles arguments + +> **Group-related server state lives on the node. Always.** + +Files, indexes, members' devices, pending device requests, invitations, chat, +chat epoch keys, per-root availability, application settings, transfer limits — +all on the node. The hub holds accounts, the group registry and membership, +signaling, notifications, the moderation surface and instance policy, and nothing +else about content. + +This is decision **E9**, and it is the rule any new feature is measured against. +A feature that wants a row on the hub about a group's content is a feature that +has misunderstood the model. It has been re-verified at each content-model change: +`SwarmSource` carries a content hash, a node id and an endpoint — **no paths, no +filenames** — and private groups register nothing at all (**H7**). + +--- + +## 2. Trust model + +### 2.1 Adversaries + +Every claim below is written against one of these, and they are the only ones the +document uses: + +| Adversary | What they can do | +|---|---| +| **Passive hub** | Read everything the hub legitimately stores and relays | +| **Active hub** | Also lie: forge tokens, invent accounts, substitute values it publishes, ship modified client code to a browser | +| **Malicious node operator** | Read and alter everything on their own machine, including the plaintext files they host | +| **Malicious group member** | Everything a member may do, plus anything the protocol fails to refuse | +| **Network attacker** | Observe and tamper with traffic between any two parties | +| **Local attacker** | Reach loopback services and files on a client or node machine | +| **Registered hub user with no membership** | Reach every hub endpoint that does not check membership | +| **Federated peer hub** | Push directory rows and revocations over MHP | + +### 2.2 Security claims + +| Claim | Passive hub | Active hub | Malicious node operator | Malicious member | Network attacker | +|---|---|---|---|---|---| +| Data never transits the hub | ✅ | ✅ | — | — | ✅ | +| File content is unreadable | ✅ | ❌ **T3** (browser) · ✅ native | ❌ by design — the operator hosts the files | ❌ members share the group key | ✅ | +| The file index is unreadable | ✅ | ❌ T3 · ✅ native | ❌ | ❌ | ✅ | +| Chat content is unreadable | ✅ | ❌ T3 · ✅ native | ❌ — the operator is a member | ❌ | ✅ | +| Chat is unreadable **off a stolen disk** | ✅ | ✅ | ✅ without the keystore passphrase | ✅ | ✅ | +| Content cannot be modified | ✅ | ✅ | ❌ by design | ✅ | ✅ | +| The node cannot be impersonated | ✅ | ✅ | — | ✅ | ✅ | +| Client code integrity | ❌ **T3, accepted** (browser) · ✅ ships in the package (native) | ❌ T3 · ⚠️ native: **detectable, not prevented** | ✅ | ✅ | ✅ | +| The hub cannot obtain the group key | ✅ | ✅ **except** in an open-join group, where it can join legitimately (§7.3) | — | — | ✅ | +| Node content authority | ✅ | ✅ | ✅ sovereign | ✅ | ✅ | +| Devices cannot be added by the hub | ✅ | ✅ — the hub holds no user key and cannot countersign | ⚠️ a node adds a device only to itself, where it already reads everything | ✅ | ✅ | +| Chat senders are authenticated to each other | ✅ | ✅ | ⚠️ only for accounts the reader has already seen (§3.3) | ✅ | ✅ | +| Keypair bundles (**C4**) | closed for native devices | closed for native devices | ⚠️ **open for any account that also signs in from a browser** | — | — | +| Deleting your account erases you | ✅ hub-side | ✅ hub-side | ❌ files, pinned identity and bundle stay on the node (§7.7) | — | — | +| Your identity keys stay yours | ✅ | ✅ | ⚠️ offline attack on the bundle they hold — succeeds against a weak passphrase, and yields the identity used **on that node only** | ✅ | ✅ | + +### 2.3 What the project must not claim + +Three sentences are forbidden, each for a deliberate reason: + +- **"Everything is encrypted and unreadable by other parties, even the hub."** + A hub that ships the code can lift keys from the page regardless of protocol + design (**T3**). That is an artifact-level attack, not a silent directory lie, + and it is removed for native clients — not for browsers. +- **"A native client makes the hub untrusted."** It converts an undetectable, + per-request, per-user attack into a persistent artifact that can be hashed and + compared. That value is realised by reproducible builds and published hashes, + not by the packaging format. A build signed with a key the hub operator holds + *relocates* trust; it does not remove it. +- **"C4 is closed."** It is closed for a native device unconditionally, and open + for any account that also uses a browser. **An account is only as strong as its + weakest client.** + +"End-to-end" here describes **client ↔ node**, never client ↔ client. Members and +the operator read everything in their group; that is what a group is. + +### 2.4 One boundary worth naming + +An operator hosts your content by design. They should not be able to become +*you*. They can still try — a keypair bundle sits on their disk and a weak +passphrase gives it up — but what it gives up is **the identity you use with +them**, which unlocks nothing they did not already hold. Reading what they host +is by design; reading what *other* operators host is not, and does not follow +(§3.2). + +--- + +## 3. Identity + +### 3.1 Accounts + +An account lives on the hub: a username, an encrypted email address, a status and +a role. The passphrase never leaves the client. It derives **two independent +values**, both salted by the trimmed username: + +| Value | Derivation | Consumer | +|---|---|---| +| `auth_key` | PBKDF2-SHA512, 600 000 iterations, domain `meshbay:auth:v1:<user>` | hub authentication — the hub stores an Argon2id hash of it | +| `bundle_key` | Argon2id 128 MB / t=3 / p=1, domain `meshbay:bundle:v2:<user>` | AES-GCM key for the per-node identity bundle, held on each node | + +This split is **T1**, and its consequence is structural: **the hub never sees a +passphrase**, so the passphrase floor — 12 characters and roughly 60 estimated +bits — can only be enforced client-side, and is. + +The two values have different fates. The hub holds a verifier for `auth_key` and +can reset it from an email code. **Nobody can reset `bundle_key`**: the hub has +held no key material since the invite redesign, and cannot reach a +`keypair_bundles` row, which is served only over MNP to authenticated members. +That asymmetry is why passphrase change and passphrase recovery are two features +and not one (§3.6). + +### 3.2 Identity keys are per node + +A person's identity keypair is created **at first contact with a node**, +encrypted under `bundle_key`, and left on that node. It is never reused +elsewhere. + +The reason is blast radius. The adversary is concrete: an operator holding their +own node's disk, attacking a bundle offline at their leisure. What cracking one +yields is the identity that person uses **on that node** — where the operator +already holds the content, the index and every byte they serve. It is not a key +anywhere else: each node gets its own, and a key one node pinned is a stranger to +the next, which asks for a code like any first contact. + +Two consequences fall out and both are wanted. Two operators cannot tell they +host the same person by comparing keys. And **the hub stores and publishes no +user keys at all** — `users.pk_ed25519` / `pk_x25519` are dropped, `PUT /me/keys` +does not exist, and `/pubkeys` returns an account id and the node's linking key. +There is no directory to substitute from, which is what **H3** was. + +Two further rules follow directly: + +- **Tokens carry no `pk_user` claim.** A key chosen by the hub must never become + the identity a node records for an upload, or whoever issues tokens decides who + may delete a file. Attribution uses the roster pin. +- **There is no key rotation endpoint.** Rotation is per node: `member unpin` plus + a fresh code, which already exists. + +Registration therefore generates nothing, which has a pleasant side effect: a +scripted signup produces a real account, and a wiped hub and node can be taken to +a working demo without a browser. + +### 3.3 Devices + +One person may hold several devices on one node — a browser and a desktop client, +two laptops. `identities` is keyed by `(user_id, pk_ed25519)` with `label`, +`added_at`, `added_by_pk`, `revoked_at` and the evidence columns below. Pinning is +never `INSERT OR REPLACE`: a silent overwrite is a hole the moment a second key is +legitimate. + +**A new device is admitted by a key the node already pinned.** The authority is +therefore a key the node established locally, exactly as for operator +authorisation — and **the hub cannot produce it**, because it holds no user keys. +Device linking adds no hub-reachable authority. + +The binding is a **one-time code the new device generates and displays**, hashed +together with the new keys: `code_hash = sha256(code ‖ new_pk_ed25519 ‖ +new_pk_x25519)`. The approving device asks the node for this account's pending +requests **with their stored hashes** and recomputes the hash for each until one +matches. + +> **The code never reaches the node.** That is what makes a substituted key +> impossible rather than merely detectable: a node offering fabricated keys would +> have to produce a hash over a code it has never seen. + +The approval is deliberately **not** a human comparing digits. Safety numbers were +evaluated and refused, permanently, under decision 19: **no new code exchanges +between people.** A device-linking code is a code between a person's own devices, +which is a different thing and was already accepted. + +Bounds, all of them anti-abuse rather than the security boundary — the security +comes from the hash binding: + +| Bound | Value | +|---|---| +| Request TTL | 1 h, `[node] device_request_ttl_minutes` | +| Devices per account per node | 5 | +| Attempts per connection | 5, then a node-wide lockout | +| Filing a request | requires the account to have at least one pinned identity already | + +`member unpin <user>` removes **every** device. `device revoke` marks one rather +than deleting the row, because a deleted row is a key the node would happily pin +again. + +**Which device is on a connection** is proved separately from which account. The +handshake proves the account; `device_hello` — signed over a transcript naming the +node, the group and this connection's nonce — proves the device, and is refused +unless the key is a live device of that account in the node's own roster. Without +it the node would fall back to the account's oldest key and record it as the +author of everything. + +**Members can verify each other's device keys** (Tier 2). `group_roster_req` / +`group_roster_resp`, sealed under the group key, answers **any member** with, for +each live device of each active member: the key, the key that countersigned it, +and the signature, nonce and timestamp that prove it. The client walks the chain +itself — the node decides nothing, because the node is the party the property +holds against. A device the node lists but cannot evidence never enters the +verified set, so a fabricated key is not laundered in by being mentioned. + +Three rules keep that honest: + +- **A root is a device that names no countersigner**, not one that fails to + produce a signature. Treating "no proof" as "root" would admit anything a node + chose to write. +- **First sight pins everything the node says**, not the verified subset — an + alarm that fires on legitimate second devices stops being read, and the budget + for this whole feature is exactly one notice: *"this account's key changed"*. +- **A device pinned before the evidence columns existed is unevidenced and reads + as such.** + +The property, stated exactly: + +> Once a member's client has seen an account, **a node that later substitutes a +> key for it is detected.** Nothing is gained at first sight, where the client +> has nothing to compare against. + +That second sentence is not a caveat to be dropped. Closing first sight needs an +attestation rooted outside the node — an operator-signed roster (Tier 3), which is +deferred with nothing depending on it, and worth building only where the operator +is not the machine. + +**The cost, which is real:** the roster is member-visible, so every member of a +group learns how many devices every other member holds and what their public keys +are. It stays inside the group, the hub is not involved, and it is scoped to one +group. A member who cannot see the keys cannot check them, so this is not +avoidable. + +**Where device linking does not hold:** an approval performed *in a browser* +inherits **T3** — the hub serves that browser its code and can read the typed +code. The first browser-to-native link is the moment of highest exposure for an +account, and it happens once. An account created natively does that first link in +the safe direction. + +### 3.4 Admission: invitations and pairing codes + +**The node wraps the group key**, for a key the recipient proved possession of, +over an authenticated channel, bound to an identity the operator admitted with a +one-time code the hub never sees. + +``` +operator (SSH) meshbay-node member invite bob → CODE R3H8-TB6V + (or the same from the group's Settings tab, signed by the paired browser) +operator sends the code to bob out of band +bob opens the group; the client holds no group key +bob → node join_request {pk_ed25519, pk_x25519, code, sig} ← pre-proof window +node code valid for this account → pin the identity, admit to the group +node → bob the group key, wrapped for the X25519 key bob just proved he holds +``` + +Four properties, each load-bearing: + +1. **No public key is ever fetched from a directory.** The joiner's keys arrive + from the joiner, both signed together in one transcript (`meshbay:join:v1`), so + the identity key vouches for the encryption key. +2. **The code binds a key to an account**, and the hub never sees it. 40 bits, + Crockford base32 rendered `XXXX-XXXX`, single use, valid for exactly one + account in one group, stored only as `sha256(code)`. A password KDF over 40 + uniformly random bits would buy nothing. Guessing is bounded by 5 attempts per + connection and a node-wide lockout, and every attempt is an audit event. +3. **The node's roster is the authority**, not hub membership. A hub that invents + an account, adds it to a group and mints it a token gets + `not_authorized_for_group`. +4. **Wrapping happens on every connection.** Nothing is stored per member, so key + rotation propagates by itself and revocation actually takes effect. (Rotating + the key after a revocation is still required — the ex-member holds the current + one, and no protocol can take that back.) + +Node authority is established the same way, once per node: `meshbay-node operator +pair` prints a code, the operator types it into their own browser, and the node +pins that identity. **It is never learned from the hub** — a hub able to name the +operator's key could install itself as node administrator, which is **NS4**/**M3**. + +Code lifetimes differ because the acts differ: + +| Code | Default | Setting | +|---|---|---| +| Member invitation | **7 days** | `[node] invite_ttl_hours` | +| Operator pairing | 24 h | `[node] pair_ttl_hours` | +| Device add request | 1 h | `[node] device_request_ttl_minutes` | + +An invitation waits for someone to read their messages; a pairing code is typed +during the SSH session that printed it. + +**Why a code and not something lighter** — the question is what stops the hub from +being bob on his first connection: + +| Option | What an active hub can do | | +|---|---|---| +| Wrap for whatever key the peer presents | Forge a token for bob, present its own key, receive the key | worse than nothing | +| Bind to the key the inviter fetched from the hub | Substitute at invite time | **H3**, relocated | +| TOFU: first connection wins | Race the real bob with a forged token | small window, total consequence | +| Safety-number comparison | Nothing — but it needs two humans reading digits at the worst moment | correct, unusable as a default | +| **One-time pairing code** | **Nothing: the code never reaches the hub** | **adopted** | + +**Delegation is designed and deferred.** `invite_create` is authorised as a *role* +check against the roster rather than an equality test against the operator, and +the `delegate` role value is reserved, so a group admin who does not run the node +becomes a roster row and a CLI command — no protocol change, no migration. + +### 3.5 Open-join groups + +A group whose `join_policy` is `open` pins on first contact (TOFU) and wraps the +key immediately. A code there protects nothing — the hub can create an account, +join through the front door, and be a legitimate member — so it would be pure +friction. + +Stated plainly, per the convention: **in an open-join group the hub can obtain the +group key.** That is a property of open joining, not a defect of this design. +Content in such a group is protected from the network and from non-members, and +from nobody else. + +Note the axis. **`visibility`** (public/private) controls discoverability and swarm +hash registration (**H7**). **`join_policy`** (open/request/invite) controls +admission. Only the second decides whether a code is required: a public group with +`join_policy = "invite"` keeps the code, because being findable is not being open. + +**`join_policy` is read from `node.toml`, never from the hub.** A hub able to +declare a group open would be handed its key. An unknown group reads as `invite`. + +### 3.6 Passphrase change and recovery + +**Changing a known passphrase** re-wraps every reachable node's identity bundle +from the old `bundle_key` to the new one **before** touching the hub — if the +fan-out fails, the account is unchanged. Only then is `POST /v1/users/password` +called with the old and new `auth_key`. Nodes that were unreachable are named to +the user, with the operator fallback (`member unpin` plus a fresh code) as the way +to fix each one. Every refresh-token family is revoked. + +**Recovering a lost passphrase** splits into what each key can reach: + +| | Recovered by | +|---|---| +| Hub login (`auth_key`) | an email code alone | +| Per-node identity keys — group key unwrap, provable upload ownership, chat identity, device countersigning | the **recovery key**, per reachable node | +| An identity on a node with no recovery-wrapped copy, or offline at recovery time | operator `member unpin` plus a fresh code | + +The recovery key is a full-entropy 32-byte secret `R` **generated by the client**, +rendered as a grouped mnemonic. `recovery_key = HKDF-SHA256(R, info = +"meshbay:recovery:v1:" + username)` — HKDF and not Argon2, because `R` has 256 bits +and there is nothing to brute-force. Every time an identity bundle is written to a +node, a **second copy** is written beside it wrapped under `recovery_key` +(`bundle_enc_recovery`, additive on the wire). `R` is a pass-through: offered in +the registration email by default, never written to any database, never logged. + +The reset endpoints are built to leak nothing. `POST /v1/users/password/reset-request` +requires **the username and the email on file as a pair**, checked against a blind +index and never decrypted; a mismatch, an unknown username and a non-active +account all take the identical no-op path and return the same `200 +{"status": "sent_if_exists"}`. So it cannot be used to spray reset mail at an +inbox from a username alone. + +A reset **deletes every `UserDevice` row** on the hub. Three different things are +called "device" here and only one is touched: + +| | What it is | A reset | +|---|---|---| +| `user_devices` (hub) | an Ed25519 key that lets a client skip the passphrase prompt on launch. A hub-login convenience — no group key is wrapped for it | **deleted** | +| per-node identity (`identities` on each node) | the keys that unwrap the group key, prove upload ownership and sign chat — **this is group access** | **recovered** from the recovery copy, or via the operator fallback | +| the roster pin | which identities a node has admitted | untouched | + +Deleting `user_devices` costs one passphrase prompt per client, which is the +point: after a "control may be lost" event, a laptop still carrying a stored +hub-auth key must stop signing in on its own. + +### 3.7 The keypair bundle, and what it is worth (C4) + +A bundle carries **one node's** identity keys, encrypted under the owner's +passphrase, stored on that node. It is what lets a second browser open the same +account there — the ordinary expectation, and the only mechanism available to a +browser, which keeps nothing durable of its own. + +**Why Argon2id.** PBKDF2 is compute-only, which is exactly what a GPU is good at. +Measured: PBKDF2-SHA512 600k costs 241 ms per guess on one core, Argon2id 128 MB +/ t=3 costs 88 ms — the defender pays *less* — but only one of them forces an +attacker to find 128 MB per guess. + +**The honest size of the gain.** On a single high-end card the ceiling moves from +roughly 8k guesses/s to roughly 2k: a factor of four, not a thousand. What it +really buys is the cost of scale — 128 MB per lane caps a 24 GB card near 187 +concurrent guesses and makes custom hardware unattractive, where SHA-512 silicon +is cheap. + +**The passphrase decides this, not the KDF.** At ~2k guesses/s a +dictionary-and-rules run of 10⁹ candidates takes about six days on one card. Four +random words (~52 bits) outlasts the sun. No parameter choice saves a weak +passphrase; it only moves it from hours to days. + +Operational facts that constrain changes: + +- Argon2id runs in **WebAssembly, vendored** under `static/vendor/` with its + provenance. The CSP forbids external hosts and must keep `wasm-unsafe-eval` in + `script-src`. +- **Never change these parameters in one place.** `keyderive.js`, `keyderive.py`, + the desktop client and the test harness are held byte-identical by + `test_bundle_kdf_parity.py`. A mismatch does not look like an error — it looks + like an account nobody can open. +- Bundles carry an `MBK2` marker; an older PBKDF2 form is still readable and is + re-encrypted on the next backup. +- Cost is paid **once per sign-in** (≈650 ms bundle + ≈239 ms `auth_key`). + Reloading a page derives nothing: the key lives in IndexedDB. +- The pre-proof window that serves bundles is bounded (4 fetches) and audited. + +**C4 is reduced, not closed.** Bundles still sit on disks their owner does not +control. It closes for a native device unconditionally, because that device's key +is in no bundle anywhere. It closes for an *account* only when no browser needs a +bundle on that node — which needs `device_policy {allow_bundle: false}`, **signed +by a pinned key** so the decision is the user's and never the hub's (open item +O3). + +--- + +## 4. Cryptography + +### 4.1 Key hierarchy + +``` +User identity key Ed25519 signing, authentication — per node (§3.2) +User exchange key X25519 key agreement — per node +Group encryption key AEAD 256-bit content and index encryption — the group secret +Chat epoch key 32 bytes per group, per epoch — node-generated (§4.5) +Session keys X25519/HKDF per-connection, from DTLS/TLS +``` + +Every private key lives in an encrypted keystore on the machine that owns it. The +hub never sees one. + +**Domain separation is consistent and mandatory.** Every derivation uses a +distinct `info` string, and the AES variant adds an `:aes` suffix so two ciphers +can never derive the same key from one group key. This is a small detail that +prevents cross-protocol key reuse, and it is checked rather than assumed. + +### 4.2 Group key wrapping (ECIES) + +``` +wrap: sk_eph, pk_eph = X25519.generate() # fresh per bundle + shared = X25519(sk_eph, pk_recipient) + wrap_key = HKDF(shared, salt=pk_eph, info="meshbay:gek_wrap:v1", len=32) + wrapped = AEAD(wrap_key).encrypt(nonce, gek, aad=pk_recipient) + bundle = pk_eph ‖ nonce ‖ wrapped + +unwrap: shared = X25519(sk_recipient, pk_eph) # same derivation +``` + +Three properties are why this shape: + +- **The ephemeral keypair is fresh per bundle**, so the same key to the same + recipient produces different ciphertext every time. +- **The AAD binds the bundle to its recipient**, so a bundle reused for a + different member is rejected by the tag rather than by a check somebody has to + remember to write. +- **A wrong private key fails at the AEAD tag** — an immediate, unambiguous + refusal. + +The node produces every copy of the key itself, from its own CSPRNG. **Nothing +arriving over MNP can activate a group key** (**C5b**). Read that precisely: it +targets *key material arriving from outside*, not the instruction. An +operator-signed `gek_rotate` where the node generates the key is a different shape +and is allowed. The initial `gek-init` stays local, because with no key there is +no completed session to carry a signed op. + +### 4.3 On-the-fly encryption + +Files are stored **in plaintext on the operator's disk** and encrypted at read +time. This avoids double storage and makes key rotation feasible without +re-encrypting terabytes. + +``` +disk (plaintext) → compress → per-chunk AEAD under a group-derived key → transport → client +``` + +- Chunk size 1 MB: amortises AEAD overhead and enables seeking, because each chunk + is independently decryptable. +- `chunk_key = HKDF(GEK, salt=None, info="file:" ‖ blake3(file) ‖ ":chunk:" ‖ index)`. + The salt is omitted deliberately: the group key is CSPRNG output and already + uniform, so the file and chunk context belongs in `info`, which is the correct + HKDF usage (**M5**, first review). +- **Chunk authentication is the AEAD tag**, not a per-chunk signature. The tag + authenticates the ciphertext under a key only members hold, which is what the + signature was for. +- Compression precedes encryption, because compression is ineffective on + ciphertext. +- Upload chunk size is 48 KB, which is what fits the SCTP limit after msgpack + overhead. + +Crypto is not the bottleneck: encrypt-and-send costs single-digit milliseconds per +megabyte against tens to hundreds for the network. + +### 4.4 The group envelope + +`meshbay_common/groupbox.py`, mirrored by `sealGroup`/`openGroup` in +`static/crypto.js`, is **one envelope with purpose-separated subkeys** derived +from the group key. AAD is `"<msg_type>|<group_id>"`. + +| Purpose | Info string | Carries | +|---|---|---| +| `index` | `meshbay:index:v1` | `index_sync`, `index_delta` | +| `ack` | `meshbay:ack:v1` | the handshake ack's configuration payload | +| `upload` | `meshbay:upload:v1` | `file_upload` / `file_upload_ack` | +| `chat_keys` | `meshbay:chat_keys:v1` | chat epoch key delivery | +| `roster` | `meshbay:roster:v1` | `group_roster_req` / `resp` | + +**Never reuse the chunk key with a pseudo-file for this.** The purposes are what +keep two message classes from sharing a key. + +Three things about the sealed messages must stay straight: + +- **Sealing the ack line is integrity, not confidentiality.** The handshake + transcript names no ack field, so `is_node_admin`, `enabled_apps`, the roots + table and the rest were authenticated by the channel alone. The AEAD tag comes + from a key the hub does not hold. +- **Sealing the index is defence in depth against our own next bug**, of a class + that has shipped twice (**C1**, **C6**). It buys nothing against an observer, + the hub or a member. That is the whole claim. +- **A payload that does not open ends the session**, never a default. An + unopenable `enabled_apps` reads as "the operator disabled every app" and an + unopenable index as "the group is empty" — both indistinguishable from + legitimate states. + +What stays in clear, and none of it is content: the handshake itself; `type`, `v`, +`group_id` and the ack's `node_pk`/`proof`/`sig`, because a receiver must route and +**authenticate** before it would trust a decryption; `upload_id` and `chunk_index`, +because the node routes and orders on them; `index_progress`, which is counters +only, every 2 s; transfer-lease fields, which are an opaque id and two numbers +(§5.5); and the media-metadata replies. + +### 4.5 Chat encryption + +Chat is encrypted with one key **per group, per epoch, per device**, derived by +name from an epoch key the node generates and delivers wrapped under the group +key: + +``` +epoch_key 32 random bytes, generated BY THE NODE (C5b) +device_key(d) = HKDF(epoch_key, info="meshbay:chat:dev:v1|"+group_id+"|"+d) + where d = base64(device pk_ed25519), the roster's own identifier +``` + +Every member derives every device's key from the epoch key, so **nothing is +distributed per device and nothing is stored per device**. + +**There is no mutable sending state at all.** That is the point, and it is a +stronger guarantee than per-device chains rather than a weaker one: a shared +chain advanced by two clients produces key and nonce reuse (first review **C1**, +one level down), and a design with no sending state cannot have that hazard. + +> **Two clients of one account normally share a device key.** A second browser +> recovers the existing identity from the keypair bundle rather than minting a new +> one; device *linking* is the exception, not the rule. What makes that safe is +> the nonce, not the derivation: **96 random bits, never a counter.** Two +> independent senders under one key collide only on the birthday bound, which at +> chat volume is unreachable; two senders advancing one counter collide +> immediately. + +Each message is **sealed and then signed over the ciphertext** with the device key +the node pinned, so a receiver verifies before decrypting and anyone holding the +roster can verify. The AAD binds the group and the epoch, so a ciphertext cannot +be replayed into another group or attributed to another epoch. `sender_name` lives +**inside** the sealed payload — as a wire field it was free to spoof. + +`sender_id` stays a clear field **set by the node from the authenticated session** +(**NS6**). It is what the store keys on and what the UI groups by; it is not what +authenticates the message. The *device* claim is checked against the connection's +own `device_hello`, or a member could sign as anyone. Replay is refused by a +unique `(device, nonce)` — a replay is a validly signed copy, so nothing about the +signature refuses it. + +**Epochs.** A new epoch is opened when, and only when, the set of devices that may +read *future* messages shrinks: `member revoke`, `member unpin`, `device revoke`, +`gek_rotate`, or an explicit `chat rotate`. Epoch 1 is opened at group load — a +group with no epoch is a group nobody can speak in. + +**Old epochs are kept and still delivered.** That is what keeps history readable +to everyone who could already read it, and it is why rotating the group key is a +**re-wrap** rather than the destruction of the whole archive: the archive is not +encrypted under the group key, only the epoch keys are wrapped with it in transit. +Nothing anywhere deletes an epoch. Epoch keys are stored ECIES-wrapped to the +node's own X25519 key in `bundles.db`, never raw — a plaintext key store beside +`chat.db` would collapse the threat model silently, and it is the obvious thing to +write. + +**Why not a ratchet.** Under group-key distribution *and* server-served history, +the node must retain each chain's **earliest** key, and a chain key at iteration +*i* yields every message key from *i* onward by pure HKDF. **Forward secrecy is +therefore zero either way.** What a ratchet was left buying is a large amount of +stateful client code with silent failure modes, three of which are concrete: any +member could sign as any other, a second device dropped the first's chain, and the +skipped-key cache grew without bound (§13.6, F1–F3). Forward secrecy is given up +**deliberately and on the record**. If it ever becomes a real requirement it +belongs in 1:1 DM, where there is no server-side history to contradict it. + +**Neither a sender-key nor a ratchet implementation exists in the tree.** Both +were written, neither was ever called, and both are deleted — the reasoning that +ruled them out lives at the top of `chatbox.py`, the module that replaced them, +where it stands on its own instead of pointing at a file to compare against. + +> **Kept code that nothing calls is worse than absent code.** It reads as an +> alternative somebody may reach for, its green tests read as evidence of a +> protection that is not in the product, and it has to be maintained past every +> refactor to stay compiling — maintenance spent on a decision already made. If +> forward secrecy ever becomes a real requirement, it belongs in 1:1 DM, where +> there is no server-side history to contradict it, and it starts from the +> requirement rather than from a module somebody left behind. + +**What chat encryption protects against, in the words the user-facing docs should +use:** someone who obtains the node's storage **without the keystore passphrase** — +a hosting provider imaging the machine, a leaked backup, a seizure where the +passphrase is not surrendered. It does **not** protect chat from the operator or +any current member (they hold the group key, and the chat key is delivered under +it); from anyone holding any one device of any member; from a former member, for +messages sent before the epoch changed; from the hub as regards *metadata*; or +from the node as regards *links posted*, which it fetches to unfurl. + +**Deliberately not encrypted**, stated so nobody reads more into the feature than +it does: `sender_id`, timestamps, message sizes and the fact of a message are in +the clear to the node, which is the relay and cannot route otherwise. The hub +learns per message the group, the time and the sender's account id, so it can skip +the author when creating notifications — it can build a social graph with timings +without reading a word, and that is a known metadata leak rather than a solved +problem. Attachments are ordinary files on a root and stay plaintext on disk; the +*reference* to one is inside the sealed payload, but the file and its name are in +the index. + +### 4.6 Parameters + +| Parameter | Value | +|---|---| +| Node keystore KDF | Argon2id **256 MB**, t=3, lanes=4 — recorded per envelope, so raising it does not orphan existing keystores | +| Hub password verifier | Argon2id **256 MB**, t=3, over the client-derived `auth_key` — `pw_version` migrates transparently on next login | +| Browser bundle key | Argon2id **128 MB**, t=3, p=1 | +| Browser `auth_key` | PBKDF2-SHA512, **600 000** iterations | +| Chunk cipher | AEAD, 1 MB chunks, per-chunk key by HKDF | +| Chat nonce | 96 random bits per message, never a counter | +| Invite / pair codes | 40 bits, Crockford base32, single use, stored as `sha256` | + +--- + +## 5. The node protocol (MNP) + +### 5.1 Transports + +| Listener | Role | Status | +|---|---|---| +| **WebRTC DataChannel** | primary, browser **and** desktop client | the path everything is tested on | +| **QUIC** | LAN, port-forwarded, hub-less `group://` | **off by default** (`[node] quic_enabled`) — nothing ships a QUIC client yet, and it does not implement chat (below) | +| ~~TCP + TLS~~ | — | does not exist (**C6**) | +| ~~HTTP file API~~ | — | does not exist (**C1**) | + +**QUIC carries no chat, deliberately.** It implements neither the per-device +sealing nor the device identification the chat rules require, and a message +reaching a group's archive without them would be a plaintext row in an encrypted +history — indistinguishable from one somebody actually wrote. The handler was +**removed rather than gated**: refusing there would mean maintaining a second, +weaker set of rules for a transport with no client, which is how a bypass branch +survives. An unimplemented message type is logged and dropped, like every other +one this transport does not have. + +> That is the general form of the parity rule. **A transport either implements a +> content rule fully or does not serve that content at all.** "Implements the +> authentication but not the authorization" is the shape of finding **C6** and of +> the third review's **M2**, twice. + +**ICE/STUN is the NAT traversal mechanism**, for native clients too — via +`aiortc` in Python. `punch_nat()` is a direct-connection helper, not a traversal +stack: one UDP probe to one address, no STUN client, no candidate gathering, no +dual-stack fallback, and it requires the client to already know its own external +address. It was validated on one ISP and one NAT type. ICE has been validated +across two ISPs, two browsers, IPv4 STUN and IPv6 direct, and 4G CGNAT +(§11.1) — **no TURN relay is needed**. + +**Several STUN servers, two providers deep**, because a single server is a silent +single point of failure that adds the full gathering timeout to every connection +when it is slow: + +``` +stun:stun.l.google.com:19302 +stun:stun1.l.google.com:19302 +stun:stun.cloudflare.com:3478 +``` + +Both sides carry the same defaults and gather independently; neither learns which +server the other used. **A STUN server learns the querier's public IP and NAT +mapping — that is its purpose.** No content, no credentials and no group metadata +passes through it; adding one is trusting its operator to learn your NAT topology, +nothing more. + +The node's list is editable three ways (Node page, `meshbay-node stun`, +`node.toml`) and hot-swapped on save. **The browser's list is hardcoded and not +configurable**: the hub relays SDP, not ICE policy, and no mechanism exists — or +is wanted — for a node to push ICE configuration to a browser. + +**ICE interface filtering** (`ice_interfaces`) is node-side only, and controls +which local addresses the node offers as candidates. Two modes: auto (excludes +virtual and VPN adapters by heuristic) and manual (an explicit whitelist). It +exists because VPN clients add virtual interfaces whose `.local` mDNS candidates a +remote node spends seconds failing to resolve. The browser's own gathering is +governed by the engine and is outside this application's control. + +### 5.2 The handshake + +One implementation, `meshbay_common/handshake.py`, called by both transports. Two +implementations of one security check is **C6** waiting to happen. + +``` +client → node handshake {token, group_id, nonce_c, v, v_min} +node authorize_token() JWT · scope · denylist · group_id · membership · hosting +node → client handshake_challenge {nonce_s, node_pk} + ── pre-proof window: bundle fetch, join ── +client → node handshake_response {proof} +node verify HMAC(GEK, client transcript) +node → client handshake_ack {proof, sig, node_pk, sealed configuration} +client verify HMAC(GEK, node transcript) + Ed25519(node_pk), + and that ack.node_pk is the key announced above +``` + +**The transcript is length-prefixed, domain-separated and role-bound:** + +``` +"meshbay:mnp:handshake:v1" ‖ len‖role ‖ len‖group_id ‖ len‖nonce_c ‖ len‖nonce_s ‖ len‖binding +``` + +Every field is length-prefixed so the concatenation is unambiguous, and the role +is bound in so a client proof can never be replayed as a node proof (**L4**). + +**Channel binding is mandatory and an absent one is refused** — never degraded to +nonce-only, which would silently drop MitM detection: + +| Transport | Anchor | +|---|---| +| WebRTC | both DTLS certificate fingerprints | +| QUIC | SHA-256 of the server certificate. On a **resumed** session, which carries no certificate, the anchor travels with the session ticket — sound, because the ticket derives from the handshake where the certificate was presented | + +**Authentication is mutual** (**C3**). The node proves possession of the group key +over a **client-chosen** nonce *and* signs the transcript with its long-term key. +The client verifies both, refuses a bare ack, and **TOFU-pins `pk_node`** per node, +refusing a changed key outright with a deliberate reset path in Settings for a +legitimate rotation. + +> `node_pk` is announced in the challenge because joining needs it before the ack: +> a first-time member signs a transcript naming this node and has no group key to +> complete a handshake with. It is unverified at that point and is never a +> substitute for the ack. + +**The pre-proof window is three messages, and that is a bound to defend.** Only +the two bundle fetches and `join_request` are served before the proof, because +each is something a caller needs *in order to* prove possession of the group key: +a returning browser has to recover its identity, and a new member has no key to +prove with. Nothing else qualifies. Device messages are **authenticated-only** — +a device request is countersigned later by a device already pinned, so requiring +the caller to finish its own handshake first costs nothing and keeps the surface +at three. The bundle fetches are additionally bounded (4 per connection) and +audited, because that window is the neighbourhood **C4** and **C5b** came from. + +**Pinning is defence in depth, not the primary control.** A substituted node +already fails the key proof. Pinning covers the case where an attacker *holds* the +group key — an ex-member, a leaked key — and swaps the node underneath, which the +proof alone cannot distinguish from the genuine node. + +**Authorization rules:** + +- `group_id` is **mandatory**. Omitting it once skipped the membership check and + fell back to the node's first group (**M1**). +- `scope == "user"` is enforced by default; node-scoped daemon tokens are refused + on the client path (**M9**, **NS7**). +- The denylist is consulted for user, `jti` **and** group. +- The node **refuses connections when it holds no group key** — there is no + `gek_required: false` bypass (**NS8**). + +**Refusals carry a code**, not only a sentence, because a client can act on a code. +`not_a_member` in particular is usually a token issued before the person was added +to the group — `groups` is baked in at sign-in and the hub pushes no updates — so +the client refreshes once and retries rather than telling someone who was invited a +minute ago that they are not a member. + +### 5.3 Correlation and liveness + +**Every reply carries the request id it answers** (`req_id`). Matching by arrival +order is a guess that fails silently and asymmetrically: the victim is never the +request that was answered wrongly, it is the unrelated one now waiting for a reply +already delivered elsewhere. The node stamps `req_id` on the reply from `_send`, +via a ContextVar so a handler's spawned work still answers under the right id, and +never on a broadcast, which answers nothing. + +`PING`/`PONG` is liveness on an **already-open** channel and never a discovery +mechanism — opening a connection costs a full ICE/DTLS handshake, measured at +0.6–7 s. + +### 5.4 Operator-signed operations + +Destructive and privileged operations require an **Ed25519 signature over a +structured transcript**, never a JWT. The hub controls JWT issuance and can +therefore never establish node-level authority. + +``` +"meshbay:admin:v1" ‖ len‖op ‖ len‖node_pk ‖ len‖group_id ‖ len‖subject ‖ len‖nonce ‖ len‖ts +``` + +TTL 120 s. **The client reconstructs the transcript from announced fields and +refuses to sign if the operation or subject is not what the user asked for** +(**H5**) — a challenge of opaque random bytes signed blind is an unbound signing +oracle. The transcript's subject names the *outcome*, not the operation: what the +operator is shown before signing has to be what happens. + +Verification is against `roster.operator_pks()`, rebuilt from node state, **never** +from anything in the response. + +| Operation | Authority | +|---|---| +| `file_delete` | the operator, or **any non-revoked device of the uploading account** | +| `dir_delete` | the operator alone, and only on an empty directory | +| `invite_create` | the operator (or a delegate, when delegation ships) | +| `gek_rotate` | operator-signed; the node generates the key itself | +| initial `gek-init` | **local admin API or CLI only** | +| root add/remove/update/eject/plug, `apps_enabled`, app directories, transfer limits | operator-signed | +| ~~`gek_bundle_store`~~ | **the message does not exist.** No member ever hands the node key material | + +`gek_bundle_store` was deleted rather than gated. The operator's X25519 public key +is announced in the handshake ack, so any member could wrap a key of their choosing +for it; a path that does not exist cannot be mis-authorised (**C5b**). + +**Authorisation is against the account, not the key.** With several devices per +person, verifying against the exact uploading key would refuse a person's desktop +the right to delete what their phone uploaded. `uploader_pk` stops being the +authorisation key and becomes the **audit record** of which device acted. This +remains **roster-rooted, not token-rooted**: a hub minting a token that claims to +be someone holds no key the node pinned for them, so the signature fails. + +**Ownership is provable, not merely recorded.** The uploader signs +`meshbay:upload:v1` over node, group, root, path, content hash, account and +timestamp, and the node stores it with the index entry. Ownership is then +verifiable by any member rather than asserted by the node, and the path where +overwriting a file made the attacker its recorded uploader (**C5a**) is closed a +second time. + +**One implementation, several front doors.** `meshbay_node/ops.py` holds every +operation. The loopback API, the CLI and the signed MNP handlers all call these +functions; they take the daemon state, raise `OpError`, and know nothing about +HTTP. Two implementations of one operation with two authorization checks is +**C1**/**C6** one size down. + +### 5.5 Transfer leases + +A download used to be invisible to the node: a client sent eight independent +chunk requests and reassembled the answers, and nothing said a transfer had +started or ended. There was nothing to count and therefore nothing to cap. + +**The unit is the lease** — the node's record that a peer is transferring +something, held for the length of the transfer and released by name +(`transfer_open` / `transfer_close` / `transfer_state`). Six properties are +decisions: + +- **`tr` is drawn by the client.** Re-opening after a reconnect with the same `tr` + is idempotent, so a reconnect cannot charge a member twice for one transfer. +- **A lease is scoped to the connection**, never to the account. It dies with the + session, which makes the primary reclaim deterministic. +- **A lease covers a job, not a file.** A directory zip is dozens of files and one + lease. +- **Nothing is persisted.** A restart drops every session anyway; a lease that + outlived the process would be a slot nothing can release. +- **Leases are counted, not bytes.** What a slot protects is concurrency — open + file handles, disk seeks, the channel buffer each transfer keeps full. +- **Per-member first, then node-wide.** A member at their own cap queues behind + their own transfers and never holds a node-wide slot a second member has none + of. Reversed, whoever arrives first takes everything. + +| Bound | Default | Why it exists | +|---|---|---| +| Node-wide concurrent transfers | 8 | operator's machine | +| Per account, per group | 2 | absent means this, not "unlimited" | +| Grant deadline | 30 s | a grant nobody takes up is a slot nobody can use | +| Idle timeout | 120 s | catches a peer that vanished without the connection noticing | +| Queued per member per kind | 32 | an unbounded queue is how a node runs out of memory politely | +| Missed grants before closing | 3 | without a bound the requeue is a permanent cycle | + +**Browsing a group is never subject to a transfer slot.** Not the poster grid, not +the album covers, not the video thumbnails, not the file list, not opening a photo +or a document to look at it. **A member must be able to browse a group that is at +capacity exactly as they browse an idle one.** That is a requirement, not a tuning +parameter, and it is met structurally rather than by choosing a lucky threshold. + +Navigation proper never touches this path at all: listings and metadata are their +own message types, sealed under the group key, with no relationship to a chunk +request. That half needs no rule — only a test that fails if someone later routes a +listing through the chunk path. + +The chunk path itself carries three genuinely different things, and they are +distinguished **structurally, by what the id resolves to and by which function +asked**: + +| What | Resolves to | Rule | +|---|---|---| +| Thumbnails, posters, cover art, cached audio conversions | a **media-cache id**, not an index entry | **Never leased, never counted, never queued.** One chunk each, out of a bounded cache the node built itself | +| Looking at one file — a photo full size, a document, an image | a real index entry, fetched whole | **Not leased**, subject to the bound below | +| Downloading and uploading | a real index entry | **Leased** | + +The rule stated as a sentence someone can check by reading: + +> **A transfer is something the transfers panel shows. If it does not appear +> there, it does not take a slot.** The two sets are the same three call sites, +> which is what makes this verifiable rather than a judgement call at each new one. + +**Why the exemption is expressed in concurrency and not in size or bytes**, which +is the durable part of this decision: + +- **A size threshold does not separate the two.** A raw photo out of a camera is + 60–80 MB and is *browsing*; a 40 MB archive is a *download*. Any threshold + letting the photo through lets the archive through too. +- **A byte-rate budget does not either.** It would have to be large enough for that + same photo, at which point it is large enough to be a download channel. +- Concurrency is the thing being rationed, so concurrency is what the exemption is + expressed in. + +The bound is therefore a ceiling on **how many distinct files one session may read +leaselessly at once** (12, with a 60-second idle expiry). Per session and not per +member, because this is a ceiling on what one connection can do while claiming to +be browsing, not a resource pool — a member with three tabs open is browsing in +three tabs. Two rules keep it from becoming a bug: **a file already being read is +always admitted**, whatever the count, because refusing a chunk halfway through a +photo is worse than never having admitted it; and **entries expire on idle**, +because a viewer closed mid-file stops asking and says nothing, and dead entries +would eventually refuse every later preview. + +**A preview never shows "waiting", because a preview never queues.** If the bound +is somehow reached, the request is refused with a stated reason and the person tries +again — it does not silently become a queued transfer in a panel they were not +looking at. + +**What the residual is, stated plainly.** A client that lies — labelling a bulk +download as a view — gets that bound's worth of files at a time instead of its +member cap. It is bounded, it is audited, and it is the same class of statement as +the cap itself: **this is a fairness control among cooperating clients**, in the +company of the stream cap. It is not a defence against a member determined to +saturate the operator's disk, and must never be described as one — that member is a +member, and the answer to them is `member revoke`. + +**The numbers are visible or the queue is unprovable.** `transfer_state` carries +`used`, `cap` and `ahead` so a client can say "waiting — 2 of your 2 slots are +busy" rather than showing a bare spinner; the same counters reach the loopback API +for the CLI and the Node page, and a periodic debug line. When someone reports a +transfer stuck at "waiting", that line is the only thing that will say whether the +node ever had them in a queue at all. + +The lease module is free of asyncio and of the transport: it decides, and the +caller does the I/O. A queue that reveals itself only through a DataChannel is a +queue nobody can prove things about. + +### 5.6 Versioning and flag days + +MNP and MHP version independently of the package version. Every wire message +carries `v`; both peers declare `v` and `v_min` on the handshake and refuse each +other with a code (`version_too_old` / `version_too_new` / `version_unreadable`). + +**A mismatch is a refusal, not a field that turns up missing.** A stated refusal is +a bug report; a feature that quietly does not work is a support case. + +Additive changes are MINOR and cost nothing. A change to what a peer must be able +to *do* is MAJOR even when the messages are additive — a peer that cannot ask for +a transfer lease is either refused, or not refused and transferring outside every +cap the operator set. + +**There is no compatibility switch, by policy.** An opt-in flag leaves the old +branch reachable on every node, which is **C6**'s lesson one feature later. Where +a break is required, `MNP_MIN_SUPPORTED` moves with `MNP_VERSION` and the +deployment is coordinated: the hub serves the SPA, so a browser picks up the new +client on reload; the desktop client ships its own UI, which is why +`GET /v1/hub/version` carries `client.minimum` and the client checks it **before** +connecting and says "this version can no longer connect" rather than showing a +handshake refusal nobody can act on. + +**The floor being the current version is what keeps capability branches out of the +client.** `MNP_MIN_SUPPORTED` equals `MNP_VERSION`, so `check_version` refuses +every older peer at the handshake — which means **every capability is true of every +peer the client can reach**, and there is nothing to test for. An upload is sealed +or it is not sent; a transfer has a real lease or it does not run; there is one +app-directories op and no wrappers behind it. The client records the version its +peer declared, for diagnostics, and **branches on none of it**. + +> A capability flag on a peer whose floor already guarantees the capability is a +> branch that can only ever take one path — until somebody lowers the floor, at +> which point it silently takes the other. **A field kept "just in case" is how +> the branches come back.** + +Where a break leaves data behind, a migration runs with the node stopped, backs +the database up first and is idempotent. But **a migration that has to be +remembered is a migration that does not happen**, so anything that *can* be a +read-time fallback is one instead: the roster reads an older settings key when the +new one is unset and leaves it behind on the first write, and `node.toml` keeps +being read in its older spelling. Only a transformation nothing can infer — two +settings that disagree, where only the operator knows which they meant — is +allowed to need a step somebody has to run. + +--- + +## 6. The node + +### 6.1 Node authority + +The node is the content authority. Its authority comes from **its own roster**, +established locally by pairing, and from nowhere else — never from the hub, never +from a config file, never auto-pinned from the keystore (**NS4**). A configuration +that still names a legacy admin key is warned about at startup and never obeyed. + +The division of trust: **the hub certifies identity; the node authorises content +operations.** Hub membership lets someone *reach* a node; the node's roster decides +whether it wraps anything for them. + +### 6.2 Roots + +A group's content is **a set of named roots**, each mapping to a local directory, +forming one union virtual root: + +``` +/ (group virtual root) +├── Films/ → D:\Media\Films +├── Music/ → E:\Audio (external drive, removable) +└── Documents/ → C:\Users\me\Share +``` + +**The name is the chosen directory's basename, derived once at add time and +stored.** Never recomputed from the path: renaming a folder on disk would +otherwise silently re-identify a whole library and break every stored reference to +it. Four rules make basename naming safe: + +- **A duplicate basename is refused, case-insensitively.** Collisions are common in + practice (`D:\Films` and `E:\Films`). Refusing is correct; an explicit alias is + the escape hatch (open item O11). +- **No root may contain another**, compared case-insensitively and after + canonicalisation. Two nested roots would index the same bytes twice under two + identities. +- **The basename becomes a path segment every member sees**, so it must itself pass + the portability rules (§10) — a root Windows cannot write to is a root nobody on + Windows can download from. +- **Every index path carries a root segment**, uniformly, including in a + single-root deployment. One code path, not two. + +Five consequences, none optional: + +1. **The root name is part of a file's identity**, so renaming a root rewrites + every path under it. Renaming is explicit and warned, never a cosmetic setting. +2. **Availability is per root.** With one directory an unplugged disk was a hazard; + with named roots it is a supported state — one root freezes, the others carry on. +3. **Free space, quotas and capacity are per root** — different volumes. Anything + the UI says about space names which root it means. +4. **Path resolution is per root**, in one place: a request names `<root>/<path>`, + `RootSet.resolve()` resolves it against that root's canonical path and refuses + `..`, absolute segments, symlinks and anything escaping the root. +5. **`kind` is a view hint** (`generic`/`video`/`audio`/`photo`), nothing more. + +**Each root is read-only or read-write.** + +- `writable = false` (the default) means read-only **for everyone, including the + operator**. A published library that quietly accepts writes from whoever holds + admin authority is not one, so refusing the operator is the point rather than + the defect. +- `writable = true` means any group member may upload there. + +Several roots may be writable and none need be — a fully read-only group is valid. +The operator toggles this with a signed op. + +> **There is one answer to "may this member write", and it is the root.** A single +> flag over the group cannot express "this library is published read-only and that +> folder is a drop box", which is the ordinary arrangement — so the group-wide +> switch that used to exist is gone entirely: the message, the signed operation, +> the field on the handshake ack, and the `upload` alias each root used to carry +> beside `writable`. **Two sources for one question is one too many**: whichever +> the code consulted first decided it, and a client falling back to the group flag +> whenever a root omitted `writable` is exactly that bug with a compatibility +> justification. Two names for one boolean is the same fault one size down. + +> **A control that writes must name where.** With two writable roots the node +> cannot choose without guessing, and a guess sends a member's file to a disk the +> operator did not intend. The client names a **root**, never a path; everything +> below the root is decided by the node. + +**A root that goes away must freeze, not empty.** The indexer runs a watchdog +observer and rebuilds on change; unmounting a volume either emits deletions for +the whole tree or presents an empty directory to the next scan. Both propagate as +though the owner erased their library. So a root has two independent runtime +states: + +- **`ejected`** — operator-controlled, persisted in `roster.db`. +- **`available`** — computed as `not ejected and is_live()`. This is what clients + and the indexer see. + +The distinction matters: between clicking eject and physically unplugging, +`is_live()` is still true, and without `ejected` the availability sweep would +immediately flip the root back. + +**Eject** stops that root's observer, marks it unavailable, freezes its entries and +propagates the availability change to connected peers — the operator can then +safely unplug. **Plug** checks the path is accessible first, then rescans: the +plan called for a reconciliation, but a device people carry around can come back +arbitrarily different, and the hash cache means unchanged files are not re-read. + +**`ejected` is persisted and restored at startup**, because a restart is exactly +what an operator does after noticing a drive fell off, and an in-memory flag would +let the following scan read the empty mount point as an erased library. It lives in +`roster.db` and not in `node.toml`: it is runtime state, and an operator's +hand-written config must not be rewritten because a USB drive was unplugged. + +**Auto-eject is the safety net.** If a `removable` root's path disappears, the +availability sweep sets `ejected` as though the operator had clicked it, and +reports it so the daemon persists it. Nothing is deleted: index entries, cached +metadata, thumbnails, chat history referencing those files and app directory +configurations all survive, the last flagged as temporarily invalid rather than +wrong. + +### 6.3 Indexing + +The index is **content-addressed**: `GroupIndex` is keyed by blake3, so the same +bytes at two paths inside one group are **one** entry. This is why a scan can +report ten files and index nine, and it decides how reconciliation must work: + +> **Anything comparing disk against index must compare ids, not paths.** Comparing +> paths makes the second path of a duplicated file look like a missed event, every +> sweep, forever — rewriting the entry, bumping the version and pushing an index +> update to every connected peer. + +**Hashing is partial above 40 MB.** A hash exists for content identity, and a 4 GB +file does not need 4 GB of I/O to be identified with overwhelming probability: + +| Size | Method | `hash_version` | +|---|---|---| +| ≤ 40 MB | full read | `1` | +| > 40 MB | blake3 over the first 20 MB ‖ last 20 MB ‖ 5 MB at the midpoint | `2` | + +Head and tail catch container headers, trailers and files that differ only at one +end; the mid-sample catches files sharing a header and trailer. Below the +threshold a partial read would sample the whole file anyway, so the full path is +simpler and produces the same value — which is what keeps small files +cross-comparable between nodes of different versions. A large file indexed by a +node of each version produces different ids and does not merge in cross-group +search; that resolves itself when both upgrade, and is the accepted cost of not +reading 4 TB to build a library. + +`hash_version` is an additive index field with a default, so an entry written +before it deserialises correctly and needs no protocol bump. The hash cache +carries the column and auto-migrates on open; large files are re-hashed lazily on +the first scan after an upgrade. + +**Periodic reconciliation is mandatory on every platform**, not a backstop: +`ReadDirectoryChangesW` drops events under load on Windows, and inotify is +unreliable on a FUSE-mounted volume on Linux (§10). + +**Progress accounting covers the real-time path too.** A file's size joins the +total the moment its debounce timer is first scheduled — not on every re-trigger, +or a cancelled-and-rescheduled timer double-counts — and joins the scanned total +when its hash finishes. The scanning flag clears only when no timer is pending +*and* no hash is running, because the in-flight hash of a large file is the entire +reason to show progress. + +`index_progress` is deliberately **not** sealed: counters only, every 2 s. + +### 6.4 Uploads + +Five protections, and they are the substance: + +- a **filename allowlist**; +- **no overwrite** — a colliding name gets a free one. The check is `Path.exists()`, + and `stat()` is itself case-insensitive on NTFS and exFAT, so this already holds + there; +- **strict chunk ordering**; +- a **size cap** — 4 GB per file. There is deliberately no aggregate quota yet, and + that gap is named in §15.3 rather than left to be discovered: a member can still + fill the operator's disk one capped file at a time; +- the target root must be **writable and available**, enforced by the node. + +**There is no quarantine subdirectory.** A folder appearing beside the operator's +library because somebody sent a file is the node deciding how their disk is +arranged. What made a quarantine worth having was never the subdirectory — it is +the four rules above, and they are unchanged. The client now names the destination +folder, which is safe for exactly one reason: it is resolved through +`RootSet.resolve()` (§6.2). **A member answers "which of this group's folders", +never "which path on the operator's disk".** + +If the named root is unavailable the upload fails **with a stated reason** and +never falls back to another; if the group has no writable root, uploads are refused +rather than guessed. + +**The node enforces it; the interface merely stops offering it.** Each root's +`writable` flag rides the handshake ack and the index payload, so a client knows +whether to draw the Upload button and the chat paperclip, and changes are broadcast +to everyone connected. None of that is the control: a member on an old tab, or one +speaking MNP directly, is refused by the node. + +In a group with **no** writable root the interface says so plainly rather than +picking one — a fallback that chooses whatever comes first only moves the failure +to send time, where the person has already chosen the file. + +**Archives are not the node's business.** A member downloading a folder as a zip +fetches the same encrypted chunks as any other download and assembles the archive +in the browser. The node serves no bundles, holds no temporary files, and cannot be +asked to compress anything — one fewer place where a request turns into work on +someone else's disk. + +Directory creation is not privileged. Directory **removal** is, and is refused +unless the directory is empty. The emptiness rule is the safety property: whatever +the caller intended and whatever the client sent, the operation cannot destroy +content. It is checked twice — before the challenge is issued and again after the +signature returns — because a file can land during the round trip to the +operator's browser. + +### 6.5 Derived data and enrichment + +The governing rule: + +> **Enrichment happens on the client, from data it already has. What the client +> cannot compute, the node produces — and where the node produces it for +> everybody, it caches it in its own `data_dir`, never in a shared root.** + +The client half is unchanged and is why a chat image thumbnail costs the node +nothing: the browser already decrypted the image and scales it itself. + +The node half was decided against the alternative and the reasoning is worth +keeping, because it looks like a violation of "no derived state" and is not: + +1. **A third-party API quota is per credential, not per device.** A token shipped + inside every install and called from every client scales with the number of + *devices* in existence. A node making the calls on behalf of its own members + makes the number of *nodes* the denominator, and one lookup per unique title + serves every member indefinitely. +2. **Thin clients benefit from a node that does more.** A phone should receive a + small ready-made image and a JSON blob, not decode video or hold its own + multi-gigabyte cache. +3. **Keeping the credential server-side is a strict improvement** over shipping it + to every renderer, and it preserves the desktop client's tested invariant: it + issues no request outside `/v1/` and the signaling socket. + +**The cache lives in the node's own `data_dir`** — beside `chat.db`, `audit.db`, +`bundles.db` — and never inside a shared root. That is not a compromise, it is +strictly better, for reasons independent of sovereignty: a shared root is +routinely a read-only backup mount or a share the node cannot write to; a +dot-prefixed folder is not hidden on Windows and would appear as an ordinary +folder full of previews of a private group's content to anyone who plugs the drive +into another machine; and it would need filtering out of every listing path +consistently, forever. + +**Delivery reuses the chunk path.** A thumbnail is addressed by its own blake3 +exactly as a file is by its id, so the chunk handler resolves a requested id +against either a real file or the media cache. Same transport, same group-derived +encryption, same backpressure — no parallel mechanism, and no new authorization +surface. + +**Cache lifecycle is tied to the index.** A file's thumbnail and its third-party +match are pruned by the same event that removes its index entry. Third-party +metadata is cached per external id with its own refresh window, since several files +of one show share one fetch. + +**A visible node-wide toggle turns third-party calls off entirely**, independently +of any application being enabled, for an operator who wants zero third-party +network traffic. This is genuinely new node behaviour — egress to a third party, +and a disk-resident cache with a real deletion obligation — and it is stated rather +than left implicit. + +**Anything that shells out to a media tool obeys three rules**, all of which this +codebase has paid for: + +- **its own small bounded pool with a short timeout, never the streaming pool** — + a grid of fifty videos would otherwise exhaust every streaming slot on the node, + since a stream slot is held for the length of a film; +- **drain the pipes, then wait with a timeout, and release the slot regardless** — + a process that outruns a paced reader cannot finish closing while its stdout is + full, SIGKILL or not; +- **a background task must be held**, or the loop may collect it mid-flight and the + slot is lost for good. + +**Link previews** are a further instance. A URL pasted in chat is unfurled **by the +node**: the browser cannot (a strict `img-src`/`connect-src`, and CORS), and a +direct fetch would leak every reader's IP to the linked host on each render. The +card text lives in a bounded in-memory TTL cache; the image rides the same +blake3-keyed store as any other thumbnail. **The new surface is SSRF**, because the +URL is a member's choice and it triggers an outbound request from the operator's +machine: http(s) only, no credentials, a port allowlist, every resolved address +must be globally routable, redirects followed by hand so each hop is re-checked, +the connect address re-checked against the checked one, a response-size guard, and +a per-member rate limit. The operator can switch previews off per group. + +**Every new outbound or cross-trust surface needs a bound and a named adversary in +the same commit.** That is the standing rule this section exists to enforce. + +### 6.6 Chat storage + +One SQLite database per group at `data_dir/<group_id>/chat.db`. Rows carry the +sealed ciphertext, the epoch, the sending device, the nonce and the signature, with +a unique `(device, nonce)` refusing replays. + +Paging is **backwards** — `get_recent` / `get_before` / `has_before` — because a +chat opens at the newest page. A forwards pager is not what a chat opens with. + +`meshbay-node chat prune <days>` deletes **messages only, never an epoch key**. An +epoch with no messages is harmless; an epoch key deleted while messages still need +it is an unreadable archive. + +### 6.7 Operator surface + +Two personas need different tools, and the headless one is the normal deployment: + +| Operator | Reaches the node via | +|---|---| +| Desktop | the desktop client's Node page | +| **Headless / SSH** | the CLI | + +**Every operation is reachable over SSH with no browser on the host.** `status` +deliberately reads the keystore and config directly so it works while the daemon is +stopped — the state an operator is most often in, since the daemon will not stay up +before its key is linked or a group exists. + +``` +meshbay-node status +meshbay-node group add <name> --dir <path> [--no-writable] +meshbay-node root list|add|remove|set|eject|plug +meshbay-node gek init|rotate +meshbay-node operator pair +meshbay-node member list|invite|revoke|unpin +meshbay-node member device list|revoke +meshbay-node denylist show|clear +meshbay-node file list|rm +meshbay-node chat status|prune|encrypt-history +meshbay-node stun list|add|remove|reset +meshbay-node reload +``` + +`member revoke`/`unpin` resolve a username against the roster and **refuse an +unknown one** rather than acting on nobody — a typo must not look like success. +Revocation tells the operator what it does *not* do: the ex-member stops receiving +the key on their next connection but still holds the current one, so the message +ends with the command that rotates it. + +**The node's local control API is JSON only, on loopback, behind a per-run session +token** (`X-MeshBay-Token`, printed at startup, file mode 0600). "Localhost only" +is not authentication: any local process can reach it, as can a page in the +operator's browser via DNS rebinding — and this API re-initialises group keys, +issues invitations and reads the audit log. There is no server-rendered dashboard; +the desktop client's Node page and the CLI are the two consumers, and each +operation endpoint is one `_op(...)` line onto `ops.py` (§5.4). + +**The accepted cost, recorded as a choice:** on a headless server the only admin +path is the CLI. The CLI covers every operation, so this is acceptable — but it is +a real capability reduction, not an oversight. + +> **MNP is the path that must exist; loopback is the fallback.** The operator of a +> node is not necessarily sitting at it. Any operator-facing control needs its MNP +> route first, or it renders for nobody on the web. + +### 6.8 Node settings + +Five `[node] `settings affect what the node does rather than how it starts, and +their value is invisible until something goes wrong — so they are surfaced on the +Node page: + +| Setting | Default | What it controls | +|---|---|---| +| `invite_ttl_hours` | 168 | how long a member invitation stays valid | +| `pair_ttl_hours` | 24 | how long an operator pairing code stays valid | +| `device_request_ttl_minutes` | 60 | how long a device request waits for approval. Comfort, not security: the code is bound to the keys by its hash | +| `max_concurrent_streams` | 8 | simultaneous video streams. One process per viewer, ~50 MB each; a slot is held for the length of a film, so this counts viewers | +| `transcode_incompatible_video` | true | whether browser-incompatible video is transcoded during streaming. Unlike remuxing this costs real CPU per viewer | + +> **Turning transcoding off does not mean the same thing for every source.** A +> codec with an MSE codec string falls back to a copy and the viewer's own decoder +> decides; a codec with none has nothing to fall back to and the stream is refused, +> naming this setting. + +**Settings are persisted in both `roster.db` and `node.toml`**: the database for +immediate effect with no restart, the file so the value survives a wipe or a fresh +install. On startup the file is read and a database override wins. The TOML write +is a targeted line replacement, never a round-trip through a writer — that file is +hand-written, full of comments recording decisions, and a setting changed from a +panel must not rewrite the operator's file. + +Transfer limits (§5.5) follow the same pattern, with per-group per-member caps as +an operator-signed op. + +--- + +## 7. The hub + +### 7.1 Role — chosen, not minimal + +Hub minimisation was considered and **deferred, and may be dropped** (decision D4). +The hub keeps serving the web UI and remains in the trusted path by choice. That is +a legitimate product call; what follows from it is carried deliberately rather than +by accident (§2.3). + +**Stores:** accounts (username, encrypted email, status, role), the group registry +and membership, IP logs (one year, legal retention), node registrations, refresh +tokens, notifications, the moderation blocklist, instance policy, and per-account +device keys for hub login. + +**Does not store:** file content, file names, private-group indexes, message +content, private keys, group keys, keypair bundles, user identity keys, node IPs +beyond ephemeral signaling. + +**Knows, unavoidably:** who is a member of what, when nodes connect, and when a +chat message was posted in which group and by which account id. The last is a +stable identifier the hub needs in order to skip the author when creating +notifications; it carries no content, and it is a known metadata leak rather than a +solved problem. + +**Decides nothing about keys.** Hub membership lets someone reach a node; the +node's roster decides whether it wraps anything for them. + +`user_devices` **is not the key directory that was H3**: nothing reads it but the +hub, nothing wraps a group key for it, and it is a different key from the per-node +identities. What it does cost is metadata — the hub knows how many devices an +account has and when each last signed in. + +### 7.2 Node registration and signaling + +Registration on the node socket requires a **node-scoped token**, verifies the node +record against the token subject, and **derives group claims from the database**: a +node may narrow the set to what it hosts but cannot widen it, and cannot displace a +live registration (**C2**). + +The node authenticates to the hub with an Ed25519 signature over a domain-separated +timestamped message — **no password and no auth key on a node** — and receives a +`scope: "node"` token that is refused for group management. The operator manages +groups from a client (**NS7**). + +Signaling is rate-limited, SDP-size bounded, capped per user, and **the caller must +share an active group with the target node**. Otherwise any authenticated user +could make a third party's machine allocate peer connections on demand (**H6**). +The address in a NAT-punch request must match the caller's source address. + +`X-Forwarded-For` is honoured **only from a trusted proxy, rightmost hop** (**M7**), +through one helper so the behaviour is defined in one place — including for the +rate limiter, whose keying otherwise collapses to a single global bucket behind a +loopback proxy (third review L10). + +### 7.3 Groups + +A group's **identity is its UUID**, everywhere: the route, the node's configuration, +membership. A group **name is unique per owner account**, case-insensitively and +trimmed, enforced by a functional unique index; two different owners may each have +a `photos`. Names are displayed as `name@owner`, which is a label plus a create-time +check and **not an addressing scheme**. The handle is hub-local: the same +`name@owner` on two federated hubs are different groups, and a federated row shows +its source hub rather than an account. + +`visibility` and `join_policy` are the two independent axes described in §3.5. +`join_policy` is read from the node's own configuration, never from the hub. + +**Public group creation is quota'd** — ten live public groups per owner account, +staff exempt. Public groups are the ones that cost other people something: they +appear in the directory and are brokered to strangers. The check is at creation +only, which is correct because the update endpoint refuses to change visibility. + +### 7.4 Instance policy + +`hub_settings` is a key/value table an admin edits at runtime. It is **instance +policy, not group content**: it says how this hub behaves and holds nothing about +any group's files, index, membership or keys, so §1.3 is untouched. + +The first entry is `allow_public_groups`. Switched off, server-side and read live on +every path the hub mediates: creating a public group is refused (staff included — +the way back is to re-enable, not to slip past), the public directory returns +nothing local **and** federated, open-joining is refused, a non-member is handed no +node to connect to, the "this node hosts an open group, admit anyone" signaling +fallback is dropped, and the federation export advertises nothing. + +**It is a directory-and-brokering control, not a remote kill.** Existing members +keep their membership and their access. A node whose operator set `join_policy = +"open"` still pins and serves whoever reaches it directly over MNP; what the switch +removes is the hub-provided ways to find and reach such a node. + +### 7.5 Moderation + +Two verbs on a group, and they are distinct things: + +| | `suspend` | `revoke` | +|---|---|---| +| Hub | `status = "suspended"` | `status = "revoked"` | +| Node | nothing | signed revocation broadcast → denylist + live sessions dropped, **persisted across a restart** | +| Reversible from the panel | yes | no | + +The client shows the real state, not a blanket one. **Revocation is honoured by +nodes** and the denylist survives a restart (**H4**); signaling refuses a group that +is not active. + +**Moderator is not administrator.** The user-patch handler is split by field: a +moderator may act on the fields moderation needs and may not write `role`. + +**Content reporting requires authentication, distinct reporters and a rate limit, +and is refused when public groups are off.** An unauthenticated endpoint that +blocklists a content hash after two reports is a network-wide censorship and DoS +primitive for anyone who learns a public file's id. + +The exact-hash CSAM check is **structural, not yet functional** — production +databases are perceptual — and is stated as such so it is not relied on +operationally. + +### 7.6 Federation (MHP) + +Peer hubs exchange directory rows and revocations. The trust rules: + +- a pushed row's **source is bound to the signer**, not taken from the payload; +- the **token audience is checked**; +- a push is **capped**, and replays are rejected; +- **revocation acts on the peer's own directory entries** — it does not reach + nodes, and nothing local hosts a federated group. + +### 7.7 Account lifecycle + +A user can delete their own account from Settings behind a **passphrase re-entry** +— a live token may be a borrowed laptop, and the bar for something with blast +radius is proof of the passphrase. An admin can delete one too. + +The row is **tombstoned rather than dropped**: username released, email and password +hash cleared, node linking key dropped, memberships, notifications and refresh +tokens removed, active tokens refused at once by a status check rather than left to +expire. + +Two things survive on purpose: + +- **The IP log**, for its legal retention period, and it stays attributable — + detaching it would keep the data and lose the only thing it is for. The name is + copied onto those rows as the account goes, since the join that used to supply it + would answer with the tombstone. +- **Everything on a node.** Files, the pinned identity and the keypair bundle live + on machines the hub does not command — the same sovereignty that makes admission + work. **Deleting a hub account is not an erasure request to the operators who + host you**; the operator surface is where that happens, and the docs must say so. + +Deletion is **refused while the account still owns groups**, rather than cascading +into other people's data. + +Registration is gated by a CAPTCHA whenever one is configured — **unconditionally**, +not only when some other field is absent, or the real client's ordinary request +skips it. The desktop client renders the widget too. + +--- + +## 8. Clients + +### 8.1 Two clients, deliberately + +| | Hub-served web SPA | Desktop client | +|---|---|---| +| Distribution | served by the hub | installed, signed release | +| Code integrity | **T3 accepted** — the hub can inject | detectable *if* reproducible builds ship | +| Key storage | IndexedDB / sessionStorage, plus a bundle on each node | OS-protected local storage; keys never bundled | +| Transport | WebRTC | WebRTC **+ QUIC** (sidecar, for hub-less `group://`) | +| Downloads | to disk where the engine allows it | native, streamed, unlimited | +| Positioning | **convenience tier** — zero install | recommended for sensitive use | + +**The SPA is not deprecated and stays.** It is the zero-install path, and the +objective is explicit: **a native client must not prevent web use.** It must be +labelled honestly — the application page states that the hub serves this code, and +the docs never claim end-to-end *integrity* for that path. + +**Several browsers, one identity per node.** A browser keeps nothing durable the +user controls, so the identity it creates for a node is left with that node, +encrypted under the passphrase. Any other browser recovers it there with the +passphrase alone: same identity, same pin, no second code. Joining a *different* +node creates a different key and needs that operator's code — the first contact it +has always needed. This is what makes the product behave the way people expect, and +it is also **C4**, with a blast radius of one node. + +### 8.2 The desktop client + +**Electron**, with an optional Python sidecar for hub-less `group://` over QUIC. + +The shell choice follows from what the interface actually depends on: not "the web" +in general but engine-class APIs — `RTCPeerConnection`, WebCrypto X25519/Ed25519, +MSE, service workers, File System Access. Keeping the engine keeps the transport, +crypto, key-derivation, download and player modules **as the client**. They are not +browser workarounds to be deleted once native; they are the implementation. + +**The non-negotiable: UI assets ship inside the package and load from disk.** A +shell pointing at the hub's application URL is a browser with a different icon and +fixes nothing. + +What running it establishes, and what each fact costs: + +- **A CSP in a meta tag silently drops `frame-ancestors`.** It is sent as a header + by the protocol handler. +- **Service workers do not work on a custom scheme.** The application has none and + uses the native save dialog; the worker stays for the browser. +- **A secure context is what makes crypto exist at all** — without it the whole + subtle-crypto surface is undefined, AEAD included. +- **The renderer cannot call the hub.** Its custom-scheme origin is refused by CORS, + and the hub deliberately has no CORS middleware — its API is reachable from no web + origin. Every hub call leaves from the main process, which refuses any origin that + is not the signed-in hub. **A script served by the hub is refused by the policy**, + which is T3's mitigation demonstrated rather than asserted. +- **The device's hub key lives in the main process, never in the renderer.** + Generated, stored and used there; the interface asks for a signature and is never + handed a key. Same rule as the save dialog, and for the same reason: the renderer + parses decrypted content from nodes, which is attacker-controlled input. +- **OS-backed secret storage is real on a desktop and honest without one.** With a + keyring it is keyring-backed; headless, the same code reports unavailable and + **refuses to store rather than downgrading silently**. +- **Installation places files, never secrets.** No key generation in a package's + post-install step or an installer custom action — a golden image would give every + machine the same key. + +Two guards apply to anything the hub can display **inside** the application, which +is a phishing surface: plain text or a very restricted markup subset, never raw +HTML; and a visually distinct region labelled as a message from the hub operator, +never a modal that can imitate application UI. + +The download page is a **security page**: it publishes the release key fingerprint, +and a hostile hub serves that page too. The fingerprint must also be published +somewhere the hub does not control, or the relocation of trust is circular. + +### 8.3 One UI source + +`packages/meshbay-hub/src/meshbay_hub/static/` **is** the interface, for the web and +the application alike. The client's build copies it and CI fails if the copy drifts. +**Never edit the copy by hand.** + +**Shipping the UI in a package creates version skew for the first time.** Today the +SPA and the hub deploy together, so a response shape and its caller change in one +commit. Once the UI is installed rather than served, the hub API is a compatibility +surface — which is why `GET /v1/hub/version` carries a minimum client version, and +why the client checks it before connecting (§5.6). Cheap now, awkward later. + +**A second copy of the hub address is what breaks the application, not the +protocol.** A module that decides where the hub is — "empty string, same origin" — +is true of a page the hub served and false of one loaded from a package, where a +relative API call hits the application's own protocol handler and sign-up and +sign-in fail. There is one seam, `platform.hubBase()`, and a test refuses any file +that decides where the hub is or fetches the API relative to the page origin. + +### 8.4 URL space + +| Space | Served by | Seen by the desktop client | +|---|---|---| +| landing, about, downloads, news | the static site overlay | ❌ never | +| the application | the hub (SPA) | ❌ never | +| the versioned API and the signaling socket | the hub | ✅ only this | + +**The desktop client issues no request outside the API and the signaling socket.** +That is a testable invariant, and it is what any feature adding third-party egress +must preserve — which is one of the reasons enrichment is node-side (§6.5). + +### 8.5 Downloads and streaming + +**Downloads go to disk, never through RAM, on every platform.** There are three +mechanisms — a granted directory handle, a service worker streaming a response, and +a blob as the floor — and which exist depends on the engine. + +> **A fallback chain reaches its floor silently.** Where the first two do not +> exist, every download went through memory and the only visible symptom was a save +> dialog at the end instead of the start. When a chain degrades, check what the +> floor costs on **every** platform that will reach it. + +Specific rules the download path is built on: + +- **A service worker being active is not the page being controlled.** An + uncontrolled page's requests never reach the fetch handler, so the stream is + handed over and never asked for. Require the controller, and have the worker + confirm it served the request. +- **An idle service worker is killed, and a streaming response is not "something to + do".** The page pings the worker while it writes and the worker answers, because + receiving a message is the event that resets the timer. +- **Every await on a download path is bounded and says which chunk it gave up on.** + An unbounded await is a freeze nobody can report. +- **A filename in a content-disposition header needs both forms.** The RFC 5987 + encoding uses `'` as a delimiter and the standard escaper does not escape it, so a + plain ASCII fallback rides alongside — the next surprise loses accents rather than + the whole name. +- **Three headers decide whether a page may frame itself and they must agree** — + the frame source policy, the frame-ancestors policy and the legacy frame option. + Same-origin framing is what a streamed download needs; refusing every foreign + origin is unaffected. + +**Pause and resume, and where resumability actually lives.** A pause button that +quietly restarts a download from zero is worse than no pause button, so the +interface offers only what the target can do. + +> **Resumability is a property of the *target*, not of the platform.** The same +> engine yields a resumable target from a granted folder and an unresumable one +> from a service worker, on the same page, for two files in the same batch. So +> **each target declares it itself** and the record travels with the transfer; +> the widget renders from that record. This is the fallback-chain rule (above) +> applied one level up: whatever tier a transfer ended up on, it records which, +> and nothing infers it from the platform. + +Two design decisions rather than implementation details: + +- **A paused transfer holds nothing.** Resuming rejoins the queue at the tail and + the interface says so. Anything else lets one member close the node by pausing. +- **A resumed file is truncated down to the last whole chunk, never appended to.** + Writes are sequential whole chunks, so a size that is not a chunk multiple means + an interrupted write — and **a silently corrupted download is worse than a failed + one.** The resume record also stores the file's content id, so a resume whose file + is no longer in the index fails with "this file has changed on the node", which is + the truth. + +**Uploads resume through the seal, not around it.** The node already holds the +partial state; the question "how much do you have" is asked as an ordinary sealed +upload message with no bytes and a probe index, and answered inside the sealed ack. +Asking on the lease message instead would have put the operator's filenames on an +unsealed message — precisely what sealing the write path bought. The partial state +is keyed by member, directory and filename in the **group** context rather than the +session, so a reconnect finds it, and an orphaned partial with no live lease is +reaped on a timer and at startup. The four upload protections (§6.4) are untouched +by any of this. + +**Video streaming** is fragmented-MP4 remux (or transcode where the codec has no +MSE string) fed to a source buffer, with the node holding one slot per viewer. + +- **Flow control is a window, not a debt.** Read-ahead is bounded by *time past the + playhead*, with a small window of segments in flight topped up as they land, + driven by a clock and by playback and **never by arriving data**. Granting a + credit per append pulls at network speed, fills the buffer ceiling, and then goes + silent for the length of the accumulated balance. +- **Credit follows the buffer, decided in one place.** The append path grants + nothing, because the buffer's update event fires for evictions too — crediting + from it pays the node for the player's own housekeeping. +- **Flow-control accounting comes before every early return.** A segment that + arrived is no longer in flight, whatever is then done with it. +- **A new stream starts from a known state**, reset at the *start* of the stream and + never in the teardown of the one before, which is skippable. +- **A viewer holding credit deliberately must say so**, or the node's stall timeout + ends a film that is merely paused. +- **Seeking restarts the source with an index seek before the input**, clamped away + from the end and echoed back; the client supplies the timestamp offset, because + copying timestamps does not preserve position. +- **Losing a peer must stop its work, not merely forget it** — anything holding a + resource is shut down on the way out, or a closed tab transcodes for the length of + the credit timeout. +- **Resume positions are per file, per device, in local storage.** No protocol, and + nothing new learns what you watch. + +--- + +## 9. Group applications + +### 9.1 The plug-in architecture + +A group's UI is a **set of pluggable applications**, not one page. Two things +motivated the split: one file had become the thing every unrelated change touched, +and the roadmap wanted several more group-level surfaces — none of which need a +protocol change, because the indexer already classifies files as video, audio or +image and they read the same index, chunk and stream messages the explorer already +uses. + +``` +group-page.js ─┬─ the shell: the connection, the file index, the tab bar, + │ the video/preview modals — nothing app-specific + ├─ apps.js ─── the registry: [{ key, icon, labelKey, Component, Settings? }] + ├─ chat-app.js · files-app.js · video-app.js · music-app.js · photos-app.js + └─ group-settings.js — not an app, always present, never toggleable +``` + +Settings **is not an app and cannot be disabled** — it is the one way back if +everything else were turned off. + +Shared infrastructure lives in its own modules (`icon.js`, `file-utils.js`, +`hub-client.js`, `settings-ui.js`, `folder-tree.js`, `source-merge.js`) rather than +being re-exported from the shell, because **the shell importing an app that imports +the shell is a cycle**, and ES modules answer that with a temporal-dead-zone error +at first render: the component simply does not appear, with nothing in the console +to say why. + +### 9.2 What every application receives + +The shell builds one props object per render and **spreads** it into whichever +application is active. Every registered component gets the same context and +destructures what it needs — a new application does not get a bespoke prop list. + +| Prop | Why it is here rather than local state | +|---|---| +| `entries`, `nodeDirs`, `nodeRoots` | the group's file index. Chat needs it too, for image attachments — lifting it avoids two copies going stale against each other | +| `applyIndex(...)` | anything that mutates files calls this, so every application sees the result | +| `onPreview(entry)` | opens the shell's modal; an application does not own modal state | +| `transportRef`, `gekRef` | **refs**, never state, so a reconnect does not re-render every application | +| `deviceReady` | **the exception, and why it is a prop.** A ref not re-rendering is right for a transport reached into on demand and wrong for a *fact about the connection* an application renders from | +| `mayUpload` | computed once; a second derivation would eventually disagree with the first | + +An application that needs local state owns it. One pattern is worth carrying: **any +notion of "current location within the group" resets on group change**, because a +directory from the group just left rarely exists in the one just entered. + +### 9.3 Enablement and settings + +`enabled_apps` is a per-group setting on the same pattern as everything else the +operator decides: **stored on the node** (`roster.db` — not the hub, not +`node.toml`, for the reason in §1.3 and §6.8), **changed by a signed operator +instruction**, and **enforced by the node** refusing an unrecognised or empty set. + +Validation happens before a signature is ever requested. The whole set is signed in +one message rather than one op per application, so ticking several boxes costs one +signature. The subject is the sorted, comma-joined list, built identically on both +sides so the two arrive at identical bytes. + +The node's allow-list is the server-side enforcement — **a client that names an +application this node does not know is refused**, and an application the node +refused could not demonstrate anything. That entry and the client's registry line +are the whole of what adding an application costs. + +A group that has said nothing gets **Chat and Files**. `files` is **always enabled +and not toggleable**, and is added to the list at every writer so the two agree. +The ability to hide it was misleading: the protocol permits root exploration +regardless, so hiding the tab only ever misled. + +**A group with no stored set gets the default, never the whole registry.** Falling +back to everything registered would turn on an application nobody chose — including +one shipped behind a development flag. + +Changes are broadcast to everyone connected, so a disabled tab disappears without +waiting for a reconnection. A client that has not yet received the list shows +everything registered — a node that predates an application hides nothing. + +**An application's directories are the same shape one level down**: one generic +signed op (`app_directories`) keyed by the application's own registry name, stored +under `<key>_directories`, one MNP message, one loopback route. Adding an +application adds **no function, no message type and no route** — which is what +"plug-in architecture" has to mean to be worth the phrase. + +> **The per-application variants are gone**, and the reason is the interesting +> part. Three messages, three signed ops, three handlers and three `ops` wrappers +> were the same instruction three times, differing only in the key they wrote and +> whether they carried a string or a list. That shape is what made adding an +> application mean adding a message type, an op, a handler and a widget — and it +> meant three validation paths, of which the older ones validated nothing: a typo +> was stored, matched no entry, and the application showed an empty tab with **no +> way to tell "misconfigured" from "no files yet"**. One op has one validation +> path, and an unknown application name is refused rather than stored. +> +> What stays is a *read* fallback: the roster still reads the older per-app keys +> out of its settings table, because that is a key on an operator's disk rather +> than on the wire, and a node upgraded into this must find its own configuration +> (§5.6). + +**Each application's settings pane is its own file**, named in its registry entry, +and every pane takes the same props and nothing else. The split is the point: + +> **What every application has, the page does generically; what one application +> alone has, the pane does itself.** Pointing an application at folders goes +> through the shared saver; a third-party credential or a per-group switch is the +> pane's own business, made with the transport it is handed. + +An application that only needs directories therefore touches neither the settings +page nor the shell. + +**The folder picker asks the node for nothing.** The tree is derived from paths the +client already holds, so it shows what the group's index contains and no more. +There is no folder-browsing protocol and this does not add one. + +### 9.4 Adding an application + +1. **`<name>-app.js`** exporting a component with the standard props shape. +2. **Register it** in the registry. `key` is the wire identifier: it must match the + node's allow-list and it is the row the application's directories are stored + under. **One identifier per application, everywhere.** +3. **`<name>-app-settings.js`** if it has anything to configure. Do not import the + settings page — that is the cycle in §9.1. +4. **Add the key to the node's allow-list.** +5. **i18n:** at minimum a tab label key **in all ten catalogues**. A settings key + added to the client must be added ten times; write the table and generate the + insert. +6. **Add both files to the asset fingerprint list.** A module reached through the + registry is not imported by name anywhere, so nothing else would notice it + changing and a browser would keep serving the cached copy. +7. **Add the file to the test file-set lists** — hook ordering, sticky headers, + transport contracts. A file missing from those lists is never checked, which + fails silently rather than loudly. +8. **Toolbars pin.** A toolbar is a direct child of the page root and is opaque, or + content scrolls visibly through it; if anything pins below it, it must publish + its own height, which is never a constant because it wraps on a phone. An + application with no toolbar renders none — an empty band still holds a strip of + the page open. + +No protocol change, no hub change, no daemon change. Steps 4 and 7 are the only +node-side and test-side touches, and both are allow-lists. + +**A reference application exists in the tree behind a development flag.** Every +other test of this architecture reads source for the *absence* of application names, +which proves nobody wrote a special case — not that a new application works. +Writing a real one immediately found two places the claim was only nearly true. + +### 9.5 Files + +The explorer: roots, folders, sorting, selection, upload where the current root is +writable and available, download, folder-as-zip, and the eject/plug control beside +each removable root. Entries from unavailable roots are filtered out. + +### 9.6 Chat + +Messages, threads, attachments, and link previews (§6.5). The composer gates on the +connection having identified a device (§3.3) and on a writable root existing for +attachments; the attachment directory is a single writable folder chosen by the +operator, and the paperclip is disabled with a stated reason when it is not usable. + +Own-ness is decided from the **account id**, in one place, never by comparing +display names — and the optimistic local echo carries an explicit flag rather than +inventing an identity for itself. + +### 9.7 Videos + +A poster browser over the video files in the group's configured folders. + +**Two modes**, both driven by the index: a **poster grid** with third-party +metadata and artwork, falling back to a thumbnail card with the cleaned filename +when the lookup returns nothing or a low-confidence match; and a **flat, +folder-driven list** with no third-party dependency, which keeps working with the +service switched off node-wide. The mode toggle is a **per-device display +preference** in local storage — it has no authority implication, so it needs none +of the signed-op treatment. + +**Scoping to folders is not a display preference**, because it decides what *every* +member's tab shows. It is a per-group setting changed by a signed op, broadcast to +connected members, and **validated before a signature is requested**: a candidate +path is resolved against the group's real root set and must name a real, readable +directory, so a stale path never reaches the operator's browser as a signing prompt. + +**Grouping is by unit, not by file**: one card per film, one per show — expanding to +seasons and episodes. Seasons carry their own text where the service supplies it, +falling back to the show's. An operator can **correct a wrong automatic match**, and +correcting one applies to the unit rather than to a single file (for shows) or to +the one file (for films), because those are the units each actually is. + +The filename parser is a fallback, and directory context is what bare-filename +parsing cannot supply. Matching is a **scored ladder** rather than the first +candidate to clear a threshold, with a year-exact rescue for a weak top hit. + +**Thumbnails, probes and metadata are node-side** (§6.5). Index-time probing runs in +its own bounded pool after a file is first seen: the file appears in the index +immediately with size and hash, and an index delta fills in the technical and parsed +fields once ready. **No scan is blocked waiting for enrichment.** + +### 9.8 Music + +An album browser and a player over the audio files in the group's configured +folders. + +**Metadata mostly already exists in the files themselves**, which is the real +difference from Videos. The order of trust is embedded tags, then filename and +folder parsing for what tags do not supply, then a third-party lookup for canonical +spelling, a missing field, or cover art where none is embedded — node-side and +cached like any other enrichment, and **needing no credential**, unlike Videos. + +**No playback protocol change is needed at all.** A track is a few megabytes, so +playback reuses the ordinary download-and-decrypt pipeline and hands a blob to an +audio element. No streaming request, no transcode pool, no stream slot, nothing +added to the node's streaming machinery. The one exception is narrow and +extension-gated: two container formats tag perfectly and decode in no mainstream +engine, so for those the node performs a **one-shot whole-file conversion**, cached +under its own content hash and served through the ordinary chunk path. + +**The player is persistent across tabs**, at shell level: closing the tab must not +stop the music. The next queued track is prefetched while one plays — client-side +only, a small in-memory cache evicted as the queue moves. + +Music is scoped to folders on the same mechanism as Videos. The reason is not cost — +tag reads are cheap — it is **mixing**: a shared tree with more than one kind of +thing under it puts everything into one undifferentiated view with no way to narrow +it. + +### 9.9 Photos + +An album browser over the image files in the group's configured folders. It is +**smaller** than Videos and Music, deliberately, in three ways: + +- **Several root folders rather than one.** +- **One album-grid view, no mode toggle**, because there is nothing to fall back + from. +- **No third-party service at all** — there is nothing to match a photo *to*. It + already is what it is, per its own folder and filename. + +An **album is a directory**, exactly as a season is a folder in Videos. Thumbnails +are node-side, orientation-corrected, and delivered through the same chunk path. + +**EXIF is read locally on the node** and narrowed on purpose: a capture time and a +camera, and **never GPS**, anywhere, in any field a client receives. The claim this +supports is precise, and the one it must not make matters more: + +> GPS **is** in the file, for most phone photos, in the original bytes any member +> with file access can already download. What this design controls is what the +> *application* computes and surfaces — not what the underlying file contains. + +### 9.10 Playlists — designed, not built + +Playlists are the first feature to need **per-account state that spans several +groups on several nodes**, so the shape is settled here even though the code is +not written. Full design: `playlists.md`. + +A playlist belongs to **one account and is never shared with other members.** That +scope is what keeps the merge problem small. + +**Where the state lives.** Not the hub — and the operative rule is narrower than +"the hub stores nothing about a user", because it already holds small per-account +preferences behind an allowlist. The rule is: + +> **No content metadata on the hub.** + +A playlist is literally a list of content hashes of private-group files, plus the +titles needed to render while nodes are offline. That is the exact object **H7** +removed from the hub, and it is the same rule that keeps resume positions local: +*nothing new learns what you watch.* An encrypted blob on the hub is technically +trivial and is still refused. + +It lives on the node instead, as **an opaque per-account blob in `bundles.db`** — +the same shape as the keypair bundle, which the node already stores and cannot +read. **No new trust boundary**: the node is not asked to hold a kind of thing it +does not already hold for that same account. + +**The key is the one thing that must not be got wrong.** Identity keys are per node +(§3.2), so a blob encrypted under one is unreadable from every other node — the +precise opposite of the requirement. The only secret an account holds *everywhere* +is the bundle key, so `playlist_key = HKDF(bundle_key, info = +"meshbay:playlists:v1")`: one derivation at sign-in, two handles, no second +Argon2 run, and a purpose-separated subkey rather than the bundle key reused with a +different AAD (§4.4's rule). The nonce is 96 random bits and never a counter, for +exactly the reason chat's is (§4.5): two devices of one account derive the *same* +key, which is the point. + +**Merge is the hard third, and the granularity is what makes it tractable.** The +unit is **one playlist, not the collection**; revision counters order writes, never +the wall clock; and **a deletion is a tombstone, never an absence** — an absence is +indistinguishable from a device that has not seen the addition yet. A node that is +offline for a month therefore cannot corrupt anything: it holds an older revision of +some playlists and is overwritten per playlist, not wholesale. + +**It adds no dialing and no new streaming path.** Sync rides connections the client +already makes, and playback is unchanged (§9.8). + +### 9.11 Cross-group search and source merging + +The Search page mounts the same media applications across every group the reader +belongs to. The difference lives entirely on the entries, in fields the group page +never sets: which group serves an entry, that group's transport and key, a +connection generation to use as a refetch key, and the merged source list. + +**A file shared by two groups is one entry, not two.** Identity is the content +hash: two entries with the same id are the same file, whatever group announced them +and whatever their path. Within one group this cannot arise, because the index is +already keyed by hash; the duplication is created by concatenating N independently +keyed indexes, and by nothing else. + +**One source is chosen per logical unit** — a film, a show, an album, a photo album — +not per file, so a show's episodes never stream from two different nodes. A group +hosted by the local node wins; otherwise the pick is deterministic and +pseudo-random, stable for one reader and spread across readers. If the chosen source +is unreachable the unit fails over. **Which source was picked is never shown**; the +badge names the group when there is one source and counts them when there are more. + +Two rules for a new application here: + +- **Use the shared source tag rather than an entry's group name** — a merged entry + has several groups. Pass it the whole unit, not the entry the card was drawn from, + which is usually chosen for its thumbnail and would under-report. +- **Never re-derive unit keys.** Call the application's own exported grouping + function. A copy keeps agreeing until one of them changes, and the symptom is a + show whose episodes stream from two different nodes. + +**The Files explorer is deliberately not merged**, and not "mostly not": there each +group is a top-level folder, and merging would remove a file from one of them. A +test refuses a build that changes this. + +One consequence is deliberate and is not a bug: albums are keyed by directory, so +two groups whose roots have *different* basenames put the same photo into two +differently-named albums, and the merge — scoped to a unit — leaves it in both. They +are two albums. + +--- + +## 10. Filesystem portability + +**exFAT and NTFS on Windows are the common case, not an edge case.** Most users are +expected to share from an external drive. The consequences below are correctness +requirements, not compatibility notes. + +| Property | What has to be true | +|---|---| +| **Case-insensitive, case-preserving** | The index needs a canonical identity and a **case-folding collision check** at scan time, reported to the operator rather than resolved silently. One directory indexed as two roots is the same problem one level up | +| **Unicode normalization** | A name written on one platform in decomposed form and on another in composed form are different byte strings. **Normalize to NFC for identity and comparison; preserve the original bytes for display and for opening the file** | +| **Reserved names and characters** | A group indexed on one platform can hold names another cannot create. The client sanitises on save **and says so**; the upload allowlist is the intersection across platforms, or some files are simply undownloadable | +| **Path length limits** | Use extended-length paths on Windows, in the node and the client alike | +| **Coarse timestamps, local time** | mtime alone is not a change detector. Size plus mtime with tolerance, and rehash when in doubt | +| **Watcher reliability** | Change notification drops events under load on Windows, and inotify is unreliable on a FUSE mount. **Periodic reconciliation is mandatory on both platforms** | +| **No symlinks, no POSIX permissions** | Simplifications: nothing to defend against, and the node runs as the user anyway | + +**Case folding is for comparisons the code makes itself** — index identity, +collision reporting, root names, nesting checks. It is *not* needed for the +no-overwrite rule, where the filesystem's own case-insensitive `stat()` already +gives a colliding upload a free name. + +**A volume that disappears freezes its root's subtree** and never empties it +(§6.2). Emptying propagates deletions for a whole library as though the owner had +erased it. + +**Never assume POSIX, systemd, or case sensitivity.** One platform-specific trap +with no counterpart elsewhere: a service unit with filesystem-protection options +gets its own mount namespace, so a volume mounted on the host *after* the service +started is invisible inside it — the directory reads as empty with everything else +configured correctly. + +--- + +## 11. Platforms + +### 11.1 NAT traversal, measured + +**QUIC native path** — residential ISP A to a hosted VPS: port-restricted cone NAT, +direct connection established from the server socket's own probe. + +**WebRTC browser path** — mobile and laptop clients to nodes behind two different +residential ISPs: + +| Path | Result | +|---|---| +| LAN, IPv6 direct | OK, ~100 ms | +| Mobile data, IPv6 inter-network | OK, ~600 ms | +| Mobile data, IPv4 only, STUN hole-punch | OK, ~650 ms | +| Laptop → second ISP's node, IPv6 inter-network | OK, ~7 s | +| Laptop → second ISP's node, IPv4 only, STUN hole-punch | OK, ~6.9 s | + +**Two ISPs validated, both without TURN.** The hub relays under a kilobyte of +signaling; the data path is peer to peer. + +### 11.2 Windows + +The bulk of the codebase is portable because the portability rules in §10 were +treated as correctness from the start. What the port needed is registered as +**W1–W9** (§13.7) and is done; packaging is built and awaits a clean-machine run. + +Two Windows-specific design points worth stating here: + +- **Autostart has two modes, chosen at install and switchable afterwards** from the + Node page: a per-user startup launcher (the default) and a scheduled-task service + mode. A per-user default is right for the desktop persona; a service is what a + machine that must serve while nobody is logged in needs. +- **A user service unit cannot carry a system unit's user directive.** Two unit + templates exist, held apart by a test that parses directives rather than + searching the file — searching matched the *comment* explaining why the directive + is absent. + +The hub stays Linux. macOS is not planned. Windows on ARM and store packaging are +out of scope. + +### 11.3 Android + +A client, not a host. The platform is hostile to *hosting* a node — background +execution, storage, battery — and fine as a *client*, which is one of the reasons +enrichment is node-side (§6.5). + +### 11.4 Casting + +An HTTP relay in the desktop client serves a standard fragmented-MP4 stream that +any LAN renderer can play; the relay is device-agnostic. Chromecast discovery and +control ship. DLNA/UPnP is designed and not built: it is a second device backend +beside the first, not a second relay. + +--- + +## 12. Testing posture + +**Security tests are negative assertions** — "this attack does not work" — and are +verified to fail against the pre-fix source before being trusted. A suite that +tests only that features work will happily **pin a vulnerability in place as +expected behaviour**, and a refactor that accidentally fixed one would be reported +as a regression. That has happened here, to four findings at once. + +Rules that follow from what has actually escaped this suite: + +- **A syntax check validates names not at all**, and a module syntax check must + force the module parser or it accepts template syntax pasted into an object + literal and reports success. +- **A second implementation of the client proves the protocol and nothing about the + client.** An end-to-end harness written in the right order by construction cannot + see ordering or lifecycle faults. Source-reading tests are weak evidence and are + sometimes the only evidence available; prefer ones that **re-derive** a value from + the source over ones that restate it. +- **A test that models a fix agrees with it by construction.** Lift the real + functions out of the source *as text* and execute them; model the environment, + never the code under test. +- **A test that reads source and inspects "the first" occurrence of a call stops + guarding anything the moment a new call is inserted before it** — and keeps + passing. +- **Measure, do not read.** A stylesheet does not tell you where anything lands; a + browser measuring the real stylesheet does. Assert on geometry, and check the test + fails with the fix removed. +- **A fixture narrower than real data tests the fixture.** Names are the one thing a + file browser cannot be given short. +- **Pace a probe like the real thing.** A stress probe fast enough to finish inside a + timeout hides every time-based fault. +- **"It works now" is not evidence against a race.** Force the worst case. +- **Exercise every browser's branch of a shared path before shipping a fix for one + of them**, and finish with a live pass: launching the real thing finds what source + reading cannot. +- **A passing suite over code nothing calls is evidence about that code, never + about the product.** Two full protocol implementations sat green and unreferenced + for months (§13.3 **L7**). Before trusting a suite, check that production imports + what it tests. +- **A comment that contradicts the constant beside it is worse than no comment** — + one of them is wrong and the reader cannot tell which. The same holds for a + document: a number restated away from its definition is a number that will drift. + +--- + +## 13. Register of labelled findings and decisions + +Each entry states **the rule the label names today**. Where the label originally +named a defect, the subject is given in one clause so a code comment citing it can +be understood, not so the incident can be retold. + +### 13.1 First review (design review) + +| Label | The rule it names | +|---|---| +| **C1** | Group chat has **no shared mutable sending state**. A pairwise ratchet shared across a group produces key and nonce reuse; the design that shipped has no sending state at all, which is stronger than partitioning it per device (§4.5) | +| **C2** | Tokens carry a `groups` claim and **the node verifies membership before serving content** (§5.2) | +| **S1** | Every admin endpoint has an authorization check | +| **S2** | Email addresses are **encrypted at rest**, with a blind index for lookup | +| **S3** | Token revocation reaches nodes over the hub socket, and the denylist persists (§7.5) | +| **S4** | AEAD nonces are **96-bit**, per NIST SP 800-38D | +| **S5** | Refresh tokens **rotate, one-time-use**, with family-based reuse detection | +| **M1** *(first review)* | Account enumeration is a known, bounded property of the account-management endpoints; `login`, device auth and reset-request are uniform | +| **M2** *(first review)* | Node TLS certificates are transport confidentiality only; identity is the Ed25519 key checked at the MNP layer | +| **M3** *(first review)* | Rate limits on key-adjacent endpoints | +| **M4** *(first review)* | Delegation is designed and deferred; the role check is written so it drops in (§3.4) | +| **M5** | **Chunk key derivation uses `info`, with `salt=None`** — correct HKDF usage, because the group key is already uniform CSPRNG output (§4.3) | +| **M6** *(first review)* | Argon2id production parameters are applied, and recorded per envelope so they can be raised (§4.6) | +| **N1–N5** | Notes, no action: the forward-secrecy model is appropriate to the deployment; login error messages are correct; the hub's legal exposure model is well-positioned; the NAT probe payload is fine; the web/CLI derivation difference is by design | + +### 13.2 Node sovereignty + +| Label | The rule it names | +|---|---| +| **NS1** | The client proves possession of the group key in the handshake, and the node verifies it | +| **NS2** | Admin operations are **Ed25519 challenge-response over a structured transcript**, never a token (§5.4) | +| **NS3** | **The node never serves the group key in plaintext.** There is no request message for it; the constants are gone from the protocol | +| **NS4** | **Operator authority comes from the node's roster and from nowhere else.** No auto-pin from the keystore, no resolution through the hub, no config key — a config naming one is warned about and never obeyed (§3.4, §6.1) | +| **NS5** | The proof is **bound to the transport channel** (DTLS fingerprints / certificate hash), so a signaling relay that substitutes its own cannot produce it (§5.2) | +| **NS6** | **`sender_id` is enforced from the authenticated session, never the wire.** It is what the store keys on; it is not what authenticates a message — the device signature is (§4.5) | +| **NS7** | The node authenticates to the hub with **Ed25519 and no password**, and its token's scope is refused for group management (§7.2) | +| **NS8** | **The node refuses connections when it holds no group key.** There is no bypass switch | + +### 13.3 Second review (code review) — the default numbering + +**Critical** + +| Label | The rule it names | +|---|---| +| **C1** | **All content travels over the authenticated protocol.** The node exposes no unauthenticated HTTP surface; the per-group file API that served private indexes and plaintext files on all interfaces was deleted rather than repaired, because it duplicated MNP without any of its controls | +| **C2** | A node's signaling identity is **resolved against the database and derived from it**, never taken from the client's first message (§7.2) | +| **C3** | **Authentication is mutual**: the node proves key possession over the client's nonce and signs the transcript, and the client verifies both and pins the key (§5.2) | +| **C4** | **Keypair bundles are per node, Argon2id-protected, and closed for native devices** — and **open for any account that also uses a browser** (§3.7) | +| **C5** | The two halves below, cited together where a comment means "a member must not be able to write what the node then trusts" | +| **C5a** | Uploads cannot overwrite, are allowlisted, ordered and capped, and ownership is signed by the uploader (§6.4) | +| **C5b** | **No key material arrives from outside.** The member-supplied bundle message does not exist; the node generates every copy of a group key itself (§4.2) | +| **C6** | **One handshake implementation, shared by every transport.** A transport that accepts a bare token is the standing example of what an opt-in compatibility branch costs (§5.2, §5.6) | + +**High** + +| Label | The rule it names | +|---|---| +| **H1** | **Per-group isolation on a multi-group node**: the chat store, the peer registry and the broadcast set are per group | +| **H2** | Every value that originates outside the node — filenames chosen by members, usernames originating at the hub — is escaped where it is rendered. A CSP contains exfiltration but cannot prevent injected inline script, so escaping is the actual fix | +| **H3** | **No public key is ever fetched from a directory to wrap a group key for.** The node wraps for a key the recipient proved possession of, bound to an account by a code the hub never sees (§3.4) | +| **H4** | Revocation reaches nodes, drops live sessions, and **persists across a restart** (§7.5) | +| **H5** | An admin challenge is a **structured, domain-separated transcript naming the operation and subject**, and the client refuses to sign anything that is not what the user asked for (§5.4) | +| **H6** | Unauthenticated work a node will do is bounded: a small pre-handshake buffer, a transcode semaphore, per-user pending-offer caps, and a membership check on signaling (§7.2) | +| **H7** | **Only public groups register content hashes with the hub.** Private groups register nothing, and the swarm route requires authentication | + +**Medium** + +| Label | The rule it names | +|---|---| +| **M1** | `group_id` is **mandatory** on the handshake — there is no fallback to the node's first group (§5.2) | +| **M2** | The node keystore uses the full Argon2id parameters, recorded per envelope (§4.6) | +| **M3** | Operator authority is the roster pin, established locally by pairing. **Asking the hub for the operator's key — the obvious-looking fix — would let the hub install itself as node administrator** (§3.4) | +| **M4** | **Every reply carries the request id it answers.** Arrival-order matching is a guess that fails silently and asymmetrically (§5.3) | +| **M5** | The index has one protection level, not one per transport (§4.4) | +| **M6** | Audit rows are attributed to the row's own subject, never backfilled across rows | +| **M7** | Client addresses are taken from a trusted proxy's rightmost hop, in one helper (§7.2) | +| **M8** | A node announcement requires **proof of possession** of the key it announces | +| **M9** | Node-scoped tokens are refused on the client path (§5.2) | + +**Low** + +| Label | The rule it names | +|---|---| +| **L1** | The wire contract carries no constants for messages that do not exist | +| **L2** | MNP negotiates versions explicitly and refuses with a code (§5.6) | +| **L3** | Errors returned to a peer name no filesystem path and no exception detail | +| **L4** | Transcripts are **length-prefixed**, and an empty channel binding **raises** rather than degrading the proof to nonce-only (§5.2) | +| **L5** | The hub serves a CSP and security headers on the application it serves | +| **L6** | Registration validates the email field it declares | +| **L7** | **A module nothing imports is not a protection**, and it is not kept. The sender-key and ratchet implementations this finding named were unreferenced for months and are now deleted; a green test suite over uncalled code is evidence about that code, never about the product (§4.5) | +| **L8** | An uploader record identifies a file by its id, not by a name at a root | + +### 13.4 Third review + +| Label | The rule it names | +|---|---| +| **H1** *(third review)* | **Moderator is not administrator.** The user-patch handler is split by field so a moderator cannot write `role` (§7.5) | +| **H2** *(third review)* | Content reporting requires authentication, distinct reporters and a rate limit, and is refused when public groups are off (§7.5) | +| **M1** *(third review)* | The registration CAPTCHA gate is **unconditional** when a captcha is configured — never conditioned on a field the real client always sends (§7.7) | +| **M2a** | `sender_id` comes from the authenticated session (**NS6**). Now guaranteed by there being **one** chat implementation: QUIC does not carry chat at all (§5.1) | +| **M2b** | Chat broadcast is per group (**H1**), on the one transport that carries chat | +| **M2c** | No transport runs a synchronous media process on the event loop, and every one is capped | +| **M3** *(third review)* | Link-preview SSRF is gated: rate limit, port allowlist, globally-routable check, per-hop re-check, connect-address re-check, size guard (§6.5) | +| **M4** *(third review)* | Federation binds a pushed row's source to the signer, checks the token audience, caps the push, rejects replays, and scopes revocation to the peer's own entries (§7.6) | +| **M5** *(third review)* | A CSP and security headers apply to the hub-served application, verified against the running app — a mis-tuned CSP shows as a blank page | +| **M6** *(third review)* | **Withdrawn.** It misread the node registering a hub membership during the CLI invite flow — which is deliberate — as authorization drift | +| **L1–L11** *(third review)* | Opportunistic hardening: relay-registry proof of possession; delete orphaned modules rather than leaving them to be rewired; decide and document account enumeration; an aggregate upload quota; header-only control-API tokens; a freshness bound on revocation replay; state that the exact-hash content check is structural; validate group-name length and charset; require `exp` and bind an audience on token decode; key the rate limiter through the same client-address helper as everything else; keep diagnostic logging truncated | + +Two structural recommendations from that review stand as rules: + +- **Make transport parity a test, not a habit.** Shared helpers in common code with + a test that fails if a transport calls a chat or stream path that bypasses them. +- **Every new outbound or cross-trust surface needs a rate limit and a named + adversary in the same commit** (§6.5). + +### 13.5 Standing trust limits + +| Label | The rule it names | +|---|---| +| **T1** | **The password split.** The hub never sees a passphrase; it holds a verifier for a client-derived `auth_key`. The passphrase floor can therefore only be enforced client-side (§3.1) | +| **T2** | The hub was the key directory. **Closed** by admission redesign, not by safety numbers: the invite path reads no directory at all (§3.4). Reclassified as **H3** | +| **T3** | **The hub serves the SPA. Accepted permanently for browser users.** It is the only remaining way an active hub reads content, it is an artifact-level attack rather than a silent lie, and it does not exist for a native client — whose value is realised by reproducible builds, not by packaging (§2.3, §8.2) | + +### 13.6 Chat design findings + +| Label | The rule it names | +|---|---| +| **F1** | **A sender's signing key is bound to the roster**, never generated fresh inside a distribution any member can produce. Otherwise every member can replace another's chain and forge them silently | +| **F2** | A second device cannot destroy the first's ability to send or be read — there is no per-device chain to drop (§4.5) | +| **F3** | There is no skipped-key cache to grow without bound | +| **F4** | **Rotating the group key is a re-wrap, not the destruction of the archive.** Epoch keys are wrapped at delivery, never at rest (§4.5) | +| **F5** | **History is served to devices that were not present**, which is what "load older" means and is incompatible with a ratchet | +| **F6** | Ciphertext is not corrupted on the history path | +| **F7** | One account can hold two connected devices: the peer registry is keyed per connection, and a broadcast excludes the sending **session**, not the sending account | +| **R1–R22** | The regression register for that work — each row a concrete way the feature could break something that already worked, with the guard that stops it. R1 (rotation keeps history readable), R5 (an older client is refused with a stated reason, not left showing gibberish), R11 (own-ness from the account id), R15 (no epoch key in a plaintext store), R16 (retention deletes messages, never epoch keys) are the ones cited elsewhere | + +### 13.7 Windows port + +| Label | The rule it names | +|---|---| +| **W1** | Platform directories: no hardcoded XDG paths | +| **W2** | Signal handling is platform-guarded | +| **W3** | Daemon lifecycle: a per-user startup launcher by default, a scheduled-task service mode offered, switchable after install (§11.2) | +| **W4** | Packaging: one per-user installer carrying client and node, with media tools bundled | +| **W5** | File permission calls are skipped where they have no meaning | +| **W6** | Media-tool discovery fails at startup with a stated reason rather than at first use | +| **W7** | CLI messages name the right paths and commands for the platform | +| **W8** | The test suite is green on the platform, encoding included | +| **W9** | ICE interface matching works where an adapter's name is a GUID rather than a kernel name, and **fails open** | + +### 13.8 Desktop client decisions + +| Label | The decision | +|---|---| +| **E1** | **Electron**, plus an optional Python sidecar for hub-less `group://` over QUIC | +| **E2** | **Device linking**: an already-pinned key countersigns, bound by a one-time code the new device generates (§3.3) | +| **E3** | **Hybrid account creation**: register with a passphrase-derived `auth_key` — the recovery path — then authenticate day to day with a device key | +| **E4** | **Signed admin ops over MNP** for everything group-scoped; first run stays local; the loopback API is never exposed to the network | +| **E5** | LAN enrolment of a headless node is **out of scope for v1**, kept implementable | +| **E6** | **The browser SPA stays.** A native client must not prevent web use | +| **E7** | **Several named roots** per group, forming one virtual root (§6.2) | +| **E8** | **exFAT/NTFS and Windows are the common case.** Linux ships first; that is build order, not population (§10) | +| **E9** | **Group-related server state lives on the node. Always** (§1.3) | +| **O1** | Initial key setup in the pre-proof window — deferred; that window is where C4 and C5b came from | +| **O2** | A LAN enrolment door — one endpoint, bounded window, one-time code, closing permanently on success | +| **O3** | `device_policy {allow_bundle: false}`, signed by a pinned key — **the mechanism that actually closes C4** (§3.7) | +| **O4** | Isolating the node-admin panel from the process holding user keys | +| **O5** | An unlock key in the environment, for the **node** | +| **O6** | The engine version floor, verified rather than assumed | +| **O8** | A minimum client version in the hub version endpoint — **done** (§5.6) | +| **O10** | Canonical file identity across filesystems, defined once and shared (§10) | +| **O11** | A root **alias** where the basename cannot be used (§6.2) | +| **O12** | Derived data: **revised** — the node caches durably in its own `data_dir` (§6.5) | +| **O13** | **Hub identity pinning.** The client points at a hub by URL and nothing pins that hub's identity. Bounded, because a substituted hub can neither read content nor ship the code to a native client — worth doing all the same | +| **V1–V13**, **P1–P5** | Per-application open items: wording of a disabled-service state, whether artwork reuses the chunk path, cache TTL, multi-track surfacing, HEIC/RAW support, a fuller EXIF panel, lightbox preloading, album-boundary behaviour, cover selection | + +--- + +## 14. Decisions that are not revisited + +### 14.1 Structural + +1. Multi-group on a single port. +2. Signaling punch and connect through the hub socket. +3. Chat is a core feature, not a module. +4. **Chat encryption is a sealed archive with per-device keys and signed messages** — not sender keys, not a ratchet (§4.5). +5. Tokens carry group claims and the node checks them. +6. Admin roles are hub configuration. +7. Refresh-token rotation is family-based. +8. Email is encrypted at rest. +9. Argon2id parameters are versioned and migrate on login. +10. **Browser transport is WebRTC DataChannel with ICE/STUN.** +11. **The hub is a registrar and signaling relay, never in the data path.** +12. **Chat is stored on nodes, never on the hub.** +13. The web UI is a small-framework SPA: dark/light, responsive, i18n in ten languages. +14. Site-specific pages are an overlay, separate from the generic hub. +15. Video streaming is fragmented-MP4 remux on the node, source buffer in the client. +16. **ICE is primary for browser and native alike. QUIC is kept at parity for LAN, port-forwarded and hub-less access. TCP+TLS and the node HTTP API do not exist.** +17. **`punch_nat()` is a direct-connection helper, not a traversal stack** (§5.1). +18. **The desktop shell is Electron.** What is unchanged and non-negotiable: **UI assets ship inside the package and load from disk** (§8.2). +19. **A second device is admitted by device linking, not by an operator code** (§3.3). +20. **Private keys never leave the device on native clients.** Qualified: a browser has no durable storage of its own and still needs a bundle on each node, so C4 closes for an *account* only when it opts out of browser use. +21. **Hub minimisation is enforced by an acceptance test, not by policy.** The hub must be *unable* to see keys, content or file listings. +22. **No new code exchanges between people.** Safety numbers are refused for identity verification, permanently. The device-linking code is between a person's own devices and is unaffected. The total user-visible cost of the whole authorship story is **one notice**: *"this account's key changed"*. + +### 14.2 Client architecture + +| # | Decision | +|---|---| +| **D1** | **The hub keeps serving the web UI.** It is the zero-install path and it stays. What must then be true: a strict CSP, a signed digest of the served bundle that any third party can verify, an explicit reduced-trust notice, and docs that never claim end-to-end integrity for that path | +| **D2** | **A native client, offered alongside the SPA** — not as a replacement. The browser-extension options (an extension that *verifies* the served bundle; an extension that *ships* the UI) were analysed and are not taken up | +| **D3** | Transport: ICE primary, QUIC at parity, TCP and HTTP removed | +| **D4** | **Hub minimisation deferred, and may be dropped.** The hub stays in the trusted path by choice | + +**The reason a native client is justified, and the reason it is not justified.** It +is justified on *product* grounds: durable keys, no browser tab, background +connectivity, better video, hub-less access. It is **not** the fix for T3 unless +reproducible builds ship with it — a binary from the same operator relocates trust +rather than removing it. What genuinely changes is **detectability**: a browser +attack is one HTTP response aimed at one user, leaving no artifact; a native attack +requires shipping a build, which is hashable, archivable and comparable. + +Its real costs were under-weighted once and are recorded: **patch velocity** is +owned rather than inherited from a browser vendor, and the renderer parses +attacker-controlled content from nodes. Accepted deliberately. + +**For node operators specifically, the CLI beats every client.** The operator holds +the group key and is the content authority; a CLI removes their dependency on +hub-served code at a fraction of any client's cost. If only one thing were built +against T3, it should be that — and it was. + +--- + +## 15. State of the build + +### 15.1 Built and running + +The hub, the node daemon, both transports, the unified handshake, admission and +pairing, device linking with member-visible evidence, per-node identity, named roots +with RO/RW and eject/plug, the indexer with partial hashing, uploads, the sealed +index and sealed upload path, encrypted chat with epochs, video streaming with +seeking, transfer leases with queueing, pause and resume, the group-application +framework with Chat, Files, Videos, +Music and Photos, cross-group search with source merging, the operator CLI and +loopback control API, the desktop client through its identity and download stages, +account recovery, and the Windows port through packaging. + +### 15.2 Not built + +| | | +|---|---| +| **Stage D5** | Node management panel over the operator ops, root selection included | +| **Stage D6** | First-run wizard — detect, enable the unit, link, group, initialise, pair | +| **Stage D9** | Python sidecar — `group://` over QUIC | +| **Stage D11** | Windows clean-machine install verification and code signing | +| **Stage D12** | Release key, signed repositories, updates through the OS | +| — | DLNA/UPnP casting (§11.4) | +| — | Delegation (§3.4) | +| — | Tier 3 roster attestation (§3.3) | +| — | Playlists (§9.10) | +| — | Android client | + +### 15.3 Open, and why each is where it is + +| Item | Status | +|---|---| +| **C4** for browser-using accounts | Open until the signed bundle opt-out ships (O3) | +| **T3** for browser users | **Accepted permanently.** Removed for native clients, and that removal's value depends on reproducible builds | +| **Hub identity pinning** (O13) | Nothing pins the hub's key. Bounded, because a substituted hub can neither read content nor ship code to a native client | +| **Aggregate upload quota** | Per-file caps exist; a per-user or per-group total does not | +| Forward secrecy in group chat | **Given up deliberately and on the record** (§4.5). If it becomes a requirement it belongs in 1:1 DM | +| Metadata at the hub | Membership, and who posted in which group and when. A known leak, not a solved problem (§7.1) | +| The exact-hash content check | Structural, not functional (§7.5) | + +--- + +## 16. Concordance + +Code comments, tests and older documents cite sections of the documents this one +replaces. The mapping: + +| Cited as | Read | +|---|---| +| `draft-v5 §2`, `draft-v6 §4` — security claims | §2.2 | +| `draft-v5 §3` — transport, NAT traversal | §5.1 | +| `draft-v5 §4`, §4.1–4.4 — handshake, transcript, channel binding, mutual auth | §5.2 | +| `draft-v5 §5.1`, `draft-v6 §2.3`, `§2.4b` — privileged operations, key activation, authorship | §5.4 | +| `draft-v5 §5.2`, `draft-v6 §2.1`, `§2.1b` — uploads | §6.4 | +| `draft-v5 §5.2` — nothing derived beside the originals | §6.5 | +| `draft-v5 §5.2b` — removing a directory | §6.4 | +| `draft-v5 §5.3`, `§5.4` — operator interface, local admin UI | §6.7 | +| `draft-v5 §5.5`, `invite-pairing-v1.md` §3, §5, §7 — admission and key delivery | §3.4 | +| `draft-v5 §6.1`, `draft-v6 §2.5`, `§2.9` — hub role, group names | §7.1, §7.3 | +| `draft-v5 §6.2`–`§6.4` — node registration, signaling, client addresses | §7.2 | +| `draft-v5 §7` — cryptography | §4.1–§4.4, §4.6 | +| `draft-v5 §7.1` — the keypair bundle and C4 | §3.7 | +| `draft-v5 §8.1`, `§8.2`, `desktop-client-v1.md §2` — the two clients, the shell | §8.1, §8.2 | +| `draft-v5 §9`, `draft-v6 §5` — open items | §15.3 | +| `draft-v5 §10` — testing posture | §12 | +| `draft-v6 §2.2`, `desktop-client-v1.md §4`, `§4.5`, `§4.6` — device linking | §3.3 | +| `desktop-client-v1.md §4.8` — authorship, the tiers | §3.3, §5.4 | +| `desktop-client-v1.md §5`, `§5.1`, `auth-confirm.md` — accounts, change and recovery | §3.1, §3.6 | +| `desktop-client-v1.md §6.7`, `refactor-groups.md §1.1`, `§1.5b` — roots, RO/RW, eject | §6.2 | +| `desktop-client-v1.md §6.8`, `draft-v6 §3`, `WINDOWS-PORT.md` — portability | §10 | +| `desktop-client-v1.md §6.9` — a root that goes away | §6.2 | +| `desktop-client-v1.md §6.10`, `mediacenter.md §2`, `§5.2`, `§5.3` — views not a catalogue, derived data | §6.5, §9.1 | +| `desktop-client-v1.md §8.1` — URL space | §8.4 | +| `desktop-client-v1.md §10b`, `refactor-node-ui.md` — group settings, the node's admin surface | §9.3, §6.7 | +| `draft-v6 §2.7`, `apps.md §1`–`§3` — the application framework | §9.1–§9.3 | +| `apps.md §2b`, `refactoring-search.md` — cross-group search, source merging | §9.11 | +| `playlists.md` — per-account cross-group state | §9.10 | +| `apps.md §4`, `refactor-groups.md §3`, `§4.1` — adding an application | §9.4 | +| `draft-v6 §2.8` — instance policy, suspend vs revoke | §7.4, §7.5 | +| `draft-v6 §2.10`, `refactor-groups.md §1.7` — link previews | §6.5, §9.6 | +| `draft-v6 §2.11` — node settings | §6.8 | +| `draft-v6 §2.12` — STUN and ICE filtering | §5.1 | +| `mediacenter.md §3`, `§4`, `§5.4`–`§5.7` — Videos | §9.7 | +| `musicbay.md §2`, `§3`, `§4`, `§5`, `§6` — Music | §9.8 | +| `photos.md §2`, `§3`, `§4`, `§5` — Photos | §9.9 | +| `refactor-groups.md §1.6` — generic app-directory ops | §9.3 | +| `chat-sender-keys.md §5`, `§6`, `§8`, `§13` — chat encryption, sender authentication | §4.5, §3.3 | +| `indexing-v2.md` — partial-read hashing | §6.3 | +| `per-node-identity-v1.md` — identity keys per node | §3.2 | +| `captcha.md` — registration gate | §7.7 | +| `tmp-decisions.md` D1–D4 | §14.2 | +| `devel-phases-next.md` structural decisions | §14.1 | +| `cast-smart-tv.md` | §11.4 | + +Older references to `draft-v3` and `draft-v4` sections point into `old-draft.md`, +which is a historical archive and is not authoritative. diff --git a/docs/transfers-v1.md b/docs/transfers-v1.md new file mode 100644 index 0000000..f99424f --- /dev/null +++ b/docs/transfers-v1.md @@ -0,0 +1,1159 @@ +# Transfer slots, queueing, pause and resume — build record + +> **Status: built.** Leases, caps, queueing, the settings surface, the client +> widget, pause and resume all shipped; the compulsory-lease flag day is +> **MNP 3.0**. +> +> **The design is `MESHBAY_DESIGN.md` §5.5**, which states the lease model, the +> caps, and the reasoning behind the browsing exemption. Read that first. This +> document is kept for what a synthesis cannot carry: the failure-mode analysis +> in §5 (every way a slot can be lost, every way a client can be left waiting, +> and the invariant that checks both), the target-by-target resumability table in +> §6, and §12 — what a live pass found after the work was called done, which is +> the most useful part of the file. +> +> **Where this document and the code disagree, the code is authoritative**, and +> two places are known to disagree: the leaseless-read bound is 12 files per +> session with an idle expiry, not the 2 written in §3.4.1, and the wire section +> describes MNP 3.0 as pending when it has shipped. `MESHBAY_DESIGN.md` §5.5 +> carries the current values. +> +> Convention carried over and not negotiable: **a claim here names the failure it +> holds against.** "A slot is released when the transfer finishes" is not a +> design; "here is every way a slot can be lost and what reclaims it in each +> case" is. + +--- + +## 0. The short answer + +| Asked for | Feasible | Where the work is | +|---|---|---| +| `max_concurrent_downloads` / `max_concurrent_uploads` per node (default 8) | Yes | The node has **no concept of a transfer** today — that is the whole job | +| Per-member caps inside a group (default 2) | Yes | New per-group operator-signed setting, same shape as `apps_enabled` | +| "Max reached" refusal + client queues the rest | Yes | Better as a **node-side queue with an explicit grant**, not a client retry loop | +| A visible **waiting** state in the transfers widget | Yes | `TransferStore` already has a status field; it gains three states | +| Cancel | **Already works** (`transfers.cancel`, `transfers.js`) | Needs to also tell the node, which today it does not | +| Browsing — posters, thumbnails, previews, listings — untouched by the caps | Yes | §3.4. Structural, not a threshold: the three functions that draw a widget row are the three that take a slot | +| Pause with resume | Yes, **in three tiers** — see §6. Not every download target can be resumed, and the interface must not offer what a target cannot do | + +The risk is not in any one of these. It is that a capped resource plus a queue +is exactly the shape of bug that shows up as *"my download says waiting and +never starts"*, days later, on someone else's machine. §5 is therefore the +longest section in this document and is the one to review hardest. + +--- + +## 1. What exists today, measured + +Read this before designing anything: three of the assumptions one would +naturally make are false here. + +**A download is invisible to the node.** `pipelinedDownload` +(`static/file-utils.js`) sends `PIPELINE_WINDOW = 8` independent `file_req` +messages and reassembles the answers. Each one is served by +`_do_file_request` (`webrtc_server.py:3380`), which looks the entry up, reads +and encrypts one 1 MB chunk, waits for room on the channel +(`DOWNLOAD_BUFFER_HIGH`, `webrtc_server.py:253`) and sends it. **The node never +learns that a download started, and never learns that one ended.** There is +nothing to count and nothing to cap. This is the central fact of this work. + +**An upload is half-visible.** `_do_file_upload` (`webrtc_server.py:4471`) +keeps `self._uploads[f"{rel_dir}/{filename}"]` with `next_index` and a `.part` +file on disk. That state is **per session object and in memory**: a browser +that disconnects mid-upload leaves a `.part` file on the operator's disk that +nothing ever removes, and a retry starts at chunk 0 under a new `_free_name`. +So there is already a resume story here, it is just not reachable — and there +is already a leak. + +**Video streaming is capped, and its cap is a good model.** `_stream_video` +(`webrtc_server.py:5230`) takes `ctx["_transcode_sem"]`, refuses with +`"Server busy, retry shortly"` when it is empty, and every hard-won lesson in +this repo about slots — `_replace_stream`, `shutdown_tasks()`, `_spawn()` and +the garbage-collected task, `await proc.wait()` after `kill()` — is a lesson +about *not losing a slot*. That history is why §5 exists. + +**One bug found while reading, in scope for this work.** `ops.py:1362-1365` +hot-swaps the stream cap by assigning `webrtc._stream_sem` — an attribute that +does not exist. The real semaphore is `ctx["_transcode_sem"]` +(`webrtc_server.py:5220`), and `hasattr(webrtc, '_stream_sem')` is always +False, so **changing `max_concurrent_streams` from the Node page has never +taken effect without a restart**, contrary to §2.11 of draft v6. Two more +hot-swappable caps are about to be added next to it; fix it with one shared +helper rather than three copies of the same mistake. + +**Existing client state.** `TransferStore` (`static/transfers.js`) already +survives leaving a group, already holds the transport open until the last +transfer finishes (`releaseWhenIdle`), already cancels on sign-out, and already +computes a windowed speed. Its statuses are `running | done | cancelled | +failed` and its `run()` contract is a promise plus a polled `signal.aborted`. +That contract is the thing that has to change, and it is the only client-side +structural change. + +--- + +## 2. What a naive implementation gets wrong + +Worth stating, because each of these is a plausible first attempt. + +1. **Counting `file_req` messages.** A rate limit on chunk requests caps + throughput, not concurrency, and gives the client nothing to render as + "waiting". It also cannot distinguish a download from a thumbnail. +2. **Inferring a transfer from activity.** "A (session, file) pair with a chunk + request in the last 30 s is a live download" needs no protocol change and is + tempting. It makes the admission decision unobservable — the client cannot + be told *why* it is being refused, cannot be told when to try again, and a + paused transfer is indistinguishable from a crashed one. +3. **A client-side queue only.** The client already knows what it wants to + download; it does not know what the other five members are doing. A queue + held only by clients cannot order anything fairly and turns into a retry + storm against a busy node. +4. **A refusal the client retries on a timer.** This is the version that + produces the "stuck at waiting" report: the client polls, the node refuses, + nobody is at fault, and there is no evidence anywhere. The node must + **push** the grant. +5. **Gating every `file_req`.** Thumbnails, posters, cover art and audio + transcodes are served through the *same* `file_req` path + (`_try_serve_thumbnail`, `webrtc_server.py:3353`), and so are image and text + previews. Requiring a slot for all of them means opening a poster grid + queues behind a film. §4.4 handles this. +6. **Holding a slot while paused.** If pausing keeps the slot, one member can + pause eight downloads and close the node to everyone else. Pausing must + release. + +--- + +## 3. The design + +### 3.1 The lease + +Introduce **the transfer lease**: the node's record that a peer is transferring +something, held for the length of the transfer and released by name. + +``` +transfer_open { tr, kind: "download"|"upload", bytes, chunks, label_hint } + → transfer_state { tr, state: "granted" } + → transfer_state { tr, state: "queued", ahead: 3 } + …later… + → transfer_state { tr, state: "granted" } ← pushed, not polled +transfer_close { tr, reason: "done"|"cancelled"|"paused"|"failed" } + → transfer_state { tr, state: "closed" } +``` + +Six properties, each of which is load-bearing: + +- **`tr` is drawn by the client**, 16 random bytes hex, exactly like + `upload_id` (`transport.js:2208`). Re-opening after a reconnect with the same + `tr` is idempotent, so a reconnect cannot double-charge a member for one + transfer. +- **A lease is scoped to the connection, never to the account.** It dies with + the session, which is what makes the primary reclaim deterministic (§5). +- **A lease covers a job, not a file.** A directory zip + (`downloadDirectory`, `file-utils.js`) is dozens of files and is **one** + lease. So is a resumed transfer. +- **A lease is never persisted.** Not in `roster.db`, not on disk, not across a + node restart. A restart drops every session anyway; a lease that outlived the + process would be a slot nothing can release. +- **The node counts leases, not bytes.** What a slot protects is concurrency — + open file handles, disk seeks, and the channel buffer each transfer keeps + full — not bandwidth. +- **Every `file_req` for a real file carries its `tr`.** The exceptions are in + §4.4 and they are the delicate part of this whole design. + +### 3.2 Two caps, in one order + +| Cap | Scope | Default | Where it lives | +|---|---|---|---| +| `max_concurrent_downloads` | node-wide, all groups | 8 | `[node]` in `node.toml`, override in `roster.db` (§7) | +| `max_concurrent_uploads` | node-wide, all groups | 8 | idem | +| `max_downloads_per_member` | one group, one account, **across all their devices** | 2 | per-group, operator-signed (§7) | +| `max_uploads_per_member` | one group, one account, across devices | 2 | idem | + +Downloads and uploads have **separate pools**, and video streaming keeps its +own third pool (`max_concurrent_streams`) untouched. A member watching a film +is not charged a download slot, and a download does not make the next film +answer "server busy" — those are different resources with different costs and +merging them would make both caps meaningless. + +**Order of checks, and it matters:** per-member first, then node-wide. A member +at their own cap queues *behind their own transfers* and never consumes a +node-wide slot they would then hold while a second member has none. Reversed, +one member arriving first takes all eight. + +**"Per member" means per account, summed across their devices**, resolved with +`_sessions_of(user_id)` (`webrtc_server.py:3325`) — which exists for exactly +this reason, since device linking landed. Two browsers and a desktop client +signed in as the same person share the two slots. Anything else makes the cap a +function of how many tabs someone opens. + +### 3.3 The queue + +One FIFO per pool, per node, holding `(tr, session, user_id, group_id, +enqueued_at)`. + +- **Drained in one place.** `_release_slot()` is the only function that returns + a slot, and it is the only caller of `_pump_queue()`. Every path that ends a + transfer goes through it, in a `finally`. Two functions that both release + would be this repo's flow-control-accounting lesson (`CLAUDE.md`) one feature + later. +- **Head-of-line blocking is skipped, not waited on.** When a node-wide slot + frees, walk the queue and grant to the first entry whose *member* is under + their own cap. Granting strictly in order would let one member at their + personal cap stall the whole node. +- **A grant has an acceptance deadline.** 30 s. If no `file_req` bearing that + `tr` arrives, the grant is revoked (`state: "queued"` again, at the tail) and + passed on. Without this, a client that dies between the grant and the first + chunk holds a slot until the idle timeout. +- **The queue is bounded**: 32 entries per account per pool. Beyond that the + node answers `too_many_queued` and the client holds the rest in its own list, + sending `transfer_open` as its own transfers finish. Unbounded queues are how + a node runs out of memory politely. +- **Positions are pushed, throttled.** After each pump, recompute positions and + send `transfer_state {state:"queued", ahead:n}` only to the sessions whose + number changed, at most once every 2 s per session. + +### 3.4 What is *not* gated — posters, thumbnails, previews, navigation + +**Operator decision, 2026-09-08: browsing a group is never subject to a +transfer slot.** Not the poster grid, not the album covers, not the video +thumbnails, not the file list, not opening a photo or a PDF to look at it. A +member must be able to browse a group that is at capacity exactly as they +browse an idle one. This is a requirement, not a tuning parameter, and the +sections below are written to satisfy it structurally rather than by choosing a +lucky threshold. + +**Navigation proper never touches this path at all.** The file list, the +directory tree, the poster metadata and the album metadata travel as +`index_sync`, `index_delta`, `media_meta_req`, `music_meta_req` and +`link_preview_req` — different message types, sealed under the group key, with +no relationship to `file_req`. Nothing in this design can reach them. That half +needs no rule; it needs only to be verified by a test that fails if someone +later routes a listing through `file_req`. + +The `file_req` path carries three genuinely different things, and they are +distinguishable **structurally**, by what the id resolves to and by which +function asked: + +| What | Call site | Resolves to | Rule | +|---|---|---|---| +| Thumbnails, TMDB posters, cover art, cached audio transcodes | `MediaThumb` (`video-app.js:204`), and the Music/Photos grids through it | a **media-cache id**, not an index entry — `_try_serve_thumbnail`, `webrtc_server.py:3353` | **Never leased, never counted, never queued.** One chunk each, out of a bounded cache the node built itself | +| Looking at one file — a photo opened full size, a PDF, an image, a text file | `PhotoViewer` (`photos-app.js:171`), the Files preview modal (`files-app.js:610`) | a real index entry, fetched whole | **Not leased.** Bounded by §3.4.1 below, which no real viewer ever reaches | +| Downloading, and uploading | `downloadEntry`, `downloadDirectory` (`file-utils.js`), `uploadFile` (`transport.js:2192`) | a real index entry | **Leased.** These are exactly the three call sites that go through `transfers.start()` — the three that produce a row in the transfers widget | + +The last column is the whole rule, and it is worth stating as a sentence +someone can check by reading: **a transfer is something the transfers widget +shows. If it does not appear in that panel, it does not take a slot.** The two +sets are the same three functions, which is what makes this verifiable rather +than a matter of judgement at each new call site. + +#### 3.4.1 The bound on leaseless reads + +A `file_req` with no `tr` on a real index entry is served, subject to one +limit: **at most 2 distinct file ids in flight leaselessly per session**, with +no queue — the third is refused with `transfer_required`. + +Why this shape, and not the byte budget an earlier draft of this document +proposed: + +- **A viewer looks at one file.** The photo viewer shows one photo, the preview + modal one document. Two is already one more than any of them needs, and is + there so that prefetching the next photo stays possible. +- **A size threshold does not work here.** A RAW photo out of a camera is + 60–80 MB and is *browsing*; a 40 MB archive is a *download*. Size does not + separate them, and any threshold that let the photo through would let the + archive through too. What separates them is which function asked. +- **A byte-rate budget does not work either.** It would have to be large enough + for that same RAW photo, at which point it is large enough to be a download + channel. Concurrency is the thing being rationed, so concurrency is what the + exemption is expressed in. +- **What it costs.** A client that lies — labelling a bulk download as a view — + gets two files at a time per session instead of its member cap. That is the + residual, it is bounded, it is audited (`file_download` already goes to + `audit.db` on chunk 0), and it is the same class of statement as the cap + itself: **this is a fairness control among cooperating clients**, in the + company of `max_concurrent_streams`. It is not a defence against a member + determined to saturate the node's disk, and must never be described as one — + that member is a member, and the answer to them is `member revoke`. + +#### 3.4.2 The consequence for the interface + +A preview never shows "waiting", because a preview never queues. If the +2-in-flight bound is somehow reached, the modal reports a plain error and the +person tries again — it does not silently become a queued transfer in a panel +they were not looking at. + +### 3.5 Where the numbers are visible + +The node answers `transfer_state` with the current picture — `used`, `cap`, +`ahead` — so the client can say *"waiting — 2 of 2 of your slots are busy"* +rather than a bare spinner. The same counters go to: + +- the loopback API, `GET /api/transfers`, so the operator can see live leases + and queue depth from the CLI and the Node page, and +- a DEBUG line every 30 s: `transfer: d=3/8 u=1/8 queued=2 (skipped=1)`. + +That line is the `client_diag` lesson applied here: when someone reports a +transfer stuck at "waiting", this is the only thing that will say whether the +node ever had them in a queue. + +--- + +## 4. Wire protocol — MNP 3.0 + +### 4.1 New message types (`meshbay_common/protocol.py`) + +| Type | Direction | Carries | +|---|---|---| +| `transfer_open` | client → node | `tr`, `kind`, `bytes`, `chunks`, `from_chunk` | +| `transfer_close` | client → node | `tr`, `reason` | +| `transfer_state` | node → client | `tr`, `state`, `ahead`, `used`, `cap`, `scope` | + +`transfer_state` is the only reply, for granted, queued, revoked and closed +alike. One message type with a state field, rather than four types, because a +client that must switch on the type to find out it is still waiting is a client +that will get one branch wrong. + +`file_req` gains an optional `tr`. `file_upload` gains `tr` beside the +`upload_id` that is already there in clear. + +**Sealing.** `transfer_open` / `transfer_close` / `transfer_state` carry no +file names and no paths — `tr` is opaque, `bytes` and `chunks` are numbers — +so they stay in clear like `index_progress`, and for the same stated reason. +`label_hint` is **not** in the protocol: the client already knows what it named +the transfer, and putting a filename on the wire in clear to make a log line +prettier is exactly the trade `groupbox.py` exists to refuse. + +### 4.2 Why this is MNP 3.0 and not an additive 2.x + +The messages are additive; **the requirement is not**. A 2.0 client downloading +a 4 GB film sends no `tr`, is treated as a leaseless read, and is refused as +soon as it opens a third file (§3.4.1) — or, worse, is *not* refused and +transfers outside every cap. An opt-in switch — "enforce leases only for +clients that speak 3.0" — leaves a leaseless branch reachable on every node, +which is finding C6's lesson (`TCP accepted a bare JWT`) one feature later, and +it was already refused once for chat encryption on 2026-09-07. + +So: **`MNP_VERSION = "3.0"`, `MNP_MIN_SUPPORTED = "3.0"`**, and a 2.x peer is +refused at the handshake with `version_too_old` rather than admitted and then +mysteriously unable to download. Same play, same reasoning, and +`handshake.py`'s version range is the mechanism that already exists to pay for +it. + +**What that costs, stated plainly.** The SPA deploys with the hub, so browsers +get the new client. The **desktop client ships its own UI**, so an un-updated +app is locked out until its user updates — the version-skew consequence draft +v6 flagged under "Shipping the UI in a package". Before this ships, +`GET /v1/hub/version` needs its minimum-client-version field populated and the +client needs to show a real message ("this version can no longer connect, +update here") rather than a handshake refusal. + +### 4.3 Handshake ack + +Two fields inside the sealed configuration block (`_complete_handshake`, +`webrtc_server.py:852`), so the interface can draw correct numbers before +anything is transferred: + +```python +"transfer_limits": {"downloads": 2, "uploads": 2}, # this member, this group +``` + +Absent reads as "no limit known" and the client simply does not draw the +"1 of 2" hint — never as "unlimited", which would have the interface +contradicting the node. + +--- + +## 5. Nothing gets stuck + +The requirement the operator stated first, and the one worth over-engineering. +Two directions, and both must be closed: + +- **a slot the node never gets back** — the node fills up and everyone queues + forever; +- **a transfer the client shows as waiting when the node has forgotten it** — + one person's widget lies while the node is idle. + +### 5.1 Every way a slot can be lost + +| How it ends | What reclaims the slot | When | +|---|---|---| +| Transfer completes | Client sends `transfer_close{done}` | Immediately | +| …and the client forgets to | **Node auto-closes on serving the last chunk** — it knows `entry.size` and `CHUNK_SIZE`, so it knows the final index | Immediately | +| User cancels | `transfer_close{cancelled}` | Immediately | +| User pauses | `transfer_close{paused}` | Immediately | +| Tab closed, browser quit, app killed | `shutdown_tasks()` / `_unregister_peer()` releases every lease of the session | On WebRTC `connectionstatechange` — the same hook that already ends streams | +| Network drops, no clean close | Idle reclaim: no `file_req` under this `tr` for 120 s | ≤ 120 s | +| Client granted a slot and never used it | Acceptance deadline | 30 s | +| …and it *is* using it, but nothing said so | **`touch()` on every `file_req` carrying the lease.** Missing from the first build: the pool had the method, the tests covered it, and no caller existed — so `used` stayed False for every download and the acceptance deadline revoked each grant at 30 s while the file transferred at 20 MB/s | — | +| A grant revoked, requeued, granted again, revoked again | **Bounded at three misses, then the lease is closed.** Also missing at first: the requeue was a permanent cycle and the node logged the same reclaim every 30 s until it restarted | — | +| Client crashes mid-zip between two files | Idle reclaim (the gap between files is milliseconds) | ≤ 120 s | +| Node restarts | Leases are in memory only | Immediately | +| Group detached / member revoked mid-transfer | The existing revocation path drops the sessions, which releases their leases | Immediately | + +The first two rows are the answer to *"a user must not stay blocked once their +downloads have been done"*: the client says so, **and** the node concludes it +independently. Neither is trusted alone. + +### 5.2 Every way a client can be left waiting + +| Failure | What fixes it | +|---|---| +| Node granted a slot, the push was lost | Client watchdog: no state change for 60 s while `queued` → re-send `transfer_open` (idempotent on `tr`) | +| Reconnect: session gone, leases gone | `_onReconnected` (already in `transport.js`) re-opens the lease for every running/queued transfer, with `from_chunk` | +| Node forgot the lease (idle reclaim during a stall) | Node pushes `transfer_state{state:"revoked"}`; the widget shows **interrupted — resume**, not a silent hang | +| Node at cap and everyone is idle | Cannot happen if §5.1 holds; if it does, `GET /api/transfers` shows the leases and the operator can force-release one. Ship that endpoint | +| Queue entry for a member who left the group | Pump skips entries whose session is closed and drops them | + +### 5.3 The invariant, and how it is checked + +> **A new grant is never made past a cap, and every queue entry names a lease +> that exists. A lease is created in exactly one function and destroyed in +> exactly one function.** +> +> Not "granted leases never exceed the cap" — that was the first wording and the +> property test rejected it within a second of being written. Lowering a cap +> never interrupts a running transfer, so the count legitimately sits above the +> new value until those finish. The invariant is about what may be *handed out*, +> not about what is held. + +Two tests, not one: + +- a unit test per row of both tables above; +- a **randomised property test** — a few thousand random sequences of open, + close, drop, reconnect, pause, resume and cap changes, asserting after each + step that the counter equals the number of live leases and that no queue + entry references a dead session. The stuck-slot bug is a race by nature, and + "it works now" is not evidence against a race (`window_leak.mjs` is the + precedent in this repo). + +--- + +## 6. Pause and resume + +Cancelling already works client-side (`transfers.cancel`) and needs only to +send `transfer_close`. Pausing is new, and **whether it can be resumed depends +entirely on where the file is being written**. The interface must offer only +what the target can actually do — a pause button that quietly restarts the +download from zero is worse than no pause button. + +### 6.1 By download target + +| Target | Platform | Pause in-session | Resume after reload | Notes | +|---|---|---|---|---| +| File System Access (`showSaveFilePicker` / granted folder) | Chrome, Edge | Yes — keep the `writable` open, stop asking for chunks | **Yes** — reopen with `createWritable({keepExistingData:true})` and `write({type:"write", position})` | The full story. `downloads.js` already keeps the directory handle in IndexedDB | +| Electron native sink | Desktop app | Yes | **Yes**, once `main.js` gains a `save:resume` opening the file with `flags:'a'` and returning its current size | ~30 lines in `main.js` + `preload.js` (`main.js:797`, `preload.js:150`) | +| Service-worker stream | Firefox, Safari | **No** | No | Not on this target — see §6.5, which says what would be needed to change that. **Hide the pause button**, keep cancel | +| Blob in memory | Fallback | Yes (it is just an array) | No | Bounded by `BLOB_LIMIT` anyway | +| **Upload** (any platform) | All | **Yes** | **Yes**, via the sealed probe | A `File` is seekable and the *node* keeps the position, so there is no local target to consult. Missed entirely when 7a shipped — pause was built around the download path and uploads were refused it by the same guard that protects a transfer which cannot re-acquire its slot | + +**Corrected while implementing 7a.** `platform.capabilities` is the wrong home +for this: resumability is a property of the *target*, not of the platform. The +same Chrome yields a pausable target from a granted folder and an unpausable one +from the service worker, on the same page, for two files in the same batch. So +each target declares `pausable` itself, `prepare` carries it into the store, and +the widget renders from that record. The precedent to +follow — and the mistake not to repeat — is `_openDownloadTarget`'s silent +fallback chain (`CLAUDE.md`: *"a fallback chain reaches its floor silently"*). +Whatever the target ends up being, the transfer records which tier it got, and +the widget renders from that record. + +### 6.2 Pause releases the slot + +Stated again because it is a design decision, not an implementation detail: a +paused transfer holds nothing. Resuming rejoins the queue at the tail, and the +widget says so (*"resuming — 2 ahead"*). Anything else lets one member close +the node by pausing. + +### 6.3 Resuming the transfer itself + +Trivially available, because `pipelinedDownload` is already indexed by chunk: +resume is `nextRecv = Math.floor(bytesWritten / CHUNK_SIZE)` with the window +refired from there. Two constraints: + +- **`bytesWritten` must be a multiple of `CHUNK_SIZE`.** It is, because writes + are sequential and whole chunks — but a resumed file whose size is not a + chunk multiple (a partial write interrupted by a crash) must be **truncated + down** to the last whole chunk before resuming, never appended to. A silently + corrupted download is worse than a failed one. +- **The file must not have changed.** The entry id is its blake3 + (`GroupIndex`), so the resume record stores `{fileId, groupId, size, + bytesWritten, targetRef}`. A resume whose `fileId` is no longer in the index + fails with "this file has changed on the node", which is the truth. + +### 6.4 Resuming an upload + +The node already has the state (`self._uploads`, `.part` on disk). Make it +usable and stop it leaking: + +- **Corrected while implementing.** `transfer_open` travels in clear, and the + node identifies an upload by (member, directory, filename) — so asking there + would put the operator's filenames on an unsealed message, which is precisely + what sealing this path bought in MNP 2.0. The question is asked inside the + seal instead, as an ordinary `file_upload` with no bytes and + `UPLOAD_PROBE_INDEX` (-1); the node answers `resume_from` in the sealed ack, + writing nothing and reserving nothing. A node that predates it refuses the + index, which reads as "start from the beginning". +- Move `self._uploads` from the session to the **group context**, keyed by + `(user_id, rel_dir, filename)`, so a reconnect finds it. It is already + authorised per member; the session was never the right owner. +- **Reap orphans.** A `.part` older than 24 h with no live lease is deleted, on + a timer and at startup. This is a pre-existing leak (§1) and this work is + where it gets fixed. +- The no-overwrite rule, the filename allowlist, the size cap and the chunk + ordering are **unchanged** — draft v6 §2.1 names those four as what makes an + upload safe, and none of them is touched here. + +--- + +### 6.5 Firefox and Safari: OPFS, and what a pause would cost + +> **Corrected three times on 2026-09-08, ending here.** This section first said pause +> was impossible on Firefox; then that OPFS was the missing streaming target. +> **Both were wrong, and measurement is what settled it.** Firefox 154's OPFS +> quota is exactly 10% of the volume's size — ~12.8 GB on a 128 GB disk, hit +> *mid-download* — so OPFS cannot carry a film. The unbounded path is the +> **service worker**, which was already there and merely unreliable; it was +> fixed on 2026-09-08 (see `memory-audit-large-files.md` §6.5 and +> `test_streamed_download_reliability.py`) — and then fixed again the same +> evening, because "reliable" turned out not to include *staying alive*: an idle +> service worker is terminated after about thirty seconds and a streaming +> response does not count as activity, so every download longer than that lost +> its reader mid-file (§12.2). What survives below is only the pause question, +> for which OPFS is still the candidate, under its quota. + +Deferred by decision, so this section records **why**, because "Firefox cannot +pause" is not true as stated and would be a bad thing to leave in a plan. + +**Why the current target cannot pause.** On the service-worker path +(`downloads.js:openStreamedDownload`) the browser is already writing an HTTP +response to its own download folder. Three separate things break, and only the +first is about our code: + +- pausing means not writing to the stream, which leaves the browser's download + stalled in its own download manager — visible to the user, outside our + control, and cancellable by them in a way we never hear about; +- **a service worker is terminated when idle.** Firefox and Chromium both kill + one after tens of seconds with nothing to do. A pause that outlives that — + which any pause waiting on a queue does — takes the stream down with it; +- resume after a reload is genuinely impossible on this target. The response is + committed; no API reopens a browser-managed download to append to it. + +**The alternative that does exist: OPFS.** The Origin Private File System +(`navigator.storage.getDirectory()`) is a real, seekable, persistent +filesystem, private to the origin, and it **is** implemented in Firefox and +Safari — which is exactly what those two lack when they lack File System +Access. A download would be written there, resumable and reload-proof like any +local file, and handed to the person at the end: `handle.getFile()` returns a +`File` backed by disk, so `URL.createObjectURL` on it is a link the browser +copies disk-to-disk without loading gigabytes into the tab. + +So it is not a browser limitation. It is a **second download target to write +and maintain**, and it costs: + +- a fourth tier in `_openDownloadTarget`'s already four-deep chain — the + function whose silent floor cost this project a gigabyte of RAM per film + once already (`CLAUDE.md`); +- **quota.** OPFS is subject to storage quota and to eviction unless + `navigator.storage.persist()` is granted. A 4 GB film may simply not fit, and + finding that out at 90% is worse than not offering the path; +- **double disk usage and a real wait at the end.** The file exists twice while + the browser copies it out, and the copy is not instant for a large file; +- **cleanup.** Abandoned partial downloads sit in OPFS invisibly, consuming the + origin's quota, until something reaps them. That is a new janitor with its + own failure mode; +- **an API matrix that must be measured, not read.** The OPFS write path is not + uniform: `createSyncAccessHandle()` (workers only) and `createWritable()` have + had different availability across Firefox and Safari versions. This repo's + standing rule applies — *model the environment, never the code under test* — + so the answer comes from a probe in real browsers, not from a compatibility + table. + +**Recommendation.** Not in this work. Revisit it as its own change once transfer +slots are in production, and if it is picked up, it is worth more than pause +alone: it would give Firefox and Safari resumable downloads, reload-proof, and +retire the service-worker path and its iframe entirely. Ship it as a target +first and let pause fall out of it — the reverse order is how a fourth tier +gets bolted onto the chain in a hurry. + +## 7. Settings + +### 7.1 Node-wide — draft v6 §2.11's pattern, exactly + +Two new `[node]` keys, `max_concurrent_downloads` and +`max_concurrent_uploads`, default 8, positive int, `_positive()`-validated like +`max_concurrent_streams` (`config.py:332`). Then, mechanically: + +- `roster.py`: `SETTING_MAX_DOWNLOADS` / `SETTING_MAX_UPLOADS`, added to + `node_settings()` (`roster.py:943`); +- `ops.py`: entries in `get_node_settings` / `set_node_settings` + (`ops.py:1290`, `ops.py:1309`), written to `roster.db` **and** `node.toml`; +- **hot-swap through one helper.** `ops.set_node_settings` currently pokes + `webrtc._stream_sem`, which does not exist (§1). Replace all three with + `webrtc.set_capacity(streams=…, downloads=…, uploads=…)` on + `WebRTCTransport`, which resizes the live pools and pumps the queues. One + function, on the object that owns the state, with a test that changes a cap + on a running transport and watches a queued transfer start; +- the Node page grows two number fields beside "max concurrent streams" + (`node-page.js:1042`) and the CLI grows + `meshbay-node transfers show|set` — plus `meshbay-node transfers list`, + which prints live leases and the queue from `GET /api/transfers`. + +### 7.2 Per group, per member — `apps_enabled`'s pattern, exactly + +`OP_TRANSFER_LIMITS` in `adminop.py`, subject `"d=2,u=2"` so what the operator +signs names the outcome. `_do_transfer_limits` + +`_admin_exec_transfer_limits` in `webrtc_server.py`, stored by +`ops.set_transfer_limits` in `roster.db` group settings, broadcast as +`transfer_limits_ack` to the group's peers, surfaced in the group Settings tab +as a section beside the scan settings. + +**Absent means the default (2), not unlimited.** Deliberately unlike +`member_upload`'s "absent means allowed": a group that predates the setting and +came back unlimited would leave the node-wide cap as the only control, which is +the situation this work exists to end. Enabled by default, breaking, no opt-out +switch — the same call as MNP 2.0's chat encryption, for the same reason. No +release note and no operator prose: the project is in development and every +node is a test node. + +Bounds: 1–32. Zero is not "unlimited" and is not accepted; a member who may not +transfer at all is a member the operator revokes. + +--- + +## 8. The client + +### 8.1 `TransferStore` — three new states, one changed contract + +``` + ┌──────────┐ slot granted ┌──────────┐ + start() ─────▶│ queued │────────────────▶│ running │ + └──────────┘ └──────────┘ + ▲ │ │ │ │ + resume() │ │ cancel() pause() │ │ │ done + │ ▼ ▼ │ ▼ + ┌──────────┐ ┌────────┐ ┌──────┐ + │ paused │◀──────────────│ paused │ │ done │ + └──────────┘ └────────┘ └──────┘ + │ │ + cancel│ failed / cancelled + ▼ + ┌───────────┐ + │ cancelled │ + └───────────┘ +``` + +Statuses become `queued | running | paused | done | cancelled | failed`, and +`run({signal, onProgress})` becomes +`run({signal, onProgress, lease, resumeFrom})`: + +- `lease` — an object the store hands the runner: `await lease.acquire()` + blocks until granted, `lease.release(reason)` in a `finally`. The runner + never speaks to the transport about slots directly. One place opens a lease, + one place closes it — the same rule as the node's. +- **The slot is asked for after there is somewhere to write, and that ordering + is load-bearing.** Asking first reads better — the widget could draw a row + while the target is being chosen — and is wrong: opening a target takes tens + of seconds of streamed-download timeouts, or as long as somebody leaves a Save + As dialog open, and a grant not taken up in time is revoked. Tried the other + way during the build: three downloads started, one arrived. A test pins the + order now. +- The store gains `pause(id)`, `resume(id)`, `pauseAll()`, and a persisted + resume record for the tiers that support it. +- `_maybeRelease(transport)` must treat **paused and queued as busy** + (`transfers.js`'s `_busy`). A paused transfer whose transport was closed + because the group page went away can never resume — that is a one-line + regression waiting to happen, and it has a test. + +### 8.2 The widget + +`TransferWidget` (`app.js:141`) is a flat list of rows with a progress bar. It +becomes a small panel that can answer, at a glance, *what is happening and what +is my fault*: + +``` +┌─ Transfers ───────────────────── 2 running · 3 waiting ─┐ +│ ⏸ Pause all Clear finished │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ Some Saga (2019).mkv ⏸ ✕ │ +│ ███████████████████░░░░░░░░░░ 62% │ +│ 2.4 GB / 3.9 GB · 11.2 MB/s · 2 min left │ +├──────────────────────────────────────────────────────────┤ +│ ⬆ holiday-photos.zip ⏸ ✕ │ +│ ████████░░░░░░░░░░░░░░░░░░░░ 27% │ +│ 118 MB / 430 MB · 3.1 MB/s · 1 min left │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ A Different Show S02E04.mkv ✕ │ +│ ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ waiting · 1 ahead │ +│ Your 2 slots are busy │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ render.tar ▶ ✕ │ +│ ██████░░░░░░░░░░░░░░░░░░░░░░ paused at 21% │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ notes.pdf Finished ↗ │ +└──────────────────────────────────────────────────────────┘ +``` + +Concretely: + +- **A summary in the header** — `2 running · 3 waiting` — and the nav badge + counts running *and* waiting, so a queued transfer is never invisible. +- **Waiting rows have a striped, indeterminate bar** (a CSS gradient animation, + no JS), visibly different from a stalled progress bar. `prefers-reduced-motion` + turns the animation off, not the state. +- **The reason for waiting is written out**, from the node's own counters: + *"Your 2 slots are busy"* vs *"The node is at capacity — 1 ahead"*. Someone + should be able to tell their own limit from the operator's without asking. +- **Pause / resume / cancel as icon buttons**, with real `aria-label`s and + `title`s; pause is **absent, not disabled**, where the target cannot resume + (§6.1) — and the row carries a quiet hint saying why, once. +- **ETA** from the existing windowed speed, hidden below 5 s of samples so it + does not flicker nonsense at the start. +- **Ordering**: running, then waiting (queue order), then paused, then + finished. A row does not jump when its neighbour finishes — group headers + rather than a re-sort on every emit. +- **Live region**: state transitions announce once (`aria-live="polite"`), + progress does not. +- Colours from the existing CSS variables, so light and dark both work without + a second palette. New rules go in `style.css` beside `.transfer-*`, measured + with `tests/harness/layout_probe.py` at 320 px — the transfers panel is + *precisely* what that harness was written for, after it hung 138 px off the + left of a phone. + +### 8.3 i18n + +New keys: `transfers.waiting`, `transfers.waiting_ahead`, `transfers.paused`, +`transfers.pause`, `transfers.resume`, `transfers.pause_all`, +`transfers.your_slots`, `transfers.node_busy`, `transfers.eta`, +`transfers.interrupted`, `transfers.no_pause_here`, `transfers.running_count`, +plus the settings labels. `en.js` is the source; `test_locales.py` holds the +other nine to its key set and will fail until all ten are translated. + +--- + +## 9. Regressions to watch + +Ranked by how quietly they would ship. + +1. **Browsing degraded by a busy node — the operator's stated requirement + (§3.4).** `_try_serve_thumbnail` shares `file_req` with real files, and the + Photos viewer and the Files preview modal fetch whole index entries through + `pipelinedDownload` just as a download does. Four tests, not one: with both + pools full and a queue behind them, (a) a Videos poster grid still fills, + (b) a Music album grid still fills, (c) a photo opens full size, (d) a PDF + preview opens. Each must also fail with the exemption removed. +2. **Video streaming charged twice.** `stream_req` must not touch the download + pool. Test: fill the download pool, start a film, assert it plays. +3. **Chat attachments — decided: they take an upload slot** like any other + upload, through the same `_do_file_upload` path. The regression is therefore + in the *interface*, not the rule: a paperclip whose file silently sits in a + queue is a chat that looks broken. `chat-app.js`'s composer must render the + same waiting state as the transfers panel — the attachment row says + "waiting, 1 ahead", not nothing — and a queued attachment must not block the + composer, which is the freeze already recorded in `CLAUDE.md` for chat + sends. Test it in `chat_send_probe.py`, which drives composer and transport + together for exactly this class of seam. +4. **`_maybeRelease` closing a transport with paused transfers on it** (§8.1). +5. **Zip downloads.** One lease for the whole job. Getting this wrong means a + 40-file zip takes 40 slots and deadlocks against its own cap — a deadlock, + not a slowdown, because the job cannot finish until it holds them all. +6. **Search across groups** (`search-page.js`, `getTransport`) opens transports + to several nodes; each node caps independently. Correct, but the widget must + not present another node's queue as this one's. +7. **`_free_name` and resumed uploads.** A resumed upload must reuse + `state["stored_name"]`, not draw a new one — otherwise a reconnect produces + `file (2).mkv` next to a half-written `file.mkv.part`. +8. **The 60 s cap in `_do_file_request`'s backpressure loop** interacts with a + paused reader: a paused transfer stops requesting, so nothing is waiting — + but check that a *slow* reader is not now reclaimed as idle. The idle timer + is reset on each `file_req` under the lease, not on each chunk *sent*. +9. **`test_security_regressions.py:786`** asserts `_transcode_sem` is in the + source. Renaming it during the `set_capacity` refactor will fail that test — + which is the test doing its job. Update it deliberately. + +10. **The one this list did not anticipate, and the one that happened:** none of + the eight defects the live pass found are in this list, because every entry + here is about *slots* and seven of the eight were about the download path + underneath them (§12.2). A regression list written from the change being + made will not cover the ground the change stands on. Before the flag day, + walk the download path itself — every write target, on every browser — + rather than the diff. + +--- + +## 10. Tests + +Following the repo's rule: measure the environment, never model the code under +test. + +**Node (`packages/meshbay-node/tests/`)** + +- `test_transfer_slots.py` — real `WebRTCPeerSession` objects against a fake + DataChannel (the shape `test_webrtc_transport.py` already uses): node cap, + per-member cap across two sessions of one account, queue order, head-of-line + skip, grant deadline, idle reclaim, auto-close on the last chunk, release on + `shutdown_tasks`, no double-charge on a re-`transfer_open` with the same `tr`. +- `test_transfer_invariant.py` — the randomised property test of §5.3. +- `test_transfer_exemptions.py` — the §3.4 requirement, as behaviour: posters, + covers, thumbnails, a full-size photo and a document preview all served with + both pools full and a queue waiting; the 2-in-flight bound refusing a third + concurrent leaseless entry; and a source-reading check that the only callers + opening a lease are the three that call `transfers.start()`. +- `test_navigation_not_leased.py` — `index_sync`, `index_delta`, + `media_meta_req`, `music_meta_req` and directory listing answered normally + with both pools full. Cheap, and it fails the day someone routes a listing + through `file_req`. +- `test_upload_resume.py` — reconnect mid-upload resumes at `next_index`; + orphan `.part` reaped; `stored_name` preserved. +- `test_node_settings.py` extension — `set_capacity` actually resizes a live + pool (the test `ops.py:1362` never had). + +**Hub / SPA (`packages/meshbay-hub/tests/`)** + +- `test_transfers.py` extension, under Node as today: queued → running, pause + releases, resume re-queues, cancel while queued, `_busy` counts paused. +- `tests/harness/transfers_probe.py` — mounts the real `TransferWidget` in + Chrome against a stubbed store and reads the rendered states back. The + precedent is `chat_send_probe.py`: the seam between store and widget is where + this will break, and neither source shows it. +- `test_layout_measured.py` extension — the panel at 320 px with a waiting row. +- `test_locales.py` — passes only when all ten catalogues have the new keys. + +**Live — `packages/meshbay-node/tests/transfer_probe.py`, in the repo** + +Not in `QE/` as this section first said: it found four defects nothing else +could, and `QE/` is not versioned, so it lived on one machine. It is not +collected by pytest (the filename does not match `test_*.py`) and still needs +`QE/deploy/e2e.py` and `demo.env` at run time, which it locates and explains +rather than importing blindly. + +`--want N` measures the cap, `--pull N [--parallel]` downloads real files to +completion on one connection, and `--operator` covers the two things only the +operator's CLI can answer: that a cap raised live starts what was waiting, and +that a vanished peer's slots are back before anyone asks. + +- ~~`QE/deploy/transfer_probe.py`~~ — opens N real MNP sessions as two accounts, + starts more transfers than the caps allow, and asserts the observed + concurrency and the order they complete in. The equivalent of + `stream_probe.py`, and the only thing that answers "is it the node or the + browser" in one run. +- A two-browser manual pass: cap 2, start 5, watch the queue drain; pause one, + reload the page, resume it; kill a tab mid-transfer and watch the slot come + back within 120 s. +- `QE/migration/reap_orphan_parts.py` — one-shot, node stopped, for the `.part` + files already on deployed nodes (§6.4). + +Per the QE rule in `CLAUDE.md`: any test against meshbay.org opens the UFW +port, tests, and closes the port and kills the processes **in the same block**. + +--- + +## 11. Order of work + +Each stage is shippable and testable on its own; nothing before stage 5 changes +what a member sees. + +| # | Stage | Contents | +|---|---|---| +| 1 | ✅ **Capacity, fixed** | `set_capacity()` on `WebRTCTransport`, replacing the dead `_stream_sem` poke. No new features. Ships the §1 bug fix alone, where it can be verified alone | +| 2 | ✅ **Leases, node-side** | Lease table, pools, queue, pump, every reclaim path in §5.1, `GET /api/transfers`, the DEBUG line. No client uses it yet; the node grants everything because no client asks | +| 3 | ✅ **Settings** | `[node]` keys, roster overrides, `OP_TRANSFER_LIMITS`, Node page fields, CLI verbs, ack field | +| 4 | ✅ **MNP 3.0** | Version bump on both sides, `MNP_MIN_SUPPORTED` at 3.0, **§3.4.1's bound on leaseless reads** (which did not exist), the desktop client's version gate, every package aligned on 0.13.0 | +| 5 | ✅ **Client leases** | `TransferStore` states and the lease contract, `file_req`/`file_upload` carrying `tr`, reconnect re-open, watchdog | +| 6 | ✅ **Widget** | The panel of §8.2, i18n, layout measured | +| 7a | ✅ **Pause / resume, in session** | Per-target `pausable`, the slot released on pause and re-asked on resume, `fromChunk` in the pipeline, the widget's pause button | +| 7b | ⬜ **Resume across a reload** | The persisted resume record, `save:resume` in Electron, `createWritable({keepExistingData:true})` with a position, truncate-to-chunk | +| 8a | ✅ **Upload state + reaping** | `uploads.py`, state in the group context keyed by member, the `.part` janitor | +| 8b | ✅ **Upload resume + pause** | The sealed probe chunk, `resume_from`, uploads asking for their own slot, `touch()` on an upload chunk | +| 9 | ✅ **Live pass** — done out of order, and §12 is its report. | `transfer_probe.py`, two browsers, the manual list | +| — | *Deferred, separate change* | **OPFS as a download target** for Firefox and Safari (§6.5) — resumable downloads there, and the retirement of the service-worker path. Not part of this work | + +Stages 1–3, 5 and 6 have landed. Stage 9 was run **before** stage 4 rather than +after, deliberately: while nothing is enforced, a defect in the machinery costs +nothing, and §12 is what that bought. Stage 4 is a flag day and should land only +on the evidence §12.3 describes. + +--- + +## 12. What the live pass found, and what it changes + +Step 9 was meant to confirm the machinery. It found **eight defects**, seven of +which no test in the repo could reach, and three of which are in code the +sections above describe as settled. Four more (§12.4 to §12.7) came out of +fixing those eight, and the last of them — a hard reload silently disabling the +only unbounded write path Firefox has — was the one actually being reported all +along. They are listed here because their pattern matters more than +any one of them: every one needed a real browser, a real node and a real hub to +show itself, and the last two were only reachable on Chrome. + +### 12.1 The three in the lease machinery + +| | Found by | +|---|---| +| **`touch()` was never called.** The node ignored `tr` on `file_req`, so `used` stayed False for every download and each grant was revoked at 30 s mid-transfer | the node's own log | +| **The requeue was a permanent cycle.** Revoked → queued → granted again → revoked, every 30 s, for as long as the daemon ran | the node's own log | +| **`transfers show` reported the module defaults**, so `transfers set 2 2` answered "applied now" and the next line said 0/8 | typing the command | + +The first is the one to learn from. `TransferSlots.touch()` existed, was +covered by its own test, and **had no caller**. The pool was right, the +handlers were right, and the call between them was missing — so neither side's +tests could see it, and the property test could not either: nothing drifted. +**A seam is not tested by testing both sides of it.** + +### 12.2 The five in the download path itself + +None of these are about slots, and all of them would have become "this download +is impossible" the moment leases were compulsory: + +- **three headers** decide whether the page may frame its own `/_mbdl/<id>` URL + — `frame-src`, `frame-ancestors`, `X-Frame-Options` — and all three were + wrong. Fixed one at a time over an afternoon; all three were visible in a + single `curl -I` against the deployed hub; +- **a service worker with no event for ~30 s is terminated**, and a streaming + response does not count as activity. The reader vanished mid-file and + `writable.write()` never resolved *and never rejected*: no error, no log, a + progress bar that stopped, and a node that stayed healthy throughout; +- **`encodeURIComponent` leaves `'` alone** and `'` is RFC 5987's delimiter, so + a 449 MB film arrived complete and correct under the name + `mtsshk9w-ohqty535`; +- **a browser grants one file picker per user gesture**, so downloading three + files at once failed two of them with a message about gestures; +- **two silent returns** meant a click with no connection produced nothing at + all — no transfer, no icon, no message. + +### 12.3 What this changes about step 4 + +Step 4 makes leases compulsory and refuses 2.x peers at the handshake. It is the +only irreversible step in this plan. + +Every defect in §12.1 and §12.2 was invisible to 1169 node and 814 hub tests, +and was found by a person clicking Download and pasting a console. Three of them +were introduced *by this work* on the day it was written. Making a mechanism +compulsory is a bet that it is well understood, and the evidence of one +afternoon is that it was not. + +**The recommendation was to wait for evidence rather than for time**: a week of +ordinary use with no freeze and no lost transfer, or the equivalent in deliberate +runs of `transfer_probe.py --pull 3 --parallel` and `--operator`. + +**Overruled by the operator on 2026-09-09, and the reasoning is better than the +recommendation it replaced.** The concern was that a flag day removes the +fallback exactly when defects are still being found; the answer is that the +fallback only has value while more work is coming down the same path, and none +is — the next work is a music application with playlists, which touches none of +this. The prerequisite this section named was satisfied first: +`GET /v1/hub/version` already carried `client.minimum`, and the desktop client +now reads it before connecting instead of meeting a handshake refusal it has no +vocabulary for. + +Nothing about §4's reasoning has changed. What changed is the confidence that +the thing being made compulsory works. + +### 12.4 The silent row — fixed + +**The transfers panel showed nothing while the target was being opened.** The +row was created after `_openDownloadTarget` returned, which can take tens of +seconds, so three clicks produced no icon at all and then several rows at once. + +It was "fixed" during the build by taking the lease first, which caused §12.1's +`not_taken_up` symptom and was reverted — on a wrong diagnosis, as it turns out: +the revocations were the missing `touch()`. The revert was right anyway (§8.1). + +The shape that worked is a `prepare` step in the store, distinct from `run`: the +row appears at the click, the target is opened, and the lease is asked for last. +The reservation about `showSaveFilePicker` needing a user gesture turned out to +be real, but not in the way expected — see §12.5. + +### 12.5 One dialog per file, and three downloads frozen behind it + +Reported from Chrome, after §12.4 shipped: selecting four files produced a Save +As dialog for the first, then — once that file had finished — a dialog for the +second, while the last two timed out. On a later attempt the three remaining +transfers simply froze. + +Three facts explain it, and only the third was a surprise: + +1. `for (const entry of selected) await downloadFile(entry)` used to serialise + the target openings **by accident**. Moving the opening into `prepare` + removed the accident and four pickers raced. Fixed by `_openTargetInTurn`, + an explicit queue on the targets — never on the rows, so every download + still appears at the click. +2. A browser grants one file picker per user gesture, and selecting four files + is one gesture. The code already recovered from the `SecurityError` Chrome + throws for a picker with no gesture behind it, by streaming instead. +3. **Chrome does not throw.** It shows the dialog anyway and waits for a human. + So the recovery in (2) was never reached, and the queue from (1) turned the + unanswered dialog into a head-of-line block: the third and fourth downloads + were not frozen, they were waiting correctly for a dialog nobody had been + shown yet. From the panel that is indistinguishable from a freeze. + +The fix is to stop asking. `_openTargetInTurn` marks everything that has to wait +its turn as `batched`, and a batched opening prefers the streamed path whatever +the download mode says. The first file of a batch — the one that actually holds +the gesture — still gets its dialog, so the preference is honoured where it can +be. For the rest there is no gesture left to spend, so nothing is lost by +streaming: the file still lands on disk, in the browser's own download folder, +written as it arrives. Only the choice of folder goes, and it was not on offer. + +If the worker does not answer, a batched download falls back to the dialog +rather than failing — asking is better than losing the file (§ "a preference +must not cost a capability", which applies to the fix as much as to the bug). + +Pinned by `test_targets_are_opened_one_at_a_time` (the queue, and that the first +opening is the only unbatched one) and by four cases in `test_memory_ceiling.py` +(the branch itself, both fallbacks, and that batching never pushes a large file +into memory). + +**What that left, and what was measured.** With the queue unblocked all four +files downloaded, but a dialog still appeared for each one — with MeshBay's own +setting on "save automatically", which should never prompt. That points at the +streamed path failing, so it was measured rather than reasoned about: a real +Chrome 152 driven over CDP against the deployed hub, running the actual flow +(a `TransformStream` posted to the worker, a hidden iframe on `/_mbdl/<id>`, a +wait for `mbdl-serving`). It is served in 2–3 ms on a normal load, after a hard +reload, and twice in the same document. Ctrl+F5 does leave the page +uncontrolled — `navigator.serviceWorker.controller` is null and no +`controllerchange` arrives — but the `mbdl-claim` recovery already in +`_claimController` gets control back inside 3 s. Both probes are in +`QE/`-style scratch scripts, not in the suite: they need the deployed hub. + +**And the queue itself became the next defect.** Serialising the openings was +new in the same commit, and on Firefox it regressed what had always worked: +four downloads that opened their targets at the same time began waiting on the +slowest, and all four sat at "preparing" — the node journal showing +`d=0/8(q0) u=0/8(q0)`, not one transfer opened, so the block was entirely in +the client before any slot was asked for. Measured on Firefox 154 against the +deployed hub: `register` and `ready` return instantly, the page is controlled, +and four serialised openings are served in 5–18 ms — so the streamed path was +not the delay either. The queue was. + +Two bounds fix it, and both are narrowings of the queue rather than of any +capability. Only an opening that could actually show a dialog joins the queue, +which on Firefox and Safari — no `showSaveFilePicker` at all — means none of +them do, restoring exactly the previous behaviour. And no opening waits behind +another for longer than `TARGET_QUEUE_BUDGET_MS` (90 s), because `_targetQueue` +is never reset and an opening that never settles would otherwise leave the page +unable to start a download again until it is reloaded. Releasing early is safe: +whatever is ahead is still the only unbatched opening, so the released one takes +the streamed path and opens no second dialog. + +The general shape, third time in this section: **a queue is a way to convert one +slow participant into several stuck ones.** Every queue added here needs to say +what it protects and refuse everything it does not protect, and needs a bound. + +### 12.6 The wait with no deadline + +Bounding the queue was not enough: Firefox still showed four rows at +"preparing", with the queue bypassed, so each opening was hanging on its own. + +`_claimController` had two waits with no deadline of any kind — +`navigator.serviceWorker.register()` and `navigator.serviceWorker.ready` — +while `SW_CONTROL_BUDGET_MS` bounded only the wait that comes *after* them. And +`_swPromise` is shared, so a single unsettled one of those left every download +on the page waiting on the same promise, for the life of the page. + +Measured on Firefox 154, on a local `127.0.0.1` site so no hub was involved: + +| worker | `register()` | `ready` | +| --- | --- | --- | +| that installs | 8 ms | 0 ms | +| **whose install handler rejects** | **7 ms** | **never settles** (still pending at 10 s) | + +That is the whole mechanism. `register()` resolves as soon as the registration +object exists — carrying nothing but an *installing* worker — and `ready` is +what waits for an active one. A worker that cannot install therefore produces a +registration that looks fine and a `ready` that never comes. + +Every wait in `_claimController` is now inside one budget, with two carve-outs +that exist so a deadline never costs a capability. A `ready` that times out +while `registration.active` is set is not fatal — `ready` may be waiting on a +newer worker that cannot install while an older one serves perfectly well. And +the `mbdl-claim` recovery keeps its own budget outside the deadline, because +giving up there would cost Firefox the only unbounded way it has to write a +download to disk. + +**A deadline alone would still not have been a fix**, only a better-explained +failure: a registration stuck with nothing but an installing worker does not +heal, and every later visit finds the same one. So when `ready` times out with +no active worker, the registration is discarded (`unregister()`) and asked for +once more with a fresh budget. The page repairs itself instead of needing +developer tools. + +Pinned by four cases in `test_streamed_download_reliability.py`, each checked +against the unfixed source: a worker that never installs, a registration that +never answers, a stuck `ready` that must not throw away a working worker, and +the discard-and-retry. + +So the streamed path is not what is prompting. Exactly three things can open a +dialog per file on Chrome in automatic mode: our `showSaveFilePicker`, the +folder permission bubble from `ensurePermission` (only when a folder was granted +and the permission lapsed, and it is not a Save As), and **Chrome's own +"always ask where to save each file"** in `chrome://settings/downloads` — which +applies to the worker's response because it carries `Content-Disposition: +attachment`, and which no download mode of ours can override. The `console.info` +added with this fix distinguishes them in one line: it is written before every +dialog we open, so a dialog with nothing in the console is the browser's. + +### 12.7 The one none of the above explained: a hard reload + +Every fix in §12.5 and §12.6 was real, and none of them was the defect being +reported. Downloads on Firefox failed with "the worker did not answer the +download within 15s", every time, for the operator, while the *same profile* +driven from this machine — headless and headed, on their display, with their +files, through the real interface — succeeded every time. + +The operator's own sequence found it, and it is worth writing down verbatim +because no automated run could have produced it: a freshly started browser +downloaded four files out of four, twice; one **Ctrl+F5** and every attempt +afterwards failed; restart, fine again; Ctrl+F5 before any attempt and the very +first one failed. + +**A document fetched by a hard reload is loaded with the service worker +bypassed.** It can still be claimed afterwards — so +`navigator.serviceWorker.controller` comes back and every check in +`_claimController` passes — but the navigations that document starts keep +missing the worker, and the hidden iframe a streamed download needs *is* a +navigation. On Firefox and Safari that is the only way to write a file too +large to hold in memory, so the download cannot happen at all, for the life of +that page. The same behaviour had already been measured on Chrome in §12.6's +work and its significance was missed. + +The hard reloads were on this author's instruction, after each deployment. The +SPA's HTML is served `no-store`, so an ordinary reload has always picked up a +new build and Ctrl+F5 was never needed for anything. + +**Why the measurements could not find it.** WebDriver cannot perform a hard +reload — the key event goes to the content, not the browser chrome — so every +automated run tested, over and over, the one case that works. A fix is validated +by tests; a diagnosis is not always validated by automation, and the operator's +manual sequence should have been asked for hours earlier. + +The remedy is to stop inferring servability from control and to ask instead. At +boot the client opens a four-byte stream and a hidden iframe, exactly as a real +download would, and tears both down. If the worker does not answer, the page +reloads **once**, ordinarily, which puts it back under the worker; the flag is +in `sessionStorage` because it has to survive the reload it triggers and must +stop rather than loop. Two delays then had to go, because the repair first +landed about thirty seconds late — long enough to click and watch four rows +hang: the claim is asked for *before* the control budget rather than after it +(a page uncontrolled beside an active worker is never claimed on its own), and a +download that starts while the self-test is running waits for it instead of +racing it. + +And the refusal message no longer tells a Firefox user to switch to Chrome for a +state an ordinary reload undoes. All ten catalogues say to reload first. + +--- + +## 13. Decisions — all settled (operator, 2026-09-08) + +Nothing in this plan is waiting on an answer. + +| # | Decision | Consequence | +|---|---|---| +| 0 | **Browsing is never subject to a slot** — posters, thumbnails, covers, previews, listings | §3.4, structural: a transfer is what the widget shows | +| 1 | **MNP 3.0, hard refusal of 2.x at the handshake** | No leaseless branch survives anywhere. Flag day, stage 4 | +| 2 | **Per-member default of 2 applies to existing groups** | No release note, no migration prose — the project is in development and every node is a test node | +| 3 | **Chat attachments take an upload slot like any other upload** | §9.3: the composer must show the waiting state, or the paperclip looks broken | +| 4 | **Pause on the service-worker path (Firefox, Safari) is deferred, not abandoned** | §6.5 says what it would actually take. It is cost, not impossibility | +| 5 | **The two big caps stay node-wide** | The resource is the machine's. A per-group variant can be added later with no protocol change | + |