diff options
| author | Christophe Besson <cbesson@gmail.com> | 2026-09-10 18:44:40 +0200 |
|---|---|---|
| committer | Christophe Besson <cbesson@gmail.com> | 2026-09-10 18:44:40 +0200 |
| commit | 676df16e90f894ba034a943d669789efb6721449 (patch) | |
| tree | 60d300a7b029d9516b5d40ed03ff0a9720ae8d26 /docs | |
| parent | 07ff8b4f6143039fcc74b8cf7c423282bce093c1 (diff) | |
| download | meshbay-676df16e90f894ba034a943d669789efb6721449.tar.gz | |
docs: MESHBAY_DESIGN.md — one reference for the design
Thirty documents under docs/ described this system between them: two
architecture drafts, three security reviews, eleven design notes, a roadmap and
a decisions file. Reading any one of them meant following cross-references into
four others, and several were flatly wrong — the keystore KDF, the protocol
version, and four features whose headers still said "not implemented" months
after they shipped.
This is the synthesis. It states design rather than history: a section says why
the node wraps the group key itself, not which finding made it necessary.
Development history, spikes and reversed directions are gone. The security
findings survive as section 13, where each label names the invariant it stands
for today rather than the defect it was reported as.
Two things it is careful about, because hundreds of code comments depend on
them. Every short label — C1, H3, NS6, T3, C5b, W2, E9, F1 — is defined in
section 13, including the three colliding namespaces (each review numbered its
findings from C1, and the code means the second review's). And section 16 maps
every "<doc> section n" reference the code makes onto its replacement, so no
comment has to be edited to stay resolvable.
transfers-v1.md comes in from outside the tree with it. The lease design is
section 5.5; what a synthesis cannot carry is that document's failure-mode
analysis — every way a slot can be lost, every way a client can be left
waiting — and what a live pass found after the work was called done.
Every claim was checked against the code rather than the drafts.
Suites green: 2256 passed, 4 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YVoHVCcfBqud6ZjG4db3y7
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/MESHBAY_DESIGN.md | 2628 | ||||
| -rw-r--r-- | docs/transfers-v1.md | 1159 |
2 files changed, 3787 insertions, 0 deletions
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md new file mode 100644 index 0000000..73f6b42 --- /dev/null +++ b/docs/MESHBAY_DESIGN.md @@ -0,0 +1,2628 @@ +# MeshBay — Design + +> **Status: the specification.** This document states what MeshBay is, how it is +> built, and why each part has the shape it has. It replaces the draft series +> (`meshbay-draft-v5.md`, `meshbay-draft-v6.md`, `old-draft.md`) and the design +> notes that grew around it; §16 maps every reference those documents and the +> code make onto a section here. +> +> **The convention that is not negotiable: a claim in this document names the +> adversary it holds against.** A property that holds against a passive hub and +> not an active one is written that way. A section that says "this buys nothing" +> is as load-bearing as one that says it buys something. +> +> **The second convention: this document states design, not history.** Where a +> label like `C1`, `NS6` or `T3` appears — and hundreds of code comments cite +> them — it names the invariant that holds today, not the incident that produced +> it. §13 is the register of those labels. +> +> Wire versions at the time of writing: **MNP 3.0** (oldest peer accepted 3.0), +> **MHP 0.1**, packages **0.13.0**. The normative source for the wire format is +> `MESHBAY_NODE_PROTOCOL.md`; this document states the design the protocol +> serves, not its byte layout. + +--- + +## 0. How to read this + +### 0.1 Scope + +| Read | For | +|---|---| +| **this document** | the architecture, the trust model, and the reason each decision is what it is | +| `MESHBAY_NODE_PROTOCOL.md` | the MNP wire format, message by message | +| `transfers-v1.md` | the transfer system's failure-mode analysis, kept because a synthesis cannot carry "every way a slot can be lost" | +| `playlists.md` | the playlist design in full — decided, not built (§9.10) | +| `USERGUIDE.md` | how to use and operate a deployment | +| `PACKAGING-GUIDE.md`, `QUICKSTART.md`, `HTTPS.md`, `MAIL-SERVER.md`, `windows-build.md` | installation and server operations | +| `CLAUDE.md` | project conventions and the engineering lessons that govern how changes are made | +| `old-draft.md` | historical archive of drafts v1–v4. Nothing there is authoritative | + +### 0.2 Reference labels + +Code comments, tests and commit messages cite short labels — `C1`, `H3`, `NS6`, +`T3`, `C5b`, `W2`, `E9`, `F1`. **Every one of them is defined in §13**, stated as +the rule it names today. + +Three label families collide, and the register keeps them apart: + +- **`C*` / `H*` / `M*` / `L*` with no qualifier mean the second review's numbering** + (§13.3). That is the numbering the code uses. The first review's `C1`/`C2` and + the third review's `H1`/`H2`/`M1`–`M6`/`L1`–`L11` are always written with the + review named (§13.1, §13.4). +- `M2a`, `M2b`, `M2c` and `C5a`, `C5b` are sub-items and are unique across the set. +- **`D1`–`D4` are the client-architecture decisions** (§14.2); **`D1`–`D12` written + as "Stage D*" are desktop-client build stages** (§15.2). + +### 0.3 The one-sentence version + +> **The hub cannot read your content unless it ships you malicious client code — +> and against a native client it cannot do that undetectably.** + +Everything below either supports that sentence or states precisely where it stops. + +--- + +## 1. What MeshBay is + +### 1.1 Positioning + +MeshBay is a platform for **private, encrypted, self-hosted groups with an +application store**. A group is a set of people, a set of directories on +somebody's machine, and a set of applications over them — chat, a file explorer, +a video library, a music player, a photo album. + +It is **not** a public file-sharing network. Public groups exist as an optional +hub feature and are switched off on the reference deployment (§7.4). The design +optimises for the private case throughout: the hub keeps no file names for +private groups, registers no content hashes for them, and holds no key that +opens them. + +### 1.2 The three parties + +``` + ┌─────────┐ accounts, group registry, signaling relay, ┌─────────┐ + │ hub │ notifications, moderation, instance policy │ peer │ + └────┬────┘ — never content, never a group key │ hubs │ + │ └─────────┘ + MHP 0.1 │ signalling (SDP/ICE, <1 KB), presence, revocation push MHP + │ + ┌────┴────┐ MNP 3.0 ┌──────────┐ + │ node │◄──────── WebRTC DataChannel / QUIC ──────────►│ client │ + └─────────┘ index, file chunks, streams, chat, admin └──────────┘ + holds the files browser SPA or desktop + holds the group key client; both are the + is the content authority same UI source +``` + +- **The hub** is a registrar and a signaling relay. It is in the trusted path by + choice, not by necessity (§14.2 D4), and it is never in the data path. +- **The node** is the daemon that holds a group's directories and its group key. + It is the **sole content authority**: it decides who is served, what is served, + and who may change anything about the group's content. +- **The client** is the browser SPA or the desktop application. Both are built + from one source tree (§8.4). + +### 1.3 The rule that settles arguments + +> **Group-related server state lives on the node. Always.** + +Files, indexes, members' devices, pending device requests, invitations, chat, +chat epoch keys, per-root availability, application settings, transfer limits — +all on the node. The hub holds accounts, the group registry and membership, +signaling, notifications, the moderation surface and instance policy, and nothing +else about content. + +This is decision **E9**, and it is the rule any new feature is measured against. +A feature that wants a row on the hub about a group's content is a feature that +has misunderstood the model. It has been re-verified at each content-model change: +`SwarmSource` carries a content hash, a node id and an endpoint — **no paths, no +filenames** — and private groups register nothing at all (**H7**). + +--- + +## 2. Trust model + +### 2.1 Adversaries + +Every claim below is written against one of these, and they are the only ones the +document uses: + +| Adversary | What they can do | +|---|---| +| **Passive hub** | Read everything the hub legitimately stores and relays | +| **Active hub** | Also lie: forge tokens, invent accounts, substitute values it publishes, ship modified client code to a browser | +| **Malicious node operator** | Read and alter everything on their own machine, including the plaintext files they host | +| **Malicious group member** | Everything a member may do, plus anything the protocol fails to refuse | +| **Network attacker** | Observe and tamper with traffic between any two parties | +| **Local attacker** | Reach loopback services and files on a client or node machine | +| **Registered hub user with no membership** | Reach every hub endpoint that does not check membership | +| **Federated peer hub** | Push directory rows and revocations over MHP | + +### 2.2 Security claims + +| Claim | Passive hub | Active hub | Malicious node operator | Malicious member | Network attacker | +|---|---|---|---|---|---| +| Data never transits the hub | ✅ | ✅ | — | — | ✅ | +| File content is unreadable | ✅ | ❌ **T3** (browser) · ✅ native | ❌ by design — the operator hosts the files | ❌ members share the group key | ✅ | +| The file index is unreadable | ✅ | ❌ T3 · ✅ native | ❌ | ❌ | ✅ | +| Chat content is unreadable | ✅ | ❌ T3 · ✅ native | ❌ — the operator is a member | ❌ | ✅ | +| Chat is unreadable **off a stolen disk** | ✅ | ✅ | ✅ without the keystore passphrase | ✅ | ✅ | +| Content cannot be modified | ✅ | ✅ | ❌ by design | ✅ | ✅ | +| The node cannot be impersonated | ✅ | ✅ | — | ✅ | ✅ | +| Client code integrity | ❌ **T3, accepted** (browser) · ✅ ships in the package (native) | ❌ T3 · ⚠️ native: **detectable, not prevented** | ✅ | ✅ | ✅ | +| The hub cannot obtain the group key | ✅ | ✅ **except** in an open-join group, where it can join legitimately (§7.3) | — | — | ✅ | +| Node content authority | ✅ | ✅ | ✅ sovereign | ✅ | ✅ | +| Devices cannot be added by the hub | ✅ | ✅ — the hub holds no user key and cannot countersign | ⚠️ a node adds a device only to itself, where it already reads everything | ✅ | ✅ | +| Chat senders are authenticated to each other | ✅ | ✅ | ⚠️ only for accounts the reader has already seen (§3.3) | ✅ | ✅ | +| Keypair bundles (**C4**) | closed for native devices | closed for native devices | ⚠️ **open for any account that also signs in from a browser** | — | — | +| Deleting your account erases you | ✅ hub-side | ✅ hub-side | ❌ files, pinned identity and bundle stay on the node (§7.7) | — | — | +| Your identity keys stay yours | ✅ | ✅ | ⚠️ offline attack on the bundle they hold — succeeds against a weak passphrase, and yields the identity used **on that node only** | ✅ | ✅ | + +### 2.3 What the project must not claim + +Three sentences are forbidden, each for a deliberate reason: + +- **"Everything is encrypted and unreadable by other parties, even the hub."** + A hub that ships the code can lift keys from the page regardless of protocol + design (**T3**). That is an artifact-level attack, not a silent directory lie, + and it is removed for native clients — not for browsers. +- **"A native client makes the hub untrusted."** It converts an undetectable, + per-request, per-user attack into a persistent artifact that can be hashed and + compared. That value is realised by reproducible builds and published hashes, + not by the packaging format. A build signed with a key the hub operator holds + *relocates* trust; it does not remove it. +- **"C4 is closed."** It is closed for a native device unconditionally, and open + for any account that also uses a browser. **An account is only as strong as its + weakest client.** + +"End-to-end" here describes **client ↔ node**, never client ↔ client. Members and +the operator read everything in their group; that is what a group is. + +### 2.4 One boundary worth naming + +An operator hosts your content by design. They should not be able to become +*you*. They can still try — a keypair bundle sits on their disk and a weak +passphrase gives it up — but what it gives up is **the identity you use with +them**, which unlocks nothing they did not already hold. Reading what they host +is by design; reading what *other* operators host is not, and does not follow +(§3.2). + +--- + +## 3. Identity + +### 3.1 Accounts + +An account lives on the hub: a username, an encrypted email address, a status and +a role. The passphrase never leaves the client. It derives **two independent +values**, both salted by the trimmed username: + +| Value | Derivation | Consumer | +|---|---|---| +| `auth_key` | PBKDF2-SHA512, 600 000 iterations, domain `meshbay:auth:v1:<user>` | hub authentication — the hub stores an Argon2id hash of it | +| `bundle_key` | Argon2id 128 MB / t=3 / p=1, domain `meshbay:bundle:v2:<user>` | AES-GCM key for the per-node identity bundle, held on each node | + +This split is **T1**, and its consequence is structural: **the hub never sees a +passphrase**, so the passphrase floor — 12 characters and roughly 60 estimated +bits — can only be enforced client-side, and is. + +The two values have different fates. The hub holds a verifier for `auth_key` and +can reset it from an email code. **Nobody can reset `bundle_key`**: the hub has +held no key material since the invite redesign, and cannot reach a +`keypair_bundles` row, which is served only over MNP to authenticated members. +That asymmetry is why passphrase change and passphrase recovery are two features +and not one (§3.6). + +### 3.2 Identity keys are per node + +A person's identity keypair is created **at first contact with a node**, +encrypted under `bundle_key`, and left on that node. It is never reused +elsewhere. + +The reason is blast radius. The adversary is concrete: an operator holding their +own node's disk, attacking a bundle offline at their leisure. What cracking one +yields is the identity that person uses **on that node** — where the operator +already holds the content, the index and every byte they serve. It is not a key +anywhere else: each node gets its own, and a key one node pinned is a stranger to +the next, which asks for a code like any first contact. + +Two consequences fall out and both are wanted. Two operators cannot tell they +host the same person by comparing keys. And **the hub stores and publishes no +user keys at all** — `users.pk_ed25519` / `pk_x25519` are dropped, `PUT /me/keys` +does not exist, and `/pubkeys` returns an account id and the node's linking key. +There is no directory to substitute from, which is what **H3** was. + +Two further rules follow directly: + +- **Tokens carry no `pk_user` claim.** A key chosen by the hub must never become + the identity a node records for an upload, or whoever issues tokens decides who + may delete a file. Attribution uses the roster pin. +- **There is no key rotation endpoint.** Rotation is per node: `member unpin` plus + a fresh code, which already exists. + +Registration therefore generates nothing, which has a pleasant side effect: a +scripted signup produces a real account, and a wiped hub and node can be taken to +a working demo without a browser. + +### 3.3 Devices + +One person may hold several devices on one node — a browser and a desktop client, +two laptops. `identities` is keyed by `(user_id, pk_ed25519)` with `label`, +`added_at`, `added_by_pk`, `revoked_at` and the evidence columns below. Pinning is +never `INSERT OR REPLACE`: a silent overwrite is a hole the moment a second key is +legitimate. + +**A new device is admitted by a key the node already pinned.** The authority is +therefore a key the node established locally, exactly as for operator +authorisation — and **the hub cannot produce it**, because it holds no user keys. +Device linking adds no hub-reachable authority. + +The binding is a **one-time code the new device generates and displays**, hashed +together with the new keys: `code_hash = sha256(code ‖ new_pk_ed25519 ‖ +new_pk_x25519)`. The approving device asks the node for this account's pending +requests **with their stored hashes** and recomputes the hash for each until one +matches. + +> **The code never reaches the node.** That is what makes a substituted key +> impossible rather than merely detectable: a node offering fabricated keys would +> have to produce a hash over a code it has never seen. + +The approval is deliberately **not** a human comparing digits. Safety numbers were +evaluated and refused, permanently, under decision 19: **no new code exchanges +between people.** A device-linking code is a code between a person's own devices, +which is a different thing and was already accepted. + +Bounds, all of them anti-abuse rather than the security boundary — the security +comes from the hash binding: + +| Bound | Value | +|---|---| +| Request TTL | 1 h, `[node] device_request_ttl_minutes` | +| Devices per account per node | 5 | +| Attempts per connection | 5, then a node-wide lockout | +| Filing a request | requires the account to have at least one pinned identity already | + +`member unpin <user>` removes **every** device. `device revoke` marks one rather +than deleting the row, because a deleted row is a key the node would happily pin +again. + +**Which device is on a connection** is proved separately from which account. The +handshake proves the account; `device_hello` — signed over a transcript naming the +node, the group and this connection's nonce — proves the device, and is refused +unless the key is a live device of that account in the node's own roster. Without +it the node would fall back to the account's oldest key and record it as the +author of everything. + +**Members can verify each other's device keys** (Tier 2). `group_roster_req` / +`group_roster_resp`, sealed under the group key, answers **any member** with, for +each live device of each active member: the key, the key that countersigned it, +and the signature, nonce and timestamp that prove it. The client walks the chain +itself — the node decides nothing, because the node is the party the property +holds against. A device the node lists but cannot evidence never enters the +verified set, so a fabricated key is not laundered in by being mentioned. + +Three rules keep that honest: + +- **A root is a device that names no countersigner**, not one that fails to + produce a signature. Treating "no proof" as "root" would admit anything a node + chose to write. +- **First sight pins everything the node says**, not the verified subset — an + alarm that fires on legitimate second devices stops being read, and the budget + for this whole feature is exactly one notice: *"this account's key changed"*. +- **A device pinned before the evidence columns existed is unevidenced and reads + as such.** + +The property, stated exactly: + +> Once a member's client has seen an account, **a node that later substitutes a +> key for it is detected.** Nothing is gained at first sight, where the client +> has nothing to compare against. + +That second sentence is not a caveat to be dropped. Closing first sight needs an +attestation rooted outside the node — an operator-signed roster (Tier 3), which is +deferred with nothing depending on it, and worth building only where the operator +is not the machine. + +**The cost, which is real:** the roster is member-visible, so every member of a +group learns how many devices every other member holds and what their public keys +are. It stays inside the group, the hub is not involved, and it is scoped to one +group. A member who cannot see the keys cannot check them, so this is not +avoidable. + +**Where device linking does not hold:** an approval performed *in a browser* +inherits **T3** — the hub serves that browser its code and can read the typed +code. The first browser-to-native link is the moment of highest exposure for an +account, and it happens once. An account created natively does that first link in +the safe direction. + +### 3.4 Admission: invitations and pairing codes + +**The node wraps the group key**, for a key the recipient proved possession of, +over an authenticated channel, bound to an identity the operator admitted with a +one-time code the hub never sees. + +``` +operator (SSH) meshbay-node member invite bob → CODE R3H8-TB6V + (or the same from the group's Settings tab, signed by the paired browser) +operator sends the code to bob out of band +bob opens the group; the client holds no group key +bob → node join_request {pk_ed25519, pk_x25519, code, sig} ← pre-proof window +node code valid for this account → pin the identity, admit to the group +node → bob the group key, wrapped for the X25519 key bob just proved he holds +``` + +Four properties, each load-bearing: + +1. **No public key is ever fetched from a directory.** The joiner's keys arrive + from the joiner, both signed together in one transcript (`meshbay:join:v1`), so + the identity key vouches for the encryption key. +2. **The code binds a key to an account**, and the hub never sees it. 40 bits, + Crockford base32 rendered `XXXX-XXXX`, single use, valid for exactly one + account in one group, stored only as `sha256(code)`. A password KDF over 40 + uniformly random bits would buy nothing. Guessing is bounded by 5 attempts per + connection and a node-wide lockout, and every attempt is an audit event. +3. **The node's roster is the authority**, not hub membership. A hub that invents + an account, adds it to a group and mints it a token gets + `not_authorized_for_group`. +4. **Wrapping happens on every connection.** Nothing is stored per member, so key + rotation propagates by itself and revocation actually takes effect. (Rotating + the key after a revocation is still required — the ex-member holds the current + one, and no protocol can take that back.) + +Node authority is established the same way, once per node: `meshbay-node operator +pair` prints a code, the operator types it into their own browser, and the node +pins that identity. **It is never learned from the hub** — a hub able to name the +operator's key could install itself as node administrator, which is **NS4**/**M3**. + +Code lifetimes differ because the acts differ: + +| Code | Default | Setting | +|---|---|---| +| Member invitation | **7 days** | `[node] invite_ttl_hours` | +| Operator pairing | 24 h | `[node] pair_ttl_hours` | +| Device add request | 1 h | `[node] device_request_ttl_minutes` | + +An invitation waits for someone to read their messages; a pairing code is typed +during the SSH session that printed it. + +**Why a code and not something lighter** — the question is what stops the hub from +being bob on his first connection: + +| Option | What an active hub can do | | +|---|---|---| +| Wrap for whatever key the peer presents | Forge a token for bob, present its own key, receive the key | worse than nothing | +| Bind to the key the inviter fetched from the hub | Substitute at invite time | **H3**, relocated | +| TOFU: first connection wins | Race the real bob with a forged token | small window, total consequence | +| Safety-number comparison | Nothing — but it needs two humans reading digits at the worst moment | correct, unusable as a default | +| **One-time pairing code** | **Nothing: the code never reaches the hub** | **adopted** | + +**Delegation is designed and deferred.** `invite_create` is authorised as a *role* +check against the roster rather than an equality test against the operator, and +the `delegate` role value is reserved, so a group admin who does not run the node +becomes a roster row and a CLI command — no protocol change, no migration. + +### 3.5 Open-join groups + +A group whose `join_policy` is `open` pins on first contact (TOFU) and wraps the +key immediately. A code there protects nothing — the hub can create an account, +join through the front door, and be a legitimate member — so it would be pure +friction. + +Stated plainly, per the convention: **in an open-join group the hub can obtain the +group key.** That is a property of open joining, not a defect of this design. +Content in such a group is protected from the network and from non-members, and +from nobody else. + +Note the axis. **`visibility`** (public/private) controls discoverability and swarm +hash registration (**H7**). **`join_policy`** (open/request/invite) controls +admission. Only the second decides whether a code is required: a public group with +`join_policy = "invite"` keeps the code, because being findable is not being open. + +**`join_policy` is read from `node.toml`, never from the hub.** A hub able to +declare a group open would be handed its key. An unknown group reads as `invite`. + +### 3.6 Passphrase change and recovery + +**Changing a known passphrase** re-wraps every reachable node's identity bundle +from the old `bundle_key` to the new one **before** touching the hub — if the +fan-out fails, the account is unchanged. Only then is `POST /v1/users/password` +called with the old and new `auth_key`. Nodes that were unreachable are named to +the user, with the operator fallback (`member unpin` plus a fresh code) as the way +to fix each one. Every refresh-token family is revoked. + +**Recovering a lost passphrase** splits into what each key can reach: + +| | Recovered by | +|---|---| +| Hub login (`auth_key`) | an email code alone | +| Per-node identity keys — group key unwrap, provable upload ownership, chat identity, device countersigning | the **recovery key**, per reachable node | +| An identity on a node with no recovery-wrapped copy, or offline at recovery time | operator `member unpin` plus a fresh code | + +The recovery key is a full-entropy 32-byte secret `R` **generated by the client**, +rendered as a grouped mnemonic. `recovery_key = HKDF-SHA256(R, info = +"meshbay:recovery:v1:" + username)` — HKDF and not Argon2, because `R` has 256 bits +and there is nothing to brute-force. Every time an identity bundle is written to a +node, a **second copy** is written beside it wrapped under `recovery_key` +(`bundle_enc_recovery`, additive on the wire). `R` is a pass-through: offered in +the registration email by default, never written to any database, never logged. + +The reset endpoints are built to leak nothing. `POST /v1/users/password/reset-request` +requires **the username and the email on file as a pair**, checked against a blind +index and never decrypted; a mismatch, an unknown username and a non-active +account all take the identical no-op path and return the same `200 +{"status": "sent_if_exists"}`. So it cannot be used to spray reset mail at an +inbox from a username alone. + +A reset **deletes every `UserDevice` row** on the hub. Three different things are +called "device" here and only one is touched: + +| | What it is | A reset | +|---|---|---| +| `user_devices` (hub) | an Ed25519 key that lets a client skip the passphrase prompt on launch. A hub-login convenience — no group key is wrapped for it | **deleted** | +| per-node identity (`identities` on each node) | the keys that unwrap the group key, prove upload ownership and sign chat — **this is group access** | **recovered** from the recovery copy, or via the operator fallback | +| the roster pin | which identities a node has admitted | untouched | + +Deleting `user_devices` costs one passphrase prompt per client, which is the +point: after a "control may be lost" event, a laptop still carrying a stored +hub-auth key must stop signing in on its own. + +### 3.7 The keypair bundle, and what it is worth (C4) + +A bundle carries **one node's** identity keys, encrypted under the owner's +passphrase, stored on that node. It is what lets a second browser open the same +account there — the ordinary expectation, and the only mechanism available to a +browser, which keeps nothing durable of its own. + +**Why Argon2id.** PBKDF2 is compute-only, which is exactly what a GPU is good at. +Measured: PBKDF2-SHA512 600k costs 241 ms per guess on one core, Argon2id 128 MB +/ t=3 costs 88 ms — the defender pays *less* — but only one of them forces an +attacker to find 128 MB per guess. + +**The honest size of the gain.** On a single high-end card the ceiling moves from +roughly 8k guesses/s to roughly 2k: a factor of four, not a thousand. What it +really buys is the cost of scale — 128 MB per lane caps a 24 GB card near 187 +concurrent guesses and makes custom hardware unattractive, where SHA-512 silicon +is cheap. + +**The passphrase decides this, not the KDF.** At ~2k guesses/s a +dictionary-and-rules run of 10⁹ candidates takes about six days on one card. Four +random words (~52 bits) outlasts the sun. No parameter choice saves a weak +passphrase; it only moves it from hours to days. + +Operational facts that constrain changes: + +- Argon2id runs in **WebAssembly, vendored** under `static/vendor/` with its + provenance. The CSP forbids external hosts and must keep `wasm-unsafe-eval` in + `script-src`. +- **Never change these parameters in one place.** `keyderive.js`, `keyderive.py`, + the desktop client and the test harness are held byte-identical by + `test_bundle_kdf_parity.py`. A mismatch does not look like an error — it looks + like an account nobody can open. +- Bundles carry an `MBK2` marker; an older PBKDF2 form is still readable and is + re-encrypted on the next backup. +- Cost is paid **once per sign-in** (≈650 ms bundle + ≈239 ms `auth_key`). + Reloading a page derives nothing: the key lives in IndexedDB. +- The pre-proof window that serves bundles is bounded (4 fetches) and audited. + +**C4 is reduced, not closed.** Bundles still sit on disks their owner does not +control. It closes for a native device unconditionally, because that device's key +is in no bundle anywhere. It closes for an *account* only when no browser needs a +bundle on that node — which needs `device_policy {allow_bundle: false}`, **signed +by a pinned key** so the decision is the user's and never the hub's (open item +O3). + +--- + +## 4. Cryptography + +### 4.1 Key hierarchy + +``` +User identity key Ed25519 signing, authentication — per node (§3.2) +User exchange key X25519 key agreement — per node +Group encryption key AEAD 256-bit content and index encryption — the group secret +Chat epoch key 32 bytes per group, per epoch — node-generated (§4.5) +Session keys X25519/HKDF per-connection, from DTLS/TLS +``` + +Every private key lives in an encrypted keystore on the machine that owns it. The +hub never sees one. + +**Domain separation is consistent and mandatory.** Every derivation uses a +distinct `info` string, and the AES variant adds an `:aes` suffix so two ciphers +can never derive the same key from one group key. This is a small detail that +prevents cross-protocol key reuse, and it is checked rather than assumed. + +### 4.2 Group key wrapping (ECIES) + +``` +wrap: sk_eph, pk_eph = X25519.generate() # fresh per bundle + shared = X25519(sk_eph, pk_recipient) + wrap_key = HKDF(shared, salt=pk_eph, info="meshbay:gek_wrap:v1", len=32) + wrapped = AEAD(wrap_key).encrypt(nonce, gek, aad=pk_recipient) + bundle = pk_eph ‖ nonce ‖ wrapped + +unwrap: shared = X25519(sk_recipient, pk_eph) # same derivation +``` + +Three properties are why this shape: + +- **The ephemeral keypair is fresh per bundle**, so the same key to the same + recipient produces different ciphertext every time. +- **The AAD binds the bundle to its recipient**, so a bundle reused for a + different member is rejected by the tag rather than by a check somebody has to + remember to write. +- **A wrong private key fails at the AEAD tag** — an immediate, unambiguous + refusal. + +The node produces every copy of the key itself, from its own CSPRNG. **Nothing +arriving over MNP can activate a group key** (**C5b**). Read that precisely: it +targets *key material arriving from outside*, not the instruction. An +operator-signed `gek_rotate` where the node generates the key is a different shape +and is allowed. The initial `gek-init` stays local, because with no key there is +no completed session to carry a signed op. + +### 4.3 On-the-fly encryption + +Files are stored **in plaintext on the operator's disk** and encrypted at read +time. This avoids double storage and makes key rotation feasible without +re-encrypting terabytes. + +``` +disk (plaintext) → compress → per-chunk AEAD under a group-derived key → transport → client +``` + +- Chunk size 1 MB: amortises AEAD overhead and enables seeking, because each chunk + is independently decryptable. +- `chunk_key = HKDF(GEK, salt=None, info="file:" ‖ blake3(file) ‖ ":chunk:" ‖ index)`. + The salt is omitted deliberately: the group key is CSPRNG output and already + uniform, so the file and chunk context belongs in `info`, which is the correct + HKDF usage (**M5**, first review). +- **Chunk authentication is the AEAD tag**, not a per-chunk signature. The tag + authenticates the ciphertext under a key only members hold, which is what the + signature was for. +- Compression precedes encryption, because compression is ineffective on + ciphertext. +- Upload chunk size is 48 KB, which is what fits the SCTP limit after msgpack + overhead. + +Crypto is not the bottleneck: encrypt-and-send costs single-digit milliseconds per +megabyte against tens to hundreds for the network. + +### 4.4 The group envelope + +`meshbay_common/groupbox.py`, mirrored by `sealGroup`/`openGroup` in +`static/crypto.js`, is **one envelope with purpose-separated subkeys** derived +from the group key. AAD is `"<msg_type>|<group_id>"`. + +| Purpose | Info string | Carries | +|---|---|---| +| `index` | `meshbay:index:v1` | `index_sync`, `index_delta` | +| `ack` | `meshbay:ack:v1` | the handshake ack's configuration payload | +| `upload` | `meshbay:upload:v1` | `file_upload` / `file_upload_ack` | +| `chat_keys` | `meshbay:chat_keys:v1` | chat epoch key delivery | +| `roster` | `meshbay:roster:v1` | `group_roster_req` / `resp` | + +**Never reuse the chunk key with a pseudo-file for this.** The purposes are what +keep two message classes from sharing a key. + +Three things about the sealed messages must stay straight: + +- **Sealing the ack line is integrity, not confidentiality.** The handshake + transcript names no ack field, so `is_node_admin`, `enabled_apps`, the roots + table and the rest were authenticated by the channel alone. The AEAD tag comes + from a key the hub does not hold. +- **Sealing the index is defence in depth against our own next bug**, of a class + that has shipped twice (**C1**, **C6**). It buys nothing against an observer, + the hub or a member. That is the whole claim. +- **A payload that does not open ends the session**, never a default. An + unopenable `enabled_apps` reads as "the operator disabled every app" and an + unopenable index as "the group is empty" — both indistinguishable from + legitimate states. + +What stays in clear, and none of it is content: the handshake itself; `type`, `v`, +`group_id` and the ack's `node_pk`/`proof`/`sig`, because a receiver must route and +**authenticate** before it would trust a decryption; `upload_id` and `chunk_index`, +because the node routes and orders on them; `index_progress`, which is counters +only, every 2 s; transfer-lease fields, which are an opaque id and two numbers +(§5.5); and the media-metadata replies. + +### 4.5 Chat encryption + +Chat is encrypted with one key **per group, per epoch, per device**, derived by +name from an epoch key the node generates and delivers wrapped under the group +key: + +``` +epoch_key 32 random bytes, generated BY THE NODE (C5b) +device_key(d) = HKDF(epoch_key, info="meshbay:chat:dev:v1|"+group_id+"|"+d) + where d = base64(device pk_ed25519), the roster's own identifier +``` + +Every member derives every device's key from the epoch key, so **nothing is +distributed per device and nothing is stored per device**. + +**There is no mutable sending state at all.** That is the point, and it is a +stronger guarantee than per-device chains rather than a weaker one: a shared +chain advanced by two clients produces key and nonce reuse (first review **C1**, +one level down), and a design with no sending state cannot have that hazard. + +> **Two clients of one account normally share a device key.** A second browser +> recovers the existing identity from the keypair bundle rather than minting a new +> one; device *linking* is the exception, not the rule. What makes that safe is +> the nonce, not the derivation: **96 random bits, never a counter.** Two +> independent senders under one key collide only on the birthday bound, which at +> chat volume is unreachable; two senders advancing one counter collide +> immediately. + +Each message is **sealed and then signed over the ciphertext** with the device key +the node pinned, so a receiver verifies before decrypting and anyone holding the +roster can verify. The AAD binds the group and the epoch, so a ciphertext cannot +be replayed into another group or attributed to another epoch. `sender_name` lives +**inside** the sealed payload — as a wire field it was free to spoof. + +`sender_id` stays a clear field **set by the node from the authenticated session** +(**NS6**). It is what the store keys on and what the UI groups by; it is not what +authenticates the message. The *device* claim is checked against the connection's +own `device_hello`, or a member could sign as anyone. Replay is refused by a +unique `(device, nonce)` — a replay is a validly signed copy, so nothing about the +signature refuses it. + +**Epochs.** A new epoch is opened when, and only when, the set of devices that may +read *future* messages shrinks: `member revoke`, `member unpin`, `device revoke`, +`gek_rotate`, or an explicit `chat rotate`. Epoch 1 is opened at group load — a +group with no epoch is a group nobody can speak in. + +**Old epochs are kept and still delivered.** That is what keeps history readable +to everyone who could already read it, and it is why rotating the group key is a +**re-wrap** rather than the destruction of the whole archive: the archive is not +encrypted under the group key, only the epoch keys are wrapped with it in transit. +Nothing anywhere deletes an epoch. Epoch keys are stored ECIES-wrapped to the +node's own X25519 key in `bundles.db`, never raw — a plaintext key store beside +`chat.db` would collapse the threat model silently, and it is the obvious thing to +write. + +**Why not a ratchet.** Under group-key distribution *and* server-served history, +the node must retain each chain's **earliest** key, and a chain key at iteration +*i* yields every message key from *i* onward by pure HKDF. **Forward secrecy is +therefore zero either way.** What a ratchet was left buying is a large amount of +stateful client code with silent failure modes, three of which are concrete: any +member could sign as any other, a second device dropped the first's chain, and the +skipped-key cache grew without bound (§13.6, F1–F3). Forward secrecy is given up +**deliberately and on the record**. If it ever becomes a real requirement it +belongs in 1:1 DM, where there is no server-side history to contradict it. + +**Neither a sender-key nor a ratchet implementation exists in the tree.** Both +were written, neither was ever called, and both are deleted — the reasoning that +ruled them out lives at the top of `chatbox.py`, the module that replaced them, +where it stands on its own instead of pointing at a file to compare against. + +> **Kept code that nothing calls is worse than absent code.** It reads as an +> alternative somebody may reach for, its green tests read as evidence of a +> protection that is not in the product, and it has to be maintained past every +> refactor to stay compiling — maintenance spent on a decision already made. If +> forward secrecy ever becomes a real requirement, it belongs in 1:1 DM, where +> there is no server-side history to contradict it, and it starts from the +> requirement rather than from a module somebody left behind. + +**What chat encryption protects against, in the words the user-facing docs should +use:** someone who obtains the node's storage **without the keystore passphrase** — +a hosting provider imaging the machine, a leaked backup, a seizure where the +passphrase is not surrendered. It does **not** protect chat from the operator or +any current member (they hold the group key, and the chat key is delivered under +it); from anyone holding any one device of any member; from a former member, for +messages sent before the epoch changed; from the hub as regards *metadata*; or +from the node as regards *links posted*, which it fetches to unfurl. + +**Deliberately not encrypted**, stated so nobody reads more into the feature than +it does: `sender_id`, timestamps, message sizes and the fact of a message are in +the clear to the node, which is the relay and cannot route otherwise. The hub +learns per message the group, the time and the sender's account id, so it can skip +the author when creating notifications — it can build a social graph with timings +without reading a word, and that is a known metadata leak rather than a solved +problem. Attachments are ordinary files on a root and stay plaintext on disk; the +*reference* to one is inside the sealed payload, but the file and its name are in +the index. + +### 4.6 Parameters + +| Parameter | Value | +|---|---| +| Node keystore KDF | Argon2id **256 MB**, t=3, lanes=4 — recorded per envelope, so raising it does not orphan existing keystores | +| Hub password verifier | Argon2id **256 MB**, t=3, over the client-derived `auth_key` — `pw_version` migrates transparently on next login | +| Browser bundle key | Argon2id **128 MB**, t=3, p=1 | +| Browser `auth_key` | PBKDF2-SHA512, **600 000** iterations | +| Chunk cipher | AEAD, 1 MB chunks, per-chunk key by HKDF | +| Chat nonce | 96 random bits per message, never a counter | +| Invite / pair codes | 40 bits, Crockford base32, single use, stored as `sha256` | + +--- + +## 5. The node protocol (MNP) + +### 5.1 Transports + +| Listener | Role | Status | +|---|---|---| +| **WebRTC DataChannel** | primary, browser **and** desktop client | the path everything is tested on | +| **QUIC** | LAN, port-forwarded, hub-less `group://` | **off by default** (`[node] quic_enabled`) — nothing ships a QUIC client yet, and it does not implement chat (below) | +| ~~TCP + TLS~~ | — | does not exist (**C6**) | +| ~~HTTP file API~~ | — | does not exist (**C1**) | + +**QUIC carries no chat, deliberately.** It implements neither the per-device +sealing nor the device identification the chat rules require, and a message +reaching a group's archive without them would be a plaintext row in an encrypted +history — indistinguishable from one somebody actually wrote. The handler was +**removed rather than gated**: refusing there would mean maintaining a second, +weaker set of rules for a transport with no client, which is how a bypass branch +survives. An unimplemented message type is logged and dropped, like every other +one this transport does not have. + +> That is the general form of the parity rule. **A transport either implements a +> content rule fully or does not serve that content at all.** "Implements the +> authentication but not the authorization" is the shape of finding **C6** and of +> the third review's **M2**, twice. + +**ICE/STUN is the NAT traversal mechanism**, for native clients too — via +`aiortc` in Python. `punch_nat()` is a direct-connection helper, not a traversal +stack: one UDP probe to one address, no STUN client, no candidate gathering, no +dual-stack fallback, and it requires the client to already know its own external +address. It was validated on one ISP and one NAT type. ICE has been validated +across two ISPs, two browsers, IPv4 STUN and IPv6 direct, and 4G CGNAT +(§11.1) — **no TURN relay is needed**. + +**Several STUN servers, two providers deep**, because a single server is a silent +single point of failure that adds the full gathering timeout to every connection +when it is slow: + +``` +stun:stun.l.google.com:19302 +stun:stun1.l.google.com:19302 +stun:stun.cloudflare.com:3478 +``` + +Both sides carry the same defaults and gather independently; neither learns which +server the other used. **A STUN server learns the querier's public IP and NAT +mapping — that is its purpose.** No content, no credentials and no group metadata +passes through it; adding one is trusting its operator to learn your NAT topology, +nothing more. + +The node's list is editable three ways (Node page, `meshbay-node stun`, +`node.toml`) and hot-swapped on save. **The browser's list is hardcoded and not +configurable**: the hub relays SDP, not ICE policy, and no mechanism exists — or +is wanted — for a node to push ICE configuration to a browser. + +**ICE interface filtering** (`ice_interfaces`) is node-side only, and controls +which local addresses the node offers as candidates. Two modes: auto (excludes +virtual and VPN adapters by heuristic) and manual (an explicit whitelist). It +exists because VPN clients add virtual interfaces whose `.local` mDNS candidates a +remote node spends seconds failing to resolve. The browser's own gathering is +governed by the engine and is outside this application's control. + +### 5.2 The handshake + +One implementation, `meshbay_common/handshake.py`, called by both transports. Two +implementations of one security check is **C6** waiting to happen. + +``` +client → node handshake {token, group_id, nonce_c, v, v_min} +node authorize_token() JWT · scope · denylist · group_id · membership · hosting +node → client handshake_challenge {nonce_s, node_pk} + ── pre-proof window: bundle fetch, join ── +client → node handshake_response {proof} +node verify HMAC(GEK, client transcript) +node → client handshake_ack {proof, sig, node_pk, sealed configuration} +client verify HMAC(GEK, node transcript) + Ed25519(node_pk), + and that ack.node_pk is the key announced above +``` + +**The transcript is length-prefixed, domain-separated and role-bound:** + +``` +"meshbay:mnp:handshake:v1" ‖ len‖role ‖ len‖group_id ‖ len‖nonce_c ‖ len‖nonce_s ‖ len‖binding +``` + +Every field is length-prefixed so the concatenation is unambiguous, and the role +is bound in so a client proof can never be replayed as a node proof (**L4**). + +**Channel binding is mandatory and an absent one is refused** — never degraded to +nonce-only, which would silently drop MitM detection: + +| Transport | Anchor | +|---|---| +| WebRTC | both DTLS certificate fingerprints | +| QUIC | SHA-256 of the server certificate. On a **resumed** session, which carries no certificate, the anchor travels with the session ticket — sound, because the ticket derives from the handshake where the certificate was presented | + +**Authentication is mutual** (**C3**). The node proves possession of the group key +over a **client-chosen** nonce *and* signs the transcript with its long-term key. +The client verifies both, refuses a bare ack, and **TOFU-pins `pk_node`** per node, +refusing a changed key outright with a deliberate reset path in Settings for a +legitimate rotation. + +> `node_pk` is announced in the challenge because joining needs it before the ack: +> a first-time member signs a transcript naming this node and has no group key to +> complete a handshake with. It is unverified at that point and is never a +> substitute for the ack. + +**The pre-proof window is three messages, and that is a bound to defend.** Only +the two bundle fetches and `join_request` are served before the proof, because +each is something a caller needs *in order to* prove possession of the group key: +a returning browser has to recover its identity, and a new member has no key to +prove with. Nothing else qualifies. Device messages are **authenticated-only** — +a device request is countersigned later by a device already pinned, so requiring +the caller to finish its own handshake first costs nothing and keeps the surface +at three. The bundle fetches are additionally bounded (4 per connection) and +audited, because that window is the neighbourhood **C4** and **C5b** came from. + +**Pinning is defence in depth, not the primary control.** A substituted node +already fails the key proof. Pinning covers the case where an attacker *holds* the +group key — an ex-member, a leaked key — and swaps the node underneath, which the +proof alone cannot distinguish from the genuine node. + +**Authorization rules:** + +- `group_id` is **mandatory**. Omitting it once skipped the membership check and + fell back to the node's first group (**M1**). +- `scope == "user"` is enforced by default; node-scoped daemon tokens are refused + on the client path (**M9**, **NS7**). +- The denylist is consulted for user, `jti` **and** group. +- The node **refuses connections when it holds no group key** — there is no + `gek_required: false` bypass (**NS8**). + +**Refusals carry a code**, not only a sentence, because a client can act on a code. +`not_a_member` in particular is usually a token issued before the person was added +to the group — `groups` is baked in at sign-in and the hub pushes no updates — so +the client refreshes once and retries rather than telling someone who was invited a +minute ago that they are not a member. + +### 5.3 Correlation and liveness + +**Every reply carries the request id it answers** (`req_id`). Matching by arrival +order is a guess that fails silently and asymmetrically: the victim is never the +request that was answered wrongly, it is the unrelated one now waiting for a reply +already delivered elsewhere. The node stamps `req_id` on the reply from `_send`, +via a ContextVar so a handler's spawned work still answers under the right id, and +never on a broadcast, which answers nothing. + +`PING`/`PONG` is liveness on an **already-open** channel and never a discovery +mechanism — opening a connection costs a full ICE/DTLS handshake, measured at +0.6–7 s. + +### 5.4 Operator-signed operations + +Destructive and privileged operations require an **Ed25519 signature over a +structured transcript**, never a JWT. The hub controls JWT issuance and can +therefore never establish node-level authority. + +``` +"meshbay:admin:v1" ‖ len‖op ‖ len‖node_pk ‖ len‖group_id ‖ len‖subject ‖ len‖nonce ‖ len‖ts +``` + +TTL 120 s. **The client reconstructs the transcript from announced fields and +refuses to sign if the operation or subject is not what the user asked for** +(**H5**) — a challenge of opaque random bytes signed blind is an unbound signing +oracle. The transcript's subject names the *outcome*, not the operation: what the +operator is shown before signing has to be what happens. + +Verification is against `roster.operator_pks()`, rebuilt from node state, **never** +from anything in the response. + +| Operation | Authority | +|---|---| +| `file_delete` | the operator, or **any non-revoked device of the uploading account** | +| `dir_delete` | the operator alone, and only on an empty directory | +| `invite_create` | the operator (or a delegate, when delegation ships) | +| `gek_rotate` | operator-signed; the node generates the key itself | +| initial `gek-init` | **local admin API or CLI only** | +| root add/remove/update/eject/plug, `apps_enabled`, app directories, transfer limits | operator-signed | +| ~~`gek_bundle_store`~~ | **the message does not exist.** No member ever hands the node key material | + +`gek_bundle_store` was deleted rather than gated. The operator's X25519 public key +is announced in the handshake ack, so any member could wrap a key of their choosing +for it; a path that does not exist cannot be mis-authorised (**C5b**). + +**Authorisation is against the account, not the key.** With several devices per +person, verifying against the exact uploading key would refuse a person's desktop +the right to delete what their phone uploaded. `uploader_pk` stops being the +authorisation key and becomes the **audit record** of which device acted. This +remains **roster-rooted, not token-rooted**: a hub minting a token that claims to +be someone holds no key the node pinned for them, so the signature fails. + +**Ownership is provable, not merely recorded.** The uploader signs +`meshbay:upload:v1` over node, group, root, path, content hash, account and +timestamp, and the node stores it with the index entry. Ownership is then +verifiable by any member rather than asserted by the node, and the path where +overwriting a file made the attacker its recorded uploader (**C5a**) is closed a +second time. + +**One implementation, several front doors.** `meshbay_node/ops.py` holds every +operation. The loopback API, the CLI and the signed MNP handlers all call these +functions; they take the daemon state, raise `OpError`, and know nothing about +HTTP. Two implementations of one operation with two authorization checks is +**C1**/**C6** one size down. + +### 5.5 Transfer leases + +A download used to be invisible to the node: a client sent eight independent +chunk requests and reassembled the answers, and nothing said a transfer had +started or ended. There was nothing to count and therefore nothing to cap. + +**The unit is the lease** — the node's record that a peer is transferring +something, held for the length of the transfer and released by name +(`transfer_open` / `transfer_close` / `transfer_state`). Six properties are +decisions: + +- **`tr` is drawn by the client.** Re-opening after a reconnect with the same `tr` + is idempotent, so a reconnect cannot charge a member twice for one transfer. +- **A lease is scoped to the connection**, never to the account. It dies with the + session, which makes the primary reclaim deterministic. +- **A lease covers a job, not a file.** A directory zip is dozens of files and one + lease. +- **Nothing is persisted.** A restart drops every session anyway; a lease that + outlived the process would be a slot nothing can release. +- **Leases are counted, not bytes.** What a slot protects is concurrency — open + file handles, disk seeks, the channel buffer each transfer keeps full. +- **Per-member first, then node-wide.** A member at their own cap queues behind + their own transfers and never holds a node-wide slot a second member has none + of. Reversed, whoever arrives first takes everything. + +| Bound | Default | Why it exists | +|---|---|---| +| Node-wide concurrent transfers | 8 | operator's machine | +| Per account, per group | 2 | absent means this, not "unlimited" | +| Grant deadline | 30 s | a grant nobody takes up is a slot nobody can use | +| Idle timeout | 120 s | catches a peer that vanished without the connection noticing | +| Queued per member per kind | 32 | an unbounded queue is how a node runs out of memory politely | +| Missed grants before closing | 3 | without a bound the requeue is a permanent cycle | + +**Browsing a group is never subject to a transfer slot.** Not the poster grid, not +the album covers, not the video thumbnails, not the file list, not opening a photo +or a document to look at it. **A member must be able to browse a group that is at +capacity exactly as they browse an idle one.** That is a requirement, not a tuning +parameter, and it is met structurally rather than by choosing a lucky threshold. + +Navigation proper never touches this path at all: listings and metadata are their +own message types, sealed under the group key, with no relationship to a chunk +request. That half needs no rule — only a test that fails if someone later routes a +listing through the chunk path. + +The chunk path itself carries three genuinely different things, and they are +distinguished **structurally, by what the id resolves to and by which function +asked**: + +| What | Resolves to | Rule | +|---|---|---| +| Thumbnails, posters, cover art, cached audio conversions | a **media-cache id**, not an index entry | **Never leased, never counted, never queued.** One chunk each, out of a bounded cache the node built itself | +| Looking at one file — a photo full size, a document, an image | a real index entry, fetched whole | **Not leased**, subject to the bound below | +| Downloading and uploading | a real index entry | **Leased** | + +The rule stated as a sentence someone can check by reading: + +> **A transfer is something the transfers panel shows. If it does not appear +> there, it does not take a slot.** The two sets are the same three call sites, +> which is what makes this verifiable rather than a judgement call at each new one. + +**Why the exemption is expressed in concurrency and not in size or bytes**, which +is the durable part of this decision: + +- **A size threshold does not separate the two.** A raw photo out of a camera is + 60–80 MB and is *browsing*; a 40 MB archive is a *download*. Any threshold + letting the photo through lets the archive through too. +- **A byte-rate budget does not either.** It would have to be large enough for that + same photo, at which point it is large enough to be a download channel. +- Concurrency is the thing being rationed, so concurrency is what the exemption is + expressed in. + +The bound is therefore a ceiling on **how many distinct files one session may read +leaselessly at once** (12, with a 60-second idle expiry). Per session and not per +member, because this is a ceiling on what one connection can do while claiming to +be browsing, not a resource pool — a member with three tabs open is browsing in +three tabs. Two rules keep it from becoming a bug: **a file already being read is +always admitted**, whatever the count, because refusing a chunk halfway through a +photo is worse than never having admitted it; and **entries expire on idle**, +because a viewer closed mid-file stops asking and says nothing, and dead entries +would eventually refuse every later preview. + +**A preview never shows "waiting", because a preview never queues.** If the bound +is somehow reached, the request is refused with a stated reason and the person tries +again — it does not silently become a queued transfer in a panel they were not +looking at. + +**What the residual is, stated plainly.** A client that lies — labelling a bulk +download as a view — gets that bound's worth of files at a time instead of its +member cap. It is bounded, it is audited, and it is the same class of statement as +the cap itself: **this is a fairness control among cooperating clients**, in the +company of the stream cap. It is not a defence against a member determined to +saturate the operator's disk, and must never be described as one — that member is a +member, and the answer to them is `member revoke`. + +**The numbers are visible or the queue is unprovable.** `transfer_state` carries +`used`, `cap` and `ahead` so a client can say "waiting — 2 of your 2 slots are +busy" rather than showing a bare spinner; the same counters reach the loopback API +for the CLI and the Node page, and a periodic debug line. When someone reports a +transfer stuck at "waiting", that line is the only thing that will say whether the +node ever had them in a queue at all. + +The lease module is free of asyncio and of the transport: it decides, and the +caller does the I/O. A queue that reveals itself only through a DataChannel is a +queue nobody can prove things about. + +### 5.6 Versioning and flag days + +MNP and MHP version independently of the package version. Every wire message +carries `v`; both peers declare `v` and `v_min` on the handshake and refuse each +other with a code (`version_too_old` / `version_too_new` / `version_unreadable`). + +**A mismatch is a refusal, not a field that turns up missing.** A stated refusal is +a bug report; a feature that quietly does not work is a support case. + +Additive changes are MINOR and cost nothing. A change to what a peer must be able +to *do* is MAJOR even when the messages are additive — a peer that cannot ask for +a transfer lease is either refused, or not refused and transferring outside every +cap the operator set. + +**There is no compatibility switch, by policy.** An opt-in flag leaves the old +branch reachable on every node, which is **C6**'s lesson one feature later. Where +a break is required, `MNP_MIN_SUPPORTED` moves with `MNP_VERSION` and the +deployment is coordinated: the hub serves the SPA, so a browser picks up the new +client on reload; the desktop client ships its own UI, which is why +`GET /v1/hub/version` carries `client.minimum` and the client checks it **before** +connecting and says "this version can no longer connect" rather than showing a +handshake refusal nobody can act on. + +**The floor being the current version is what keeps capability branches out of the +client.** `MNP_MIN_SUPPORTED` equals `MNP_VERSION`, so `check_version` refuses +every older peer at the handshake — which means **every capability is true of every +peer the client can reach**, and there is nothing to test for. An upload is sealed +or it is not sent; a transfer has a real lease or it does not run; there is one +app-directories op and no wrappers behind it. The client records the version its +peer declared, for diagnostics, and **branches on none of it**. + +> A capability flag on a peer whose floor already guarantees the capability is a +> branch that can only ever take one path — until somebody lowers the floor, at +> which point it silently takes the other. **A field kept "just in case" is how +> the branches come back.** + +Where a break leaves data behind, a migration runs with the node stopped, backs +the database up first and is idempotent. But **a migration that has to be +remembered is a migration that does not happen**, so anything that *can* be a +read-time fallback is one instead: the roster reads an older settings key when the +new one is unset and leaves it behind on the first write, and `node.toml` keeps +being read in its older spelling. Only a transformation nothing can infer — two +settings that disagree, where only the operator knows which they meant — is +allowed to need a step somebody has to run. + +--- + +## 6. The node + +### 6.1 Node authority + +The node is the content authority. Its authority comes from **its own roster**, +established locally by pairing, and from nowhere else — never from the hub, never +from a config file, never auto-pinned from the keystore (**NS4**). A configuration +that still names a legacy admin key is warned about at startup and never obeyed. + +The division of trust: **the hub certifies identity; the node authorises content +operations.** Hub membership lets someone *reach* a node; the node's roster decides +whether it wraps anything for them. + +### 6.2 Roots + +A group's content is **a set of named roots**, each mapping to a local directory, +forming one union virtual root: + +``` +/ (group virtual root) +├── Films/ → D:\Media\Films +├── Music/ → E:\Audio (external drive, removable) +└── Documents/ → C:\Users\me\Share +``` + +**The name is the chosen directory's basename, derived once at add time and +stored.** Never recomputed from the path: renaming a folder on disk would +otherwise silently re-identify a whole library and break every stored reference to +it. Four rules make basename naming safe: + +- **A duplicate basename is refused, case-insensitively.** Collisions are common in + practice (`D:\Films` and `E:\Films`). Refusing is correct; an explicit alias is + the escape hatch (open item O11). +- **No root may contain another**, compared case-insensitively and after + canonicalisation. Two nested roots would index the same bytes twice under two + identities. +- **The basename becomes a path segment every member sees**, so it must itself pass + the portability rules (§10) — a root Windows cannot write to is a root nobody on + Windows can download from. +- **Every index path carries a root segment**, uniformly, including in a + single-root deployment. One code path, not two. + +Five consequences, none optional: + +1. **The root name is part of a file's identity**, so renaming a root rewrites + every path under it. Renaming is explicit and warned, never a cosmetic setting. +2. **Availability is per root.** With one directory an unplugged disk was a hazard; + with named roots it is a supported state — one root freezes, the others carry on. +3. **Free space, quotas and capacity are per root** — different volumes. Anything + the UI says about space names which root it means. +4. **Path resolution is per root**, in one place: a request names `<root>/<path>`, + `RootSet.resolve()` resolves it against that root's canonical path and refuses + `..`, absolute segments, symlinks and anything escaping the root. +5. **`kind` is a view hint** (`generic`/`video`/`audio`/`photo`), nothing more. + +**Each root is read-only or read-write.** + +- `writable = false` (the default) means read-only **for everyone, including the + operator**. A published library that quietly accepts writes from whoever holds + admin authority is not one, so refusing the operator is the point rather than + the defect. +- `writable = true` means any group member may upload there. + +Several roots may be writable and none need be — a fully read-only group is valid. +The operator toggles this with a signed op. + +> **There is one answer to "may this member write", and it is the root.** A single +> flag over the group cannot express "this library is published read-only and that +> folder is a drop box", which is the ordinary arrangement — so the group-wide +> switch that used to exist is gone entirely: the message, the signed operation, +> the field on the handshake ack, and the `upload` alias each root used to carry +> beside `writable`. **Two sources for one question is one too many**: whichever +> the code consulted first decided it, and a client falling back to the group flag +> whenever a root omitted `writable` is exactly that bug with a compatibility +> justification. Two names for one boolean is the same fault one size down. + +> **A control that writes must name where.** With two writable roots the node +> cannot choose without guessing, and a guess sends a member's file to a disk the +> operator did not intend. The client names a **root**, never a path; everything +> below the root is decided by the node. + +**A root that goes away must freeze, not empty.** The indexer runs a watchdog +observer and rebuilds on change; unmounting a volume either emits deletions for +the whole tree or presents an empty directory to the next scan. Both propagate as +though the owner erased their library. So a root has two independent runtime +states: + +- **`ejected`** — operator-controlled, persisted in `roster.db`. +- **`available`** — computed as `not ejected and is_live()`. This is what clients + and the indexer see. + +The distinction matters: between clicking eject and physically unplugging, +`is_live()` is still true, and without `ejected` the availability sweep would +immediately flip the root back. + +**Eject** stops that root's observer, marks it unavailable, freezes its entries and +propagates the availability change to connected peers — the operator can then +safely unplug. **Plug** checks the path is accessible first, then rescans: the +plan called for a reconciliation, but a device people carry around can come back +arbitrarily different, and the hash cache means unchanged files are not re-read. + +**`ejected` is persisted and restored at startup**, because a restart is exactly +what an operator does after noticing a drive fell off, and an in-memory flag would +let the following scan read the empty mount point as an erased library. It lives in +`roster.db` and not in `node.toml`: it is runtime state, and an operator's +hand-written config must not be rewritten because a USB drive was unplugged. + +**Auto-eject is the safety net.** If a `removable` root's path disappears, the +availability sweep sets `ejected` as though the operator had clicked it, and +reports it so the daemon persists it. Nothing is deleted: index entries, cached +metadata, thumbnails, chat history referencing those files and app directory +configurations all survive, the last flagged as temporarily invalid rather than +wrong. + +### 6.3 Indexing + +The index is **content-addressed**: `GroupIndex` is keyed by blake3, so the same +bytes at two paths inside one group are **one** entry. This is why a scan can +report ten files and index nine, and it decides how reconciliation must work: + +> **Anything comparing disk against index must compare ids, not paths.** Comparing +> paths makes the second path of a duplicated file look like a missed event, every +> sweep, forever — rewriting the entry, bumping the version and pushing an index +> update to every connected peer. + +**Hashing is partial above 40 MB.** A hash exists for content identity, and a 4 GB +file does not need 4 GB of I/O to be identified with overwhelming probability: + +| Size | Method | `hash_version` | +|---|---|---| +| ≤ 40 MB | full read | `1` | +| > 40 MB | blake3 over the first 20 MB ‖ last 20 MB ‖ 5 MB at the midpoint | `2` | + +Head and tail catch container headers, trailers and files that differ only at one +end; the mid-sample catches files sharing a header and trailer. Below the +threshold a partial read would sample the whole file anyway, so the full path is +simpler and produces the same value — which is what keeps small files +cross-comparable between nodes of different versions. A large file indexed by a +node of each version produces different ids and does not merge in cross-group +search; that resolves itself when both upgrade, and is the accepted cost of not +reading 4 TB to build a library. + +`hash_version` is an additive index field with a default, so an entry written +before it deserialises correctly and needs no protocol bump. The hash cache +carries the column and auto-migrates on open; large files are re-hashed lazily on +the first scan after an upgrade. + +**Periodic reconciliation is mandatory on every platform**, not a backstop: +`ReadDirectoryChangesW` drops events under load on Windows, and inotify is +unreliable on a FUSE-mounted volume on Linux (§10). + +**Progress accounting covers the real-time path too.** A file's size joins the +total the moment its debounce timer is first scheduled — not on every re-trigger, +or a cancelled-and-rescheduled timer double-counts — and joins the scanned total +when its hash finishes. The scanning flag clears only when no timer is pending +*and* no hash is running, because the in-flight hash of a large file is the entire +reason to show progress. + +`index_progress` is deliberately **not** sealed: counters only, every 2 s. + +### 6.4 Uploads + +Five protections, and they are the substance: + +- a **filename allowlist**; +- **no overwrite** — a colliding name gets a free one. The check is `Path.exists()`, + and `stat()` is itself case-insensitive on NTFS and exFAT, so this already holds + there; +- **strict chunk ordering**; +- a **size cap** — 4 GB per file. There is deliberately no aggregate quota yet, and + that gap is named in §15.3 rather than left to be discovered: a member can still + fill the operator's disk one capped file at a time; +- the target root must be **writable and available**, enforced by the node. + +**There is no quarantine subdirectory.** A folder appearing beside the operator's +library because somebody sent a file is the node deciding how their disk is +arranged. What made a quarantine worth having was never the subdirectory — it is +the four rules above, and they are unchanged. The client now names the destination +folder, which is safe for exactly one reason: it is resolved through +`RootSet.resolve()` (§6.2). **A member answers "which of this group's folders", +never "which path on the operator's disk".** + +If the named root is unavailable the upload fails **with a stated reason** and +never falls back to another; if the group has no writable root, uploads are refused +rather than guessed. + +**The node enforces it; the interface merely stops offering it.** Each root's +`writable` flag rides the handshake ack and the index payload, so a client knows +whether to draw the Upload button and the chat paperclip, and changes are broadcast +to everyone connected. None of that is the control: a member on an old tab, or one +speaking MNP directly, is refused by the node. + +In a group with **no** writable root the interface says so plainly rather than +picking one — a fallback that chooses whatever comes first only moves the failure +to send time, where the person has already chosen the file. + +**Archives are not the node's business.** A member downloading a folder as a zip +fetches the same encrypted chunks as any other download and assembles the archive +in the browser. The node serves no bundles, holds no temporary files, and cannot be +asked to compress anything — one fewer place where a request turns into work on +someone else's disk. + +Directory creation is not privileged. Directory **removal** is, and is refused +unless the directory is empty. The emptiness rule is the safety property: whatever +the caller intended and whatever the client sent, the operation cannot destroy +content. It is checked twice — before the challenge is issued and again after the +signature returns — because a file can land during the round trip to the +operator's browser. + +### 6.5 Derived data and enrichment + +The governing rule: + +> **Enrichment happens on the client, from data it already has. What the client +> cannot compute, the node produces — and where the node produces it for +> everybody, it caches it in its own `data_dir`, never in a shared root.** + +The client half is unchanged and is why a chat image thumbnail costs the node +nothing: the browser already decrypted the image and scales it itself. + +The node half was decided against the alternative and the reasoning is worth +keeping, because it looks like a violation of "no derived state" and is not: + +1. **A third-party API quota is per credential, not per device.** A token shipped + inside every install and called from every client scales with the number of + *devices* in existence. A node making the calls on behalf of its own members + makes the number of *nodes* the denominator, and one lookup per unique title + serves every member indefinitely. +2. **Thin clients benefit from a node that does more.** A phone should receive a + small ready-made image and a JSON blob, not decode video or hold its own + multi-gigabyte cache. +3. **Keeping the credential server-side is a strict improvement** over shipping it + to every renderer, and it preserves the desktop client's tested invariant: it + issues no request outside `/v1/` and the signaling socket. + +**The cache lives in the node's own `data_dir`** — beside `chat.db`, `audit.db`, +`bundles.db` — and never inside a shared root. That is not a compromise, it is +strictly better, for reasons independent of sovereignty: a shared root is +routinely a read-only backup mount or a share the node cannot write to; a +dot-prefixed folder is not hidden on Windows and would appear as an ordinary +folder full of previews of a private group's content to anyone who plugs the drive +into another machine; and it would need filtering out of every listing path +consistently, forever. + +**Delivery reuses the chunk path.** A thumbnail is addressed by its own blake3 +exactly as a file is by its id, so the chunk handler resolves a requested id +against either a real file or the media cache. Same transport, same group-derived +encryption, same backpressure — no parallel mechanism, and no new authorization +surface. + +**Cache lifecycle is tied to the index.** A file's thumbnail and its third-party +match are pruned by the same event that removes its index entry. Third-party +metadata is cached per external id with its own refresh window, since several files +of one show share one fetch. + +**A visible node-wide toggle turns third-party calls off entirely**, independently +of any application being enabled, for an operator who wants zero third-party +network traffic. This is genuinely new node behaviour — egress to a third party, +and a disk-resident cache with a real deletion obligation — and it is stated rather +than left implicit. + +**Anything that shells out to a media tool obeys three rules**, all of which this +codebase has paid for: + +- **its own small bounded pool with a short timeout, never the streaming pool** — + a grid of fifty videos would otherwise exhaust every streaming slot on the node, + since a stream slot is held for the length of a film; +- **drain the pipes, then wait with a timeout, and release the slot regardless** — + a process that outruns a paced reader cannot finish closing while its stdout is + full, SIGKILL or not; +- **a background task must be held**, or the loop may collect it mid-flight and the + slot is lost for good. + +**Link previews** are a further instance. A URL pasted in chat is unfurled **by the +node**: the browser cannot (a strict `img-src`/`connect-src`, and CORS), and a +direct fetch would leak every reader's IP to the linked host on each render. The +card text lives in a bounded in-memory TTL cache; the image rides the same +blake3-keyed store as any other thumbnail. **The new surface is SSRF**, because the +URL is a member's choice and it triggers an outbound request from the operator's +machine: http(s) only, no credentials, a port allowlist, every resolved address +must be globally routable, redirects followed by hand so each hop is re-checked, +the connect address re-checked against the checked one, a response-size guard, and +a per-member rate limit. The operator can switch previews off per group. + +**Every new outbound or cross-trust surface needs a bound and a named adversary in +the same commit.** That is the standing rule this section exists to enforce. + +### 6.6 Chat storage + +One SQLite database per group at `data_dir/<group_id>/chat.db`. Rows carry the +sealed ciphertext, the epoch, the sending device, the nonce and the signature, with +a unique `(device, nonce)` refusing replays. + +Paging is **backwards** — `get_recent` / `get_before` / `has_before` — because a +chat opens at the newest page. A forwards pager is not what a chat opens with. + +`meshbay-node chat prune <days>` deletes **messages only, never an epoch key**. An +epoch with no messages is harmless; an epoch key deleted while messages still need +it is an unreadable archive. + +### 6.7 Operator surface + +Two personas need different tools, and the headless one is the normal deployment: + +| Operator | Reaches the node via | +|---|---| +| Desktop | the desktop client's Node page | +| **Headless / SSH** | the CLI | + +**Every operation is reachable over SSH with no browser on the host.** `status` +deliberately reads the keystore and config directly so it works while the daemon is +stopped — the state an operator is most often in, since the daemon will not stay up +before its key is linked or a group exists. + +``` +meshbay-node status +meshbay-node group add <name> --dir <path> [--no-writable] +meshbay-node root list|add|remove|set|eject|plug +meshbay-node gek init|rotate +meshbay-node operator pair +meshbay-node member list|invite|revoke|unpin +meshbay-node member device list|revoke +meshbay-node denylist show|clear +meshbay-node file list|rm +meshbay-node chat status|prune|encrypt-history +meshbay-node stun list|add|remove|reset +meshbay-node reload +``` + +`member revoke`/`unpin` resolve a username against the roster and **refuse an +unknown one** rather than acting on nobody — a typo must not look like success. +Revocation tells the operator what it does *not* do: the ex-member stops receiving +the key on their next connection but still holds the current one, so the message +ends with the command that rotates it. + +**The node's local control API is JSON only, on loopback, behind a per-run session +token** (`X-MeshBay-Token`, printed at startup, file mode 0600). "Localhost only" +is not authentication: any local process can reach it, as can a page in the +operator's browser via DNS rebinding — and this API re-initialises group keys, +issues invitations and reads the audit log. There is no server-rendered dashboard; +the desktop client's Node page and the CLI are the two consumers, and each +operation endpoint is one `_op(...)` line onto `ops.py` (§5.4). + +**The accepted cost, recorded as a choice:** on a headless server the only admin +path is the CLI. The CLI covers every operation, so this is acceptable — but it is +a real capability reduction, not an oversight. + +> **MNP is the path that must exist; loopback is the fallback.** The operator of a +> node is not necessarily sitting at it. Any operator-facing control needs its MNP +> route first, or it renders for nobody on the web. + +### 6.8 Node settings + +Five `[node] `settings affect what the node does rather than how it starts, and +their value is invisible until something goes wrong — so they are surfaced on the +Node page: + +| Setting | Default | What it controls | +|---|---|---| +| `invite_ttl_hours` | 168 | how long a member invitation stays valid | +| `pair_ttl_hours` | 24 | how long an operator pairing code stays valid | +| `device_request_ttl_minutes` | 60 | how long a device request waits for approval. Comfort, not security: the code is bound to the keys by its hash | +| `max_concurrent_streams` | 8 | simultaneous video streams. One process per viewer, ~50 MB each; a slot is held for the length of a film, so this counts viewers | +| `transcode_incompatible_video` | true | whether browser-incompatible video is transcoded during streaming. Unlike remuxing this costs real CPU per viewer | + +> **Turning transcoding off does not mean the same thing for every source.** A +> codec with an MSE codec string falls back to a copy and the viewer's own decoder +> decides; a codec with none has nothing to fall back to and the stream is refused, +> naming this setting. + +**Settings are persisted in both `roster.db` and `node.toml`**: the database for +immediate effect with no restart, the file so the value survives a wipe or a fresh +install. On startup the file is read and a database override wins. The TOML write +is a targeted line replacement, never a round-trip through a writer — that file is +hand-written, full of comments recording decisions, and a setting changed from a +panel must not rewrite the operator's file. + +Transfer limits (§5.5) follow the same pattern, with per-group per-member caps as +an operator-signed op. + +--- + +## 7. The hub + +### 7.1 Role — chosen, not minimal + +Hub minimisation was considered and **deferred, and may be dropped** (decision D4). +The hub keeps serving the web UI and remains in the trusted path by choice. That is +a legitimate product call; what follows from it is carried deliberately rather than +by accident (§2.3). + +**Stores:** accounts (username, encrypted email, status, role), the group registry +and membership, IP logs (one year, legal retention), node registrations, refresh +tokens, notifications, the moderation blocklist, instance policy, and per-account +device keys for hub login. + +**Does not store:** file content, file names, private-group indexes, message +content, private keys, group keys, keypair bundles, user identity keys, node IPs +beyond ephemeral signaling. + +**Knows, unavoidably:** who is a member of what, when nodes connect, and when a +chat message was posted in which group and by which account id. The last is a +stable identifier the hub needs in order to skip the author when creating +notifications; it carries no content, and it is a known metadata leak rather than a +solved problem. + +**Decides nothing about keys.** Hub membership lets someone reach a node; the +node's roster decides whether it wraps anything for them. + +`user_devices` **is not the key directory that was H3**: nothing reads it but the +hub, nothing wraps a group key for it, and it is a different key from the per-node +identities. What it does cost is metadata — the hub knows how many devices an +account has and when each last signed in. + +### 7.2 Node registration and signaling + +Registration on the node socket requires a **node-scoped token**, verifies the node +record against the token subject, and **derives group claims from the database**: a +node may narrow the set to what it hosts but cannot widen it, and cannot displace a +live registration (**C2**). + +The node authenticates to the hub with an Ed25519 signature over a domain-separated +timestamped message — **no password and no auth key on a node** — and receives a +`scope: "node"` token that is refused for group management. The operator manages +groups from a client (**NS7**). + +Signaling is rate-limited, SDP-size bounded, capped per user, and **the caller must +share an active group with the target node**. Otherwise any authenticated user +could make a third party's machine allocate peer connections on demand (**H6**). +The address in a NAT-punch request must match the caller's source address. + +`X-Forwarded-For` is honoured **only from a trusted proxy, rightmost hop** (**M7**), +through one helper so the behaviour is defined in one place — including for the +rate limiter, whose keying otherwise collapses to a single global bucket behind a +loopback proxy (third review L10). + +### 7.3 Groups + +A group's **identity is its UUID**, everywhere: the route, the node's configuration, +membership. A group **name is unique per owner account**, case-insensitively and +trimmed, enforced by a functional unique index; two different owners may each have +a `photos`. Names are displayed as `name@owner`, which is a label plus a create-time +check and **not an addressing scheme**. The handle is hub-local: the same +`name@owner` on two federated hubs are different groups, and a federated row shows +its source hub rather than an account. + +`visibility` and `join_policy` are the two independent axes described in §3.5. +`join_policy` is read from the node's own configuration, never from the hub. + +**Public group creation is quota'd** — ten live public groups per owner account, +staff exempt. Public groups are the ones that cost other people something: they +appear in the directory and are brokered to strangers. The check is at creation +only, which is correct because the update endpoint refuses to change visibility. + +### 7.4 Instance policy + +`hub_settings` is a key/value table an admin edits at runtime. It is **instance +policy, not group content**: it says how this hub behaves and holds nothing about +any group's files, index, membership or keys, so §1.3 is untouched. + +The first entry is `allow_public_groups`. Switched off, server-side and read live on +every path the hub mediates: creating a public group is refused (staff included — +the way back is to re-enable, not to slip past), the public directory returns +nothing local **and** federated, open-joining is refused, a non-member is handed no +node to connect to, the "this node hosts an open group, admit anyone" signaling +fallback is dropped, and the federation export advertises nothing. + +**It is a directory-and-brokering control, not a remote kill.** Existing members +keep their membership and their access. A node whose operator set `join_policy = +"open"` still pins and serves whoever reaches it directly over MNP; what the switch +removes is the hub-provided ways to find and reach such a node. + +### 7.5 Moderation + +Two verbs on a group, and they are distinct things: + +| | `suspend` | `revoke` | +|---|---|---| +| Hub | `status = "suspended"` | `status = "revoked"` | +| Node | nothing | signed revocation broadcast → denylist + live sessions dropped, **persisted across a restart** | +| Reversible from the panel | yes | no | + +The client shows the real state, not a blanket one. **Revocation is honoured by +nodes** and the denylist survives a restart (**H4**); signaling refuses a group that +is not active. + +**Moderator is not administrator.** The user-patch handler is split by field: a +moderator may act on the fields moderation needs and may not write `role`. + +**Content reporting requires authentication, distinct reporters and a rate limit, +and is refused when public groups are off.** An unauthenticated endpoint that +blocklists a content hash after two reports is a network-wide censorship and DoS +primitive for anyone who learns a public file's id. + +The exact-hash CSAM check is **structural, not yet functional** — production +databases are perceptual — and is stated as such so it is not relied on +operationally. + +### 7.6 Federation (MHP) + +Peer hubs exchange directory rows and revocations. The trust rules: + +- a pushed row's **source is bound to the signer**, not taken from the payload; +- the **token audience is checked**; +- a push is **capped**, and replays are rejected; +- **revocation acts on the peer's own directory entries** — it does not reach + nodes, and nothing local hosts a federated group. + +### 7.7 Account lifecycle + +A user can delete their own account from Settings behind a **passphrase re-entry** +— a live token may be a borrowed laptop, and the bar for something with blast +radius is proof of the passphrase. An admin can delete one too. + +The row is **tombstoned rather than dropped**: username released, email and password +hash cleared, node linking key dropped, memberships, notifications and refresh +tokens removed, active tokens refused at once by a status check rather than left to +expire. + +Two things survive on purpose: + +- **The IP log**, for its legal retention period, and it stays attributable — + detaching it would keep the data and lose the only thing it is for. The name is + copied onto those rows as the account goes, since the join that used to supply it + would answer with the tombstone. +- **Everything on a node.** Files, the pinned identity and the keypair bundle live + on machines the hub does not command — the same sovereignty that makes admission + work. **Deleting a hub account is not an erasure request to the operators who + host you**; the operator surface is where that happens, and the docs must say so. + +Deletion is **refused while the account still owns groups**, rather than cascading +into other people's data. + +Registration is gated by a CAPTCHA whenever one is configured — **unconditionally**, +not only when some other field is absent, or the real client's ordinary request +skips it. The desktop client renders the widget too. + +--- + +## 8. Clients + +### 8.1 Two clients, deliberately + +| | Hub-served web SPA | Desktop client | +|---|---|---| +| Distribution | served by the hub | installed, signed release | +| Code integrity | **T3 accepted** — the hub can inject | detectable *if* reproducible builds ship | +| Key storage | IndexedDB / sessionStorage, plus a bundle on each node | OS-protected local storage; keys never bundled | +| Transport | WebRTC | WebRTC **+ QUIC** (sidecar, for hub-less `group://`) | +| Downloads | to disk where the engine allows it | native, streamed, unlimited | +| Positioning | **convenience tier** — zero install | recommended for sensitive use | + +**The SPA is not deprecated and stays.** It is the zero-install path, and the +objective is explicit: **a native client must not prevent web use.** It must be +labelled honestly — the application page states that the hub serves this code, and +the docs never claim end-to-end *integrity* for that path. + +**Several browsers, one identity per node.** A browser keeps nothing durable the +user controls, so the identity it creates for a node is left with that node, +encrypted under the passphrase. Any other browser recovers it there with the +passphrase alone: same identity, same pin, no second code. Joining a *different* +node creates a different key and needs that operator's code — the first contact it +has always needed. This is what makes the product behave the way people expect, and +it is also **C4**, with a blast radius of one node. + +### 8.2 The desktop client + +**Electron**, with an optional Python sidecar for hub-less `group://` over QUIC. + +The shell choice follows from what the interface actually depends on: not "the web" +in general but engine-class APIs — `RTCPeerConnection`, WebCrypto X25519/Ed25519, +MSE, service workers, File System Access. Keeping the engine keeps the transport, +crypto, key-derivation, download and player modules **as the client**. They are not +browser workarounds to be deleted once native; they are the implementation. + +**The non-negotiable: UI assets ship inside the package and load from disk.** A +shell pointing at the hub's application URL is a browser with a different icon and +fixes nothing. + +What running it establishes, and what each fact costs: + +- **A CSP in a meta tag silently drops `frame-ancestors`.** It is sent as a header + by the protocol handler. +- **Service workers do not work on a custom scheme.** The application has none and + uses the native save dialog; the worker stays for the browser. +- **A secure context is what makes crypto exist at all** — without it the whole + subtle-crypto surface is undefined, AEAD included. +- **The renderer cannot call the hub.** Its custom-scheme origin is refused by CORS, + and the hub deliberately has no CORS middleware — its API is reachable from no web + origin. Every hub call leaves from the main process, which refuses any origin that + is not the signed-in hub. **A script served by the hub is refused by the policy**, + which is T3's mitigation demonstrated rather than asserted. +- **The device's hub key lives in the main process, never in the renderer.** + Generated, stored and used there; the interface asks for a signature and is never + handed a key. Same rule as the save dialog, and for the same reason: the renderer + parses decrypted content from nodes, which is attacker-controlled input. +- **OS-backed secret storage is real on a desktop and honest without one.** With a + keyring it is keyring-backed; headless, the same code reports unavailable and + **refuses to store rather than downgrading silently**. +- **Installation places files, never secrets.** No key generation in a package's + post-install step or an installer custom action — a golden image would give every + machine the same key. + +Two guards apply to anything the hub can display **inside** the application, which +is a phishing surface: plain text or a very restricted markup subset, never raw +HTML; and a visually distinct region labelled as a message from the hub operator, +never a modal that can imitate application UI. + +The download page is a **security page**: it publishes the release key fingerprint, +and a hostile hub serves that page too. The fingerprint must also be published +somewhere the hub does not control, or the relocation of trust is circular. + +### 8.3 One UI source + +`packages/meshbay-hub/src/meshbay_hub/static/` **is** the interface, for the web and +the application alike. The client's build copies it and CI fails if the copy drifts. +**Never edit the copy by hand.** + +**Shipping the UI in a package creates version skew for the first time.** Today the +SPA and the hub deploy together, so a response shape and its caller change in one +commit. Once the UI is installed rather than served, the hub API is a compatibility +surface — which is why `GET /v1/hub/version` carries a minimum client version, and +why the client checks it before connecting (§5.6). Cheap now, awkward later. + +**A second copy of the hub address is what breaks the application, not the +protocol.** A module that decides where the hub is — "empty string, same origin" — +is true of a page the hub served and false of one loaded from a package, where a +relative API call hits the application's own protocol handler and sign-up and +sign-in fail. There is one seam, `platform.hubBase()`, and a test refuses any file +that decides where the hub is or fetches the API relative to the page origin. + +### 8.4 URL space + +| Space | Served by | Seen by the desktop client | +|---|---|---| +| landing, about, downloads, news | the static site overlay | ❌ never | +| the application | the hub (SPA) | ❌ never | +| the versioned API and the signaling socket | the hub | ✅ only this | + +**The desktop client issues no request outside the API and the signaling socket.** +That is a testable invariant, and it is what any feature adding third-party egress +must preserve — which is one of the reasons enrichment is node-side (§6.5). + +### 8.5 Downloads and streaming + +**Downloads go to disk, never through RAM, on every platform.** There are three +mechanisms — a granted directory handle, a service worker streaming a response, and +a blob as the floor — and which exist depends on the engine. + +> **A fallback chain reaches its floor silently.** Where the first two do not +> exist, every download went through memory and the only visible symptom was a save +> dialog at the end instead of the start. When a chain degrades, check what the +> floor costs on **every** platform that will reach it. + +Specific rules the download path is built on: + +- **A service worker being active is not the page being controlled.** An + uncontrolled page's requests never reach the fetch handler, so the stream is + handed over and never asked for. Require the controller, and have the worker + confirm it served the request. +- **An idle service worker is killed, and a streaming response is not "something to + do".** The page pings the worker while it writes and the worker answers, because + receiving a message is the event that resets the timer. +- **Every await on a download path is bounded and says which chunk it gave up on.** + An unbounded await is a freeze nobody can report. +- **A filename in a content-disposition header needs both forms.** The RFC 5987 + encoding uses `'` as a delimiter and the standard escaper does not escape it, so a + plain ASCII fallback rides alongside — the next surprise loses accents rather than + the whole name. +- **Three headers decide whether a page may frame itself and they must agree** — + the frame source policy, the frame-ancestors policy and the legacy frame option. + Same-origin framing is what a streamed download needs; refusing every foreign + origin is unaffected. + +**Pause and resume, and where resumability actually lives.** A pause button that +quietly restarts a download from zero is worse than no pause button, so the +interface offers only what the target can do. + +> **Resumability is a property of the *target*, not of the platform.** The same +> engine yields a resumable target from a granted folder and an unresumable one +> from a service worker, on the same page, for two files in the same batch. So +> **each target declares it itself** and the record travels with the transfer; +> the widget renders from that record. This is the fallback-chain rule (above) +> applied one level up: whatever tier a transfer ended up on, it records which, +> and nothing infers it from the platform. + +Two design decisions rather than implementation details: + +- **A paused transfer holds nothing.** Resuming rejoins the queue at the tail and + the interface says so. Anything else lets one member close the node by pausing. +- **A resumed file is truncated down to the last whole chunk, never appended to.** + Writes are sequential whole chunks, so a size that is not a chunk multiple means + an interrupted write — and **a silently corrupted download is worse than a failed + one.** The resume record also stores the file's content id, so a resume whose file + is no longer in the index fails with "this file has changed on the node", which is + the truth. + +**Uploads resume through the seal, not around it.** The node already holds the +partial state; the question "how much do you have" is asked as an ordinary sealed +upload message with no bytes and a probe index, and answered inside the sealed ack. +Asking on the lease message instead would have put the operator's filenames on an +unsealed message — precisely what sealing the write path bought. The partial state +is keyed by member, directory and filename in the **group** context rather than the +session, so a reconnect finds it, and an orphaned partial with no live lease is +reaped on a timer and at startup. The four upload protections (§6.4) are untouched +by any of this. + +**Video streaming** is fragmented-MP4 remux (or transcode where the codec has no +MSE string) fed to a source buffer, with the node holding one slot per viewer. + +- **Flow control is a window, not a debt.** Read-ahead is bounded by *time past the + playhead*, with a small window of segments in flight topped up as they land, + driven by a clock and by playback and **never by arriving data**. Granting a + credit per append pulls at network speed, fills the buffer ceiling, and then goes + silent for the length of the accumulated balance. +- **Credit follows the buffer, decided in one place.** The append path grants + nothing, because the buffer's update event fires for evictions too — crediting + from it pays the node for the player's own housekeeping. +- **Flow-control accounting comes before every early return.** A segment that + arrived is no longer in flight, whatever is then done with it. +- **A new stream starts from a known state**, reset at the *start* of the stream and + never in the teardown of the one before, which is skippable. +- **A viewer holding credit deliberately must say so**, or the node's stall timeout + ends a film that is merely paused. +- **Seeking restarts the source with an index seek before the input**, clamped away + from the end and echoed back; the client supplies the timestamp offset, because + copying timestamps does not preserve position. +- **Losing a peer must stop its work, not merely forget it** — anything holding a + resource is shut down on the way out, or a closed tab transcodes for the length of + the credit timeout. +- **Resume positions are per file, per device, in local storage.** No protocol, and + nothing new learns what you watch. + +--- + +## 9. Group applications + +### 9.1 The plug-in architecture + +A group's UI is a **set of pluggable applications**, not one page. Two things +motivated the split: one file had become the thing every unrelated change touched, +and the roadmap wanted several more group-level surfaces — none of which need a +protocol change, because the indexer already classifies files as video, audio or +image and they read the same index, chunk and stream messages the explorer already +uses. + +``` +group-page.js ─┬─ the shell: the connection, the file index, the tab bar, + │ the video/preview modals — nothing app-specific + ├─ apps.js ─── the registry: [{ key, icon, labelKey, Component, Settings? }] + ├─ chat-app.js · files-app.js · video-app.js · music-app.js · photos-app.js + └─ group-settings.js — not an app, always present, never toggleable +``` + +Settings **is not an app and cannot be disabled** — it is the one way back if +everything else were turned off. + +Shared infrastructure lives in its own modules (`icon.js`, `file-utils.js`, +`hub-client.js`, `settings-ui.js`, `folder-tree.js`, `source-merge.js`) rather than +being re-exported from the shell, because **the shell importing an app that imports +the shell is a cycle**, and ES modules answer that with a temporal-dead-zone error +at first render: the component simply does not appear, with nothing in the console +to say why. + +### 9.2 What every application receives + +The shell builds one props object per render and **spreads** it into whichever +application is active. Every registered component gets the same context and +destructures what it needs — a new application does not get a bespoke prop list. + +| Prop | Why it is here rather than local state | +|---|---| +| `entries`, `nodeDirs`, `nodeRoots` | the group's file index. Chat needs it too, for image attachments — lifting it avoids two copies going stale against each other | +| `applyIndex(...)` | anything that mutates files calls this, so every application sees the result | +| `onPreview(entry)` | opens the shell's modal; an application does not own modal state | +| `transportRef`, `gekRef` | **refs**, never state, so a reconnect does not re-render every application | +| `deviceReady` | **the exception, and why it is a prop.** A ref not re-rendering is right for a transport reached into on demand and wrong for a *fact about the connection* an application renders from | +| `mayUpload` | computed once; a second derivation would eventually disagree with the first | + +An application that needs local state owns it. One pattern is worth carrying: **any +notion of "current location within the group" resets on group change**, because a +directory from the group just left rarely exists in the one just entered. + +### 9.3 Enablement and settings + +`enabled_apps` is a per-group setting on the same pattern as everything else the +operator decides: **stored on the node** (`roster.db` — not the hub, not +`node.toml`, for the reason in §1.3 and §6.8), **changed by a signed operator +instruction**, and **enforced by the node** refusing an unrecognised or empty set. + +Validation happens before a signature is ever requested. The whole set is signed in +one message rather than one op per application, so ticking several boxes costs one +signature. The subject is the sorted, comma-joined list, built identically on both +sides so the two arrive at identical bytes. + +The node's allow-list is the server-side enforcement — **a client that names an +application this node does not know is refused**, and an application the node +refused could not demonstrate anything. That entry and the client's registry line +are the whole of what adding an application costs. + +A group that has said nothing gets **Chat and Files**. `files` is **always enabled +and not toggleable**, and is added to the list at every writer so the two agree. +The ability to hide it was misleading: the protocol permits root exploration +regardless, so hiding the tab only ever misled. + +**A group with no stored set gets the default, never the whole registry.** Falling +back to everything registered would turn on an application nobody chose — including +one shipped behind a development flag. + +Changes are broadcast to everyone connected, so a disabled tab disappears without +waiting for a reconnection. A client that has not yet received the list shows +everything registered — a node that predates an application hides nothing. + +**An application's directories are the same shape one level down**: one generic +signed op (`app_directories`) keyed by the application's own registry name, stored +under `<key>_directories`, one MNP message, one loopback route. Adding an +application adds **no function, no message type and no route** — which is what +"plug-in architecture" has to mean to be worth the phrase. + +> **The per-application variants are gone**, and the reason is the interesting +> part. Three messages, three signed ops, three handlers and three `ops` wrappers +> were the same instruction three times, differing only in the key they wrote and +> whether they carried a string or a list. That shape is what made adding an +> application mean adding a message type, an op, a handler and a widget — and it +> meant three validation paths, of which the older ones validated nothing: a typo +> was stored, matched no entry, and the application showed an empty tab with **no +> way to tell "misconfigured" from "no files yet"**. One op has one validation +> path, and an unknown application name is refused rather than stored. +> +> What stays is a *read* fallback: the roster still reads the older per-app keys +> out of its settings table, because that is a key on an operator's disk rather +> than on the wire, and a node upgraded into this must find its own configuration +> (§5.6). + +**Each application's settings pane is its own file**, named in its registry entry, +and every pane takes the same props and nothing else. The split is the point: + +> **What every application has, the page does generically; what one application +> alone has, the pane does itself.** Pointing an application at folders goes +> through the shared saver; a third-party credential or a per-group switch is the +> pane's own business, made with the transport it is handed. + +An application that only needs directories therefore touches neither the settings +page nor the shell. + +**The folder picker asks the node for nothing.** The tree is derived from paths the +client already holds, so it shows what the group's index contains and no more. +There is no folder-browsing protocol and this does not add one. + +### 9.4 Adding an application + +1. **`<name>-app.js`** exporting a component with the standard props shape. +2. **Register it** in the registry. `key` is the wire identifier: it must match the + node's allow-list and it is the row the application's directories are stored + under. **One identifier per application, everywhere.** +3. **`<name>-app-settings.js`** if it has anything to configure. Do not import the + settings page — that is the cycle in §9.1. +4. **Add the key to the node's allow-list.** +5. **i18n:** at minimum a tab label key **in all ten catalogues**. A settings key + added to the client must be added ten times; write the table and generate the + insert. +6. **Add both files to the asset fingerprint list.** A module reached through the + registry is not imported by name anywhere, so nothing else would notice it + changing and a browser would keep serving the cached copy. +7. **Add the file to the test file-set lists** — hook ordering, sticky headers, + transport contracts. A file missing from those lists is never checked, which + fails silently rather than loudly. +8. **Toolbars pin.** A toolbar is a direct child of the page root and is opaque, or + content scrolls visibly through it; if anything pins below it, it must publish + its own height, which is never a constant because it wraps on a phone. An + application with no toolbar renders none — an empty band still holds a strip of + the page open. + +No protocol change, no hub change, no daemon change. Steps 4 and 7 are the only +node-side and test-side touches, and both are allow-lists. + +**A reference application exists in the tree behind a development flag.** Every +other test of this architecture reads source for the *absence* of application names, +which proves nobody wrote a special case — not that a new application works. +Writing a real one immediately found two places the claim was only nearly true. + +### 9.5 Files + +The explorer: roots, folders, sorting, selection, upload where the current root is +writable and available, download, folder-as-zip, and the eject/plug control beside +each removable root. Entries from unavailable roots are filtered out. + +### 9.6 Chat + +Messages, threads, attachments, and link previews (§6.5). The composer gates on the +connection having identified a device (§3.3) and on a writable root existing for +attachments; the attachment directory is a single writable folder chosen by the +operator, and the paperclip is disabled with a stated reason when it is not usable. + +Own-ness is decided from the **account id**, in one place, never by comparing +display names — and the optimistic local echo carries an explicit flag rather than +inventing an identity for itself. + +### 9.7 Videos + +A poster browser over the video files in the group's configured folders. + +**Two modes**, both driven by the index: a **poster grid** with third-party +metadata and artwork, falling back to a thumbnail card with the cleaned filename +when the lookup returns nothing or a low-confidence match; and a **flat, +folder-driven list** with no third-party dependency, which keeps working with the +service switched off node-wide. The mode toggle is a **per-device display +preference** in local storage — it has no authority implication, so it needs none +of the signed-op treatment. + +**Scoping to folders is not a display preference**, because it decides what *every* +member's tab shows. It is a per-group setting changed by a signed op, broadcast to +connected members, and **validated before a signature is requested**: a candidate +path is resolved against the group's real root set and must name a real, readable +directory, so a stale path never reaches the operator's browser as a signing prompt. + +**Grouping is by unit, not by file**: one card per film, one per show — expanding to +seasons and episodes. Seasons carry their own text where the service supplies it, +falling back to the show's. An operator can **correct a wrong automatic match**, and +correcting one applies to the unit rather than to a single file (for shows) or to +the one file (for films), because those are the units each actually is. + +The filename parser is a fallback, and directory context is what bare-filename +parsing cannot supply. Matching is a **scored ladder** rather than the first +candidate to clear a threshold, with a year-exact rescue for a weak top hit. + +**Thumbnails, probes and metadata are node-side** (§6.5). Index-time probing runs in +its own bounded pool after a file is first seen: the file appears in the index +immediately with size and hash, and an index delta fills in the technical and parsed +fields once ready. **No scan is blocked waiting for enrichment.** + +### 9.8 Music + +An album browser and a player over the audio files in the group's configured +folders. + +**Metadata mostly already exists in the files themselves**, which is the real +difference from Videos. The order of trust is embedded tags, then filename and +folder parsing for what tags do not supply, then a third-party lookup for canonical +spelling, a missing field, or cover art where none is embedded — node-side and +cached like any other enrichment, and **needing no credential**, unlike Videos. + +**No playback protocol change is needed at all.** A track is a few megabytes, so +playback reuses the ordinary download-and-decrypt pipeline and hands a blob to an +audio element. No streaming request, no transcode pool, no stream slot, nothing +added to the node's streaming machinery. The one exception is narrow and +extension-gated: two container formats tag perfectly and decode in no mainstream +engine, so for those the node performs a **one-shot whole-file conversion**, cached +under its own content hash and served through the ordinary chunk path. + +**The player is persistent across tabs**, at shell level: closing the tab must not +stop the music. The next queued track is prefetched while one plays — client-side +only, a small in-memory cache evicted as the queue moves. + +Music is scoped to folders on the same mechanism as Videos. The reason is not cost — +tag reads are cheap — it is **mixing**: a shared tree with more than one kind of +thing under it puts everything into one undifferentiated view with no way to narrow +it. + +### 9.9 Photos + +An album browser over the image files in the group's configured folders. It is +**smaller** than Videos and Music, deliberately, in three ways: + +- **Several root folders rather than one.** +- **One album-grid view, no mode toggle**, because there is nothing to fall back + from. +- **No third-party service at all** — there is nothing to match a photo *to*. It + already is what it is, per its own folder and filename. + +An **album is a directory**, exactly as a season is a folder in Videos. Thumbnails +are node-side, orientation-corrected, and delivered through the same chunk path. + +**EXIF is read locally on the node** and narrowed on purpose: a capture time and a +camera, and **never GPS**, anywhere, in any field a client receives. The claim this +supports is precise, and the one it must not make matters more: + +> GPS **is** in the file, for most phone photos, in the original bytes any member +> with file access can already download. What this design controls is what the +> *application* computes and surfaces — not what the underlying file contains. + +### 9.10 Playlists — designed, not built + +Playlists are the first feature to need **per-account state that spans several +groups on several nodes**, so the shape is settled here even though the code is +not written. Full design: `playlists.md`. + +A playlist belongs to **one account and is never shared with other members.** That +scope is what keeps the merge problem small. + +**Where the state lives.** Not the hub — and the operative rule is narrower than +"the hub stores nothing about a user", because it already holds small per-account +preferences behind an allowlist. The rule is: + +> **No content metadata on the hub.** + +A playlist is literally a list of content hashes of private-group files, plus the +titles needed to render while nodes are offline. That is the exact object **H7** +removed from the hub, and it is the same rule that keeps resume positions local: +*nothing new learns what you watch.* An encrypted blob on the hub is technically +trivial and is still refused. + +It lives on the node instead, as **an opaque per-account blob in `bundles.db`** — +the same shape as the keypair bundle, which the node already stores and cannot +read. **No new trust boundary**: the node is not asked to hold a kind of thing it +does not already hold for that same account. + +**The key is the one thing that must not be got wrong.** Identity keys are per node +(§3.2), so a blob encrypted under one is unreadable from every other node — the +precise opposite of the requirement. The only secret an account holds *everywhere* +is the bundle key, so `playlist_key = HKDF(bundle_key, info = +"meshbay:playlists:v1")`: one derivation at sign-in, two handles, no second +Argon2 run, and a purpose-separated subkey rather than the bundle key reused with a +different AAD (§4.4's rule). The nonce is 96 random bits and never a counter, for +exactly the reason chat's is (§4.5): two devices of one account derive the *same* +key, which is the point. + +**Merge is the hard third, and the granularity is what makes it tractable.** The +unit is **one playlist, not the collection**; revision counters order writes, never +the wall clock; and **a deletion is a tombstone, never an absence** — an absence is +indistinguishable from a device that has not seen the addition yet. A node that is +offline for a month therefore cannot corrupt anything: it holds an older revision of +some playlists and is overwritten per playlist, not wholesale. + +**It adds no dialing and no new streaming path.** Sync rides connections the client +already makes, and playback is unchanged (§9.8). + +### 9.11 Cross-group search and source merging + +The Search page mounts the same media applications across every group the reader +belongs to. The difference lives entirely on the entries, in fields the group page +never sets: which group serves an entry, that group's transport and key, a +connection generation to use as a refetch key, and the merged source list. + +**A file shared by two groups is one entry, not two.** Identity is the content +hash: two entries with the same id are the same file, whatever group announced them +and whatever their path. Within one group this cannot arise, because the index is +already keyed by hash; the duplication is created by concatenating N independently +keyed indexes, and by nothing else. + +**One source is chosen per logical unit** — a film, a show, an album, a photo album — +not per file, so a show's episodes never stream from two different nodes. A group +hosted by the local node wins; otherwise the pick is deterministic and +pseudo-random, stable for one reader and spread across readers. If the chosen source +is unreachable the unit fails over. **Which source was picked is never shown**; the +badge names the group when there is one source and counts them when there are more. + +Two rules for a new application here: + +- **Use the shared source tag rather than an entry's group name** — a merged entry + has several groups. Pass it the whole unit, not the entry the card was drawn from, + which is usually chosen for its thumbnail and would under-report. +- **Never re-derive unit keys.** Call the application's own exported grouping + function. A copy keeps agreeing until one of them changes, and the symptom is a + show whose episodes stream from two different nodes. + +**The Files explorer is deliberately not merged**, and not "mostly not": there each +group is a top-level folder, and merging would remove a file from one of them. A +test refuses a build that changes this. + +One consequence is deliberate and is not a bug: albums are keyed by directory, so +two groups whose roots have *different* basenames put the same photo into two +differently-named albums, and the merge — scoped to a unit — leaves it in both. They +are two albums. + +--- + +## 10. Filesystem portability + +**exFAT and NTFS on Windows are the common case, not an edge case.** Most users are +expected to share from an external drive. The consequences below are correctness +requirements, not compatibility notes. + +| Property | What has to be true | +|---|---| +| **Case-insensitive, case-preserving** | The index needs a canonical identity and a **case-folding collision check** at scan time, reported to the operator rather than resolved silently. One directory indexed as two roots is the same problem one level up | +| **Unicode normalization** | A name written on one platform in decomposed form and on another in composed form are different byte strings. **Normalize to NFC for identity and comparison; preserve the original bytes for display and for opening the file** | +| **Reserved names and characters** | A group indexed on one platform can hold names another cannot create. The client sanitises on save **and says so**; the upload allowlist is the intersection across platforms, or some files are simply undownloadable | +| **Path length limits** | Use extended-length paths on Windows, in the node and the client alike | +| **Coarse timestamps, local time** | mtime alone is not a change detector. Size plus mtime with tolerance, and rehash when in doubt | +| **Watcher reliability** | Change notification drops events under load on Windows, and inotify is unreliable on a FUSE mount. **Periodic reconciliation is mandatory on both platforms** | +| **No symlinks, no POSIX permissions** | Simplifications: nothing to defend against, and the node runs as the user anyway | + +**Case folding is for comparisons the code makes itself** — index identity, +collision reporting, root names, nesting checks. It is *not* needed for the +no-overwrite rule, where the filesystem's own case-insensitive `stat()` already +gives a colliding upload a free name. + +**A volume that disappears freezes its root's subtree** and never empties it +(§6.2). Emptying propagates deletions for a whole library as though the owner had +erased it. + +**Never assume POSIX, systemd, or case sensitivity.** One platform-specific trap +with no counterpart elsewhere: a service unit with filesystem-protection options +gets its own mount namespace, so a volume mounted on the host *after* the service +started is invisible inside it — the directory reads as empty with everything else +configured correctly. + +--- + +## 11. Platforms + +### 11.1 NAT traversal, measured + +**QUIC native path** — residential ISP A to a hosted VPS: port-restricted cone NAT, +direct connection established from the server socket's own probe. + +**WebRTC browser path** — mobile and laptop clients to nodes behind two different +residential ISPs: + +| Path | Result | +|---|---| +| LAN, IPv6 direct | OK, ~100 ms | +| Mobile data, IPv6 inter-network | OK, ~600 ms | +| Mobile data, IPv4 only, STUN hole-punch | OK, ~650 ms | +| Laptop → second ISP's node, IPv6 inter-network | OK, ~7 s | +| Laptop → second ISP's node, IPv4 only, STUN hole-punch | OK, ~6.9 s | + +**Two ISPs validated, both without TURN.** The hub relays under a kilobyte of +signaling; the data path is peer to peer. + +### 11.2 Windows + +The bulk of the codebase is portable because the portability rules in §10 were +treated as correctness from the start. What the port needed is registered as +**W1–W9** (§13.7) and is done; packaging is built and awaits a clean-machine run. + +Two Windows-specific design points worth stating here: + +- **Autostart has two modes, chosen at install and switchable afterwards** from the + Node page: a per-user startup launcher (the default) and a scheduled-task service + mode. A per-user default is right for the desktop persona; a service is what a + machine that must serve while nobody is logged in needs. +- **A user service unit cannot carry a system unit's user directive.** Two unit + templates exist, held apart by a test that parses directives rather than + searching the file — searching matched the *comment* explaining why the directive + is absent. + +The hub stays Linux. macOS is not planned. Windows on ARM and store packaging are +out of scope. + +### 11.3 Android + +A client, not a host. The platform is hostile to *hosting* a node — background +execution, storage, battery — and fine as a *client*, which is one of the reasons +enrichment is node-side (§6.5). + +### 11.4 Casting + +An HTTP relay in the desktop client serves a standard fragmented-MP4 stream that +any LAN renderer can play; the relay is device-agnostic. Chromecast discovery and +control ship. DLNA/UPnP is designed and not built: it is a second device backend +beside the first, not a second relay. + +--- + +## 12. Testing posture + +**Security tests are negative assertions** — "this attack does not work" — and are +verified to fail against the pre-fix source before being trusted. A suite that +tests only that features work will happily **pin a vulnerability in place as +expected behaviour**, and a refactor that accidentally fixed one would be reported +as a regression. That has happened here, to four findings at once. + +Rules that follow from what has actually escaped this suite: + +- **A syntax check validates names not at all**, and a module syntax check must + force the module parser or it accepts template syntax pasted into an object + literal and reports success. +- **A second implementation of the client proves the protocol and nothing about the + client.** An end-to-end harness written in the right order by construction cannot + see ordering or lifecycle faults. Source-reading tests are weak evidence and are + sometimes the only evidence available; prefer ones that **re-derive** a value from + the source over ones that restate it. +- **A test that models a fix agrees with it by construction.** Lift the real + functions out of the source *as text* and execute them; model the environment, + never the code under test. +- **A test that reads source and inspects "the first" occurrence of a call stops + guarding anything the moment a new call is inserted before it** — and keeps + passing. +- **Measure, do not read.** A stylesheet does not tell you where anything lands; a + browser measuring the real stylesheet does. Assert on geometry, and check the test + fails with the fix removed. +- **A fixture narrower than real data tests the fixture.** Names are the one thing a + file browser cannot be given short. +- **Pace a probe like the real thing.** A stress probe fast enough to finish inside a + timeout hides every time-based fault. +- **"It works now" is not evidence against a race.** Force the worst case. +- **Exercise every browser's branch of a shared path before shipping a fix for one + of them**, and finish with a live pass: launching the real thing finds what source + reading cannot. +- **A passing suite over code nothing calls is evidence about that code, never + about the product.** Two full protocol implementations sat green and unreferenced + for months (§13.3 **L7**). Before trusting a suite, check that production imports + what it tests. +- **A comment that contradicts the constant beside it is worse than no comment** — + one of them is wrong and the reader cannot tell which. The same holds for a + document: a number restated away from its definition is a number that will drift. + +--- + +## 13. Register of labelled findings and decisions + +Each entry states **the rule the label names today**. Where the label originally +named a defect, the subject is given in one clause so a code comment citing it can +be understood, not so the incident can be retold. + +### 13.1 First review (design review) + +| Label | The rule it names | +|---|---| +| **C1** | Group chat has **no shared mutable sending state**. A pairwise ratchet shared across a group produces key and nonce reuse; the design that shipped has no sending state at all, which is stronger than partitioning it per device (§4.5) | +| **C2** | Tokens carry a `groups` claim and **the node verifies membership before serving content** (§5.2) | +| **S1** | Every admin endpoint has an authorization check | +| **S2** | Email addresses are **encrypted at rest**, with a blind index for lookup | +| **S3** | Token revocation reaches nodes over the hub socket, and the denylist persists (§7.5) | +| **S4** | AEAD nonces are **96-bit**, per NIST SP 800-38D | +| **S5** | Refresh tokens **rotate, one-time-use**, with family-based reuse detection | +| **M1** *(first review)* | Account enumeration is a known, bounded property of the account-management endpoints; `login`, device auth and reset-request are uniform | +| **M2** *(first review)* | Node TLS certificates are transport confidentiality only; identity is the Ed25519 key checked at the MNP layer | +| **M3** *(first review)* | Rate limits on key-adjacent endpoints | +| **M4** *(first review)* | Delegation is designed and deferred; the role check is written so it drops in (§3.4) | +| **M5** | **Chunk key derivation uses `info`, with `salt=None`** — correct HKDF usage, because the group key is already uniform CSPRNG output (§4.3) | +| **M6** *(first review)* | Argon2id production parameters are applied, and recorded per envelope so they can be raised (§4.6) | +| **N1–N5** | Notes, no action: the forward-secrecy model is appropriate to the deployment; login error messages are correct; the hub's legal exposure model is well-positioned; the NAT probe payload is fine; the web/CLI derivation difference is by design | + +### 13.2 Node sovereignty + +| Label | The rule it names | +|---|---| +| **NS1** | The client proves possession of the group key in the handshake, and the node verifies it | +| **NS2** | Admin operations are **Ed25519 challenge-response over a structured transcript**, never a token (§5.4) | +| **NS3** | **The node never serves the group key in plaintext.** There is no request message for it; the constants are gone from the protocol | +| **NS4** | **Operator authority comes from the node's roster and from nowhere else.** No auto-pin from the keystore, no resolution through the hub, no config key — a config naming one is warned about and never obeyed (§3.4, §6.1) | +| **NS5** | The proof is **bound to the transport channel** (DTLS fingerprints / certificate hash), so a signaling relay that substitutes its own cannot produce it (§5.2) | +| **NS6** | **`sender_id` is enforced from the authenticated session, never the wire.** It is what the store keys on; it is not what authenticates a message — the device signature is (§4.5) | +| **NS7** | The node authenticates to the hub with **Ed25519 and no password**, and its token's scope is refused for group management (§7.2) | +| **NS8** | **The node refuses connections when it holds no group key.** There is no bypass switch | + +### 13.3 Second review (code review) — the default numbering + +**Critical** + +| Label | The rule it names | +|---|---| +| **C1** | **All content travels over the authenticated protocol.** The node exposes no unauthenticated HTTP surface; the per-group file API that served private indexes and plaintext files on all interfaces was deleted rather than repaired, because it duplicated MNP without any of its controls | +| **C2** | A node's signaling identity is **resolved against the database and derived from it**, never taken from the client's first message (§7.2) | +| **C3** | **Authentication is mutual**: the node proves key possession over the client's nonce and signs the transcript, and the client verifies both and pins the key (§5.2) | +| **C4** | **Keypair bundles are per node, Argon2id-protected, and closed for native devices** — and **open for any account that also uses a browser** (§3.7) | +| **C5** | The two halves below, cited together where a comment means "a member must not be able to write what the node then trusts" | +| **C5a** | Uploads cannot overwrite, are allowlisted, ordered and capped, and ownership is signed by the uploader (§6.4) | +| **C5b** | **No key material arrives from outside.** The member-supplied bundle message does not exist; the node generates every copy of a group key itself (§4.2) | +| **C6** | **One handshake implementation, shared by every transport.** A transport that accepts a bare token is the standing example of what an opt-in compatibility branch costs (§5.2, §5.6) | + +**High** + +| Label | The rule it names | +|---|---| +| **H1** | **Per-group isolation on a multi-group node**: the chat store, the peer registry and the broadcast set are per group | +| **H2** | Every value that originates outside the node — filenames chosen by members, usernames originating at the hub — is escaped where it is rendered. A CSP contains exfiltration but cannot prevent injected inline script, so escaping is the actual fix | +| **H3** | **No public key is ever fetched from a directory to wrap a group key for.** The node wraps for a key the recipient proved possession of, bound to an account by a code the hub never sees (§3.4) | +| **H4** | Revocation reaches nodes, drops live sessions, and **persists across a restart** (§7.5) | +| **H5** | An admin challenge is a **structured, domain-separated transcript naming the operation and subject**, and the client refuses to sign anything that is not what the user asked for (§5.4) | +| **H6** | Unauthenticated work a node will do is bounded: a small pre-handshake buffer, a transcode semaphore, per-user pending-offer caps, and a membership check on signaling (§7.2) | +| **H7** | **Only public groups register content hashes with the hub.** Private groups register nothing, and the swarm route requires authentication | + +**Medium** + +| Label | The rule it names | +|---|---| +| **M1** | `group_id` is **mandatory** on the handshake — there is no fallback to the node's first group (§5.2) | +| **M2** | The node keystore uses the full Argon2id parameters, recorded per envelope (§4.6) | +| **M3** | Operator authority is the roster pin, established locally by pairing. **Asking the hub for the operator's key — the obvious-looking fix — would let the hub install itself as node administrator** (§3.4) | +| **M4** | **Every reply carries the request id it answers.** Arrival-order matching is a guess that fails silently and asymmetrically (§5.3) | +| **M5** | The index has one protection level, not one per transport (§4.4) | +| **M6** | Audit rows are attributed to the row's own subject, never backfilled across rows | +| **M7** | Client addresses are taken from a trusted proxy's rightmost hop, in one helper (§7.2) | +| **M8** | A node announcement requires **proof of possession** of the key it announces | +| **M9** | Node-scoped tokens are refused on the client path (§5.2) | + +**Low** + +| Label | The rule it names | +|---|---| +| **L1** | The wire contract carries no constants for messages that do not exist | +| **L2** | MNP negotiates versions explicitly and refuses with a code (§5.6) | +| **L3** | Errors returned to a peer name no filesystem path and no exception detail | +| **L4** | Transcripts are **length-prefixed**, and an empty channel binding **raises** rather than degrading the proof to nonce-only (§5.2) | +| **L5** | The hub serves a CSP and security headers on the application it serves | +| **L6** | Registration validates the email field it declares | +| **L7** | **A module nothing imports is not a protection**, and it is not kept. The sender-key and ratchet implementations this finding named were unreferenced for months and are now deleted; a green test suite over uncalled code is evidence about that code, never about the product (§4.5) | +| **L8** | An uploader record identifies a file by its id, not by a name at a root | + +### 13.4 Third review + +| Label | The rule it names | +|---|---| +| **H1** *(third review)* | **Moderator is not administrator.** The user-patch handler is split by field so a moderator cannot write `role` (§7.5) | +| **H2** *(third review)* | Content reporting requires authentication, distinct reporters and a rate limit, and is refused when public groups are off (§7.5) | +| **M1** *(third review)* | The registration CAPTCHA gate is **unconditional** when a captcha is configured — never conditioned on a field the real client always sends (§7.7) | +| **M2a** | `sender_id` comes from the authenticated session (**NS6**). Now guaranteed by there being **one** chat implementation: QUIC does not carry chat at all (§5.1) | +| **M2b** | Chat broadcast is per group (**H1**), on the one transport that carries chat | +| **M2c** | No transport runs a synchronous media process on the event loop, and every one is capped | +| **M3** *(third review)* | Link-preview SSRF is gated: rate limit, port allowlist, globally-routable check, per-hop re-check, connect-address re-check, size guard (§6.5) | +| **M4** *(third review)* | Federation binds a pushed row's source to the signer, checks the token audience, caps the push, rejects replays, and scopes revocation to the peer's own entries (§7.6) | +| **M5** *(third review)* | A CSP and security headers apply to the hub-served application, verified against the running app — a mis-tuned CSP shows as a blank page | +| **M6** *(third review)* | **Withdrawn.** It misread the node registering a hub membership during the CLI invite flow — which is deliberate — as authorization drift | +| **L1–L11** *(third review)* | Opportunistic hardening: relay-registry proof of possession; delete orphaned modules rather than leaving them to be rewired; decide and document account enumeration; an aggregate upload quota; header-only control-API tokens; a freshness bound on revocation replay; state that the exact-hash content check is structural; validate group-name length and charset; require `exp` and bind an audience on token decode; key the rate limiter through the same client-address helper as everything else; keep diagnostic logging truncated | + +Two structural recommendations from that review stand as rules: + +- **Make transport parity a test, not a habit.** Shared helpers in common code with + a test that fails if a transport calls a chat or stream path that bypasses them. +- **Every new outbound or cross-trust surface needs a rate limit and a named + adversary in the same commit** (§6.5). + +### 13.5 Standing trust limits + +| Label | The rule it names | +|---|---| +| **T1** | **The password split.** The hub never sees a passphrase; it holds a verifier for a client-derived `auth_key`. The passphrase floor can therefore only be enforced client-side (§3.1) | +| **T2** | The hub was the key directory. **Closed** by admission redesign, not by safety numbers: the invite path reads no directory at all (§3.4). Reclassified as **H3** | +| **T3** | **The hub serves the SPA. Accepted permanently for browser users.** It is the only remaining way an active hub reads content, it is an artifact-level attack rather than a silent lie, and it does not exist for a native client — whose value is realised by reproducible builds, not by packaging (§2.3, §8.2) | + +### 13.6 Chat design findings + +| Label | The rule it names | +|---|---| +| **F1** | **A sender's signing key is bound to the roster**, never generated fresh inside a distribution any member can produce. Otherwise every member can replace another's chain and forge them silently | +| **F2** | A second device cannot destroy the first's ability to send or be read — there is no per-device chain to drop (§4.5) | +| **F3** | There is no skipped-key cache to grow without bound | +| **F4** | **Rotating the group key is a re-wrap, not the destruction of the archive.** Epoch keys are wrapped at delivery, never at rest (§4.5) | +| **F5** | **History is served to devices that were not present**, which is what "load older" means and is incompatible with a ratchet | +| **F6** | Ciphertext is not corrupted on the history path | +| **F7** | One account can hold two connected devices: the peer registry is keyed per connection, and a broadcast excludes the sending **session**, not the sending account | +| **R1–R22** | The regression register for that work — each row a concrete way the feature could break something that already worked, with the guard that stops it. R1 (rotation keeps history readable), R5 (an older client is refused with a stated reason, not left showing gibberish), R11 (own-ness from the account id), R15 (no epoch key in a plaintext store), R16 (retention deletes messages, never epoch keys) are the ones cited elsewhere | + +### 13.7 Windows port + +| Label | The rule it names | +|---|---| +| **W1** | Platform directories: no hardcoded XDG paths | +| **W2** | Signal handling is platform-guarded | +| **W3** | Daemon lifecycle: a per-user startup launcher by default, a scheduled-task service mode offered, switchable after install (§11.2) | +| **W4** | Packaging: one per-user installer carrying client and node, with media tools bundled | +| **W5** | File permission calls are skipped where they have no meaning | +| **W6** | Media-tool discovery fails at startup with a stated reason rather than at first use | +| **W7** | CLI messages name the right paths and commands for the platform | +| **W8** | The test suite is green on the platform, encoding included | +| **W9** | ICE interface matching works where an adapter's name is a GUID rather than a kernel name, and **fails open** | + +### 13.8 Desktop client decisions + +| Label | The decision | +|---|---| +| **E1** | **Electron**, plus an optional Python sidecar for hub-less `group://` over QUIC | +| **E2** | **Device linking**: an already-pinned key countersigns, bound by a one-time code the new device generates (§3.3) | +| **E3** | **Hybrid account creation**: register with a passphrase-derived `auth_key` — the recovery path — then authenticate day to day with a device key | +| **E4** | **Signed admin ops over MNP** for everything group-scoped; first run stays local; the loopback API is never exposed to the network | +| **E5** | LAN enrolment of a headless node is **out of scope for v1**, kept implementable | +| **E6** | **The browser SPA stays.** A native client must not prevent web use | +| **E7** | **Several named roots** per group, forming one virtual root (§6.2) | +| **E8** | **exFAT/NTFS and Windows are the common case.** Linux ships first; that is build order, not population (§10) | +| **E9** | **Group-related server state lives on the node. Always** (§1.3) | +| **O1** | Initial key setup in the pre-proof window — deferred; that window is where C4 and C5b came from | +| **O2** | A LAN enrolment door — one endpoint, bounded window, one-time code, closing permanently on success | +| **O3** | `device_policy {allow_bundle: false}`, signed by a pinned key — **the mechanism that actually closes C4** (§3.7) | +| **O4** | Isolating the node-admin panel from the process holding user keys | +| **O5** | An unlock key in the environment, for the **node** | +| **O6** | The engine version floor, verified rather than assumed | +| **O8** | A minimum client version in the hub version endpoint — **done** (§5.6) | +| **O10** | Canonical file identity across filesystems, defined once and shared (§10) | +| **O11** | A root **alias** where the basename cannot be used (§6.2) | +| **O12** | Derived data: **revised** — the node caches durably in its own `data_dir` (§6.5) | +| **O13** | **Hub identity pinning.** The client points at a hub by URL and nothing pins that hub's identity. Bounded, because a substituted hub can neither read content nor ship the code to a native client — worth doing all the same | +| **V1–V13**, **P1–P5** | Per-application open items: wording of a disabled-service state, whether artwork reuses the chunk path, cache TTL, multi-track surfacing, HEIC/RAW support, a fuller EXIF panel, lightbox preloading, album-boundary behaviour, cover selection | + +--- + +## 14. Decisions that are not revisited + +### 14.1 Structural + +1. Multi-group on a single port. +2. Signaling punch and connect through the hub socket. +3. Chat is a core feature, not a module. +4. **Chat encryption is a sealed archive with per-device keys and signed messages** — not sender keys, not a ratchet (§4.5). +5. Tokens carry group claims and the node checks them. +6. Admin roles are hub configuration. +7. Refresh-token rotation is family-based. +8. Email is encrypted at rest. +9. Argon2id parameters are versioned and migrate on login. +10. **Browser transport is WebRTC DataChannel with ICE/STUN.** +11. **The hub is a registrar and signaling relay, never in the data path.** +12. **Chat is stored on nodes, never on the hub.** +13. The web UI is a small-framework SPA: dark/light, responsive, i18n in ten languages. +14. Site-specific pages are an overlay, separate from the generic hub. +15. Video streaming is fragmented-MP4 remux on the node, source buffer in the client. +16. **ICE is primary for browser and native alike. QUIC is kept at parity for LAN, port-forwarded and hub-less access. TCP+TLS and the node HTTP API do not exist.** +17. **`punch_nat()` is a direct-connection helper, not a traversal stack** (§5.1). +18. **The desktop shell is Electron.** What is unchanged and non-negotiable: **UI assets ship inside the package and load from disk** (§8.2). +19. **A second device is admitted by device linking, not by an operator code** (§3.3). +20. **Private keys never leave the device on native clients.** Qualified: a browser has no durable storage of its own and still needs a bundle on each node, so C4 closes for an *account* only when it opts out of browser use. +21. **Hub minimisation is enforced by an acceptance test, not by policy.** The hub must be *unable* to see keys, content or file listings. +22. **No new code exchanges between people.** Safety numbers are refused for identity verification, permanently. The device-linking code is between a person's own devices and is unaffected. The total user-visible cost of the whole authorship story is **one notice**: *"this account's key changed"*. + +### 14.2 Client architecture + +| # | Decision | +|---|---| +| **D1** | **The hub keeps serving the web UI.** It is the zero-install path and it stays. What must then be true: a strict CSP, a signed digest of the served bundle that any third party can verify, an explicit reduced-trust notice, and docs that never claim end-to-end integrity for that path | +| **D2** | **A native client, offered alongside the SPA** — not as a replacement. The browser-extension options (an extension that *verifies* the served bundle; an extension that *ships* the UI) were analysed and are not taken up | +| **D3** | Transport: ICE primary, QUIC at parity, TCP and HTTP removed | +| **D4** | **Hub minimisation deferred, and may be dropped.** The hub stays in the trusted path by choice | + +**The reason a native client is justified, and the reason it is not justified.** It +is justified on *product* grounds: durable keys, no browser tab, background +connectivity, better video, hub-less access. It is **not** the fix for T3 unless +reproducible builds ship with it — a binary from the same operator relocates trust +rather than removing it. What genuinely changes is **detectability**: a browser +attack is one HTTP response aimed at one user, leaving no artifact; a native attack +requires shipping a build, which is hashable, archivable and comparable. + +Its real costs were under-weighted once and are recorded: **patch velocity** is +owned rather than inherited from a browser vendor, and the renderer parses +attacker-controlled content from nodes. Accepted deliberately. + +**For node operators specifically, the CLI beats every client.** The operator holds +the group key and is the content authority; a CLI removes their dependency on +hub-served code at a fraction of any client's cost. If only one thing were built +against T3, it should be that — and it was. + +--- + +## 15. State of the build + +### 15.1 Built and running + +The hub, the node daemon, both transports, the unified handshake, admission and +pairing, device linking with member-visible evidence, per-node identity, named roots +with RO/RW and eject/plug, the indexer with partial hashing, uploads, the sealed +index and sealed upload path, encrypted chat with epochs, video streaming with +seeking, transfer leases with queueing, pause and resume, the group-application +framework with Chat, Files, Videos, +Music and Photos, cross-group search with source merging, the operator CLI and +loopback control API, the desktop client through its identity and download stages, +account recovery, and the Windows port through packaging. + +### 15.2 Not built + +| | | +|---|---| +| **Stage D5** | Node management panel over the operator ops, root selection included | +| **Stage D6** | First-run wizard — detect, enable the unit, link, group, initialise, pair | +| **Stage D9** | Python sidecar — `group://` over QUIC | +| **Stage D11** | Windows clean-machine install verification and code signing | +| **Stage D12** | Release key, signed repositories, updates through the OS | +| — | DLNA/UPnP casting (§11.4) | +| — | Delegation (§3.4) | +| — | Tier 3 roster attestation (§3.3) | +| — | Playlists (§9.10) | +| — | Android client | + +### 15.3 Open, and why each is where it is + +| Item | Status | +|---|---| +| **C4** for browser-using accounts | Open until the signed bundle opt-out ships (O3) | +| **T3** for browser users | **Accepted permanently.** Removed for native clients, and that removal's value depends on reproducible builds | +| **Hub identity pinning** (O13) | Nothing pins the hub's key. Bounded, because a substituted hub can neither read content nor ship code to a native client | +| **Aggregate upload quota** | Per-file caps exist; a per-user or per-group total does not | +| Forward secrecy in group chat | **Given up deliberately and on the record** (§4.5). If it becomes a requirement it belongs in 1:1 DM | +| Metadata at the hub | Membership, and who posted in which group and when. A known leak, not a solved problem (§7.1) | +| The exact-hash content check | Structural, not functional (§7.5) | + +--- + +## 16. Concordance + +Code comments, tests and older documents cite sections of the documents this one +replaces. The mapping: + +| Cited as | Read | +|---|---| +| `draft-v5 §2`, `draft-v6 §4` — security claims | §2.2 | +| `draft-v5 §3` — transport, NAT traversal | §5.1 | +| `draft-v5 §4`, §4.1–4.4 — handshake, transcript, channel binding, mutual auth | §5.2 | +| `draft-v5 §5.1`, `draft-v6 §2.3`, `§2.4b` — privileged operations, key activation, authorship | §5.4 | +| `draft-v5 §5.2`, `draft-v6 §2.1`, `§2.1b` — uploads | §6.4 | +| `draft-v5 §5.2` — nothing derived beside the originals | §6.5 | +| `draft-v5 §5.2b` — removing a directory | §6.4 | +| `draft-v5 §5.3`, `§5.4` — operator interface, local admin UI | §6.7 | +| `draft-v5 §5.5`, `invite-pairing-v1.md` §3, §5, §7 — admission and key delivery | §3.4 | +| `draft-v5 §6.1`, `draft-v6 §2.5`, `§2.9` — hub role, group names | §7.1, §7.3 | +| `draft-v5 §6.2`–`§6.4` — node registration, signaling, client addresses | §7.2 | +| `draft-v5 §7` — cryptography | §4.1–§4.4, §4.6 | +| `draft-v5 §7.1` — the keypair bundle and C4 | §3.7 | +| `draft-v5 §8.1`, `§8.2`, `desktop-client-v1.md §2` — the two clients, the shell | §8.1, §8.2 | +| `draft-v5 §9`, `draft-v6 §5` — open items | §15.3 | +| `draft-v5 §10` — testing posture | §12 | +| `draft-v6 §2.2`, `desktop-client-v1.md §4`, `§4.5`, `§4.6` — device linking | §3.3 | +| `desktop-client-v1.md §4.8` — authorship, the tiers | §3.3, §5.4 | +| `desktop-client-v1.md §5`, `§5.1`, `auth-confirm.md` — accounts, change and recovery | §3.1, §3.6 | +| `desktop-client-v1.md §6.7`, `refactor-groups.md §1.1`, `§1.5b` — roots, RO/RW, eject | §6.2 | +| `desktop-client-v1.md §6.8`, `draft-v6 §3`, `WINDOWS-PORT.md` — portability | §10 | +| `desktop-client-v1.md §6.9` — a root that goes away | §6.2 | +| `desktop-client-v1.md §6.10`, `mediacenter.md §2`, `§5.2`, `§5.3` — views not a catalogue, derived data | §6.5, §9.1 | +| `desktop-client-v1.md §8.1` — URL space | §8.4 | +| `desktop-client-v1.md §10b`, `refactor-node-ui.md` — group settings, the node's admin surface | §9.3, §6.7 | +| `draft-v6 §2.7`, `apps.md §1`–`§3` — the application framework | §9.1–§9.3 | +| `apps.md §2b`, `refactoring-search.md` — cross-group search, source merging | §9.11 | +| `playlists.md` — per-account cross-group state | §9.10 | +| `apps.md §4`, `refactor-groups.md §3`, `§4.1` — adding an application | §9.4 | +| `draft-v6 §2.8` — instance policy, suspend vs revoke | §7.4, §7.5 | +| `draft-v6 §2.10`, `refactor-groups.md §1.7` — link previews | §6.5, §9.6 | +| `draft-v6 §2.11` — node settings | §6.8 | +| `draft-v6 §2.12` — STUN and ICE filtering | §5.1 | +| `mediacenter.md §3`, `§4`, `§5.4`–`§5.7` — Videos | §9.7 | +| `musicbay.md §2`, `§3`, `§4`, `§5`, `§6` — Music | §9.8 | +| `photos.md §2`, `§3`, `§4`, `§5` — Photos | §9.9 | +| `refactor-groups.md §1.6` — generic app-directory ops | §9.3 | +| `chat-sender-keys.md §5`, `§6`, `§8`, `§13` — chat encryption, sender authentication | §4.5, §3.3 | +| `indexing-v2.md` — partial-read hashing | §6.3 | +| `per-node-identity-v1.md` — identity keys per node | §3.2 | +| `captcha.md` — registration gate | §7.7 | +| `tmp-decisions.md` D1–D4 | §14.2 | +| `devel-phases-next.md` structural decisions | §14.1 | +| `cast-smart-tv.md` | §11.4 | + +Older references to `draft-v3` and `draft-v4` sections point into `old-draft.md`, +which is a historical archive and is not authoritative. diff --git a/docs/transfers-v1.md b/docs/transfers-v1.md new file mode 100644 index 0000000..f99424f --- /dev/null +++ b/docs/transfers-v1.md @@ -0,0 +1,1159 @@ +# Transfer slots, queueing, pause and resume — build record + +> **Status: built.** Leases, caps, queueing, the settings surface, the client +> widget, pause and resume all shipped; the compulsory-lease flag day is +> **MNP 3.0**. +> +> **The design is `MESHBAY_DESIGN.md` §5.5**, which states the lease model, the +> caps, and the reasoning behind the browsing exemption. Read that first. This +> document is kept for what a synthesis cannot carry: the failure-mode analysis +> in §5 (every way a slot can be lost, every way a client can be left waiting, +> and the invariant that checks both), the target-by-target resumability table in +> §6, and §12 — what a live pass found after the work was called done, which is +> the most useful part of the file. +> +> **Where this document and the code disagree, the code is authoritative**, and +> two places are known to disagree: the leaseless-read bound is 12 files per +> session with an idle expiry, not the 2 written in §3.4.1, and the wire section +> describes MNP 3.0 as pending when it has shipped. `MESHBAY_DESIGN.md` §5.5 +> carries the current values. +> +> Convention carried over and not negotiable: **a claim here names the failure it +> holds against.** "A slot is released when the transfer finishes" is not a +> design; "here is every way a slot can be lost and what reclaims it in each +> case" is. + +--- + +## 0. The short answer + +| Asked for | Feasible | Where the work is | +|---|---|---| +| `max_concurrent_downloads` / `max_concurrent_uploads` per node (default 8) | Yes | The node has **no concept of a transfer** today — that is the whole job | +| Per-member caps inside a group (default 2) | Yes | New per-group operator-signed setting, same shape as `apps_enabled` | +| "Max reached" refusal + client queues the rest | Yes | Better as a **node-side queue with an explicit grant**, not a client retry loop | +| A visible **waiting** state in the transfers widget | Yes | `TransferStore` already has a status field; it gains three states | +| Cancel | **Already works** (`transfers.cancel`, `transfers.js`) | Needs to also tell the node, which today it does not | +| Browsing — posters, thumbnails, previews, listings — untouched by the caps | Yes | §3.4. Structural, not a threshold: the three functions that draw a widget row are the three that take a slot | +| Pause with resume | Yes, **in three tiers** — see §6. Not every download target can be resumed, and the interface must not offer what a target cannot do | + +The risk is not in any one of these. It is that a capped resource plus a queue +is exactly the shape of bug that shows up as *"my download says waiting and +never starts"*, days later, on someone else's machine. §5 is therefore the +longest section in this document and is the one to review hardest. + +--- + +## 1. What exists today, measured + +Read this before designing anything: three of the assumptions one would +naturally make are false here. + +**A download is invisible to the node.** `pipelinedDownload` +(`static/file-utils.js`) sends `PIPELINE_WINDOW = 8` independent `file_req` +messages and reassembles the answers. Each one is served by +`_do_file_request` (`webrtc_server.py:3380`), which looks the entry up, reads +and encrypts one 1 MB chunk, waits for room on the channel +(`DOWNLOAD_BUFFER_HIGH`, `webrtc_server.py:253`) and sends it. **The node never +learns that a download started, and never learns that one ended.** There is +nothing to count and nothing to cap. This is the central fact of this work. + +**An upload is half-visible.** `_do_file_upload` (`webrtc_server.py:4471`) +keeps `self._uploads[f"{rel_dir}/{filename}"]` with `next_index` and a `.part` +file on disk. That state is **per session object and in memory**: a browser +that disconnects mid-upload leaves a `.part` file on the operator's disk that +nothing ever removes, and a retry starts at chunk 0 under a new `_free_name`. +So there is already a resume story here, it is just not reachable — and there +is already a leak. + +**Video streaming is capped, and its cap is a good model.** `_stream_video` +(`webrtc_server.py:5230`) takes `ctx["_transcode_sem"]`, refuses with +`"Server busy, retry shortly"` when it is empty, and every hard-won lesson in +this repo about slots — `_replace_stream`, `shutdown_tasks()`, `_spawn()` and +the garbage-collected task, `await proc.wait()` after `kill()` — is a lesson +about *not losing a slot*. That history is why §5 exists. + +**One bug found while reading, in scope for this work.** `ops.py:1362-1365` +hot-swaps the stream cap by assigning `webrtc._stream_sem` — an attribute that +does not exist. The real semaphore is `ctx["_transcode_sem"]` +(`webrtc_server.py:5220`), and `hasattr(webrtc, '_stream_sem')` is always +False, so **changing `max_concurrent_streams` from the Node page has never +taken effect without a restart**, contrary to §2.11 of draft v6. Two more +hot-swappable caps are about to be added next to it; fix it with one shared +helper rather than three copies of the same mistake. + +**Existing client state.** `TransferStore` (`static/transfers.js`) already +survives leaving a group, already holds the transport open until the last +transfer finishes (`releaseWhenIdle`), already cancels on sign-out, and already +computes a windowed speed. Its statuses are `running | done | cancelled | +failed` and its `run()` contract is a promise plus a polled `signal.aborted`. +That contract is the thing that has to change, and it is the only client-side +structural change. + +--- + +## 2. What a naive implementation gets wrong + +Worth stating, because each of these is a plausible first attempt. + +1. **Counting `file_req` messages.** A rate limit on chunk requests caps + throughput, not concurrency, and gives the client nothing to render as + "waiting". It also cannot distinguish a download from a thumbnail. +2. **Inferring a transfer from activity.** "A (session, file) pair with a chunk + request in the last 30 s is a live download" needs no protocol change and is + tempting. It makes the admission decision unobservable — the client cannot + be told *why* it is being refused, cannot be told when to try again, and a + paused transfer is indistinguishable from a crashed one. +3. **A client-side queue only.** The client already knows what it wants to + download; it does not know what the other five members are doing. A queue + held only by clients cannot order anything fairly and turns into a retry + storm against a busy node. +4. **A refusal the client retries on a timer.** This is the version that + produces the "stuck at waiting" report: the client polls, the node refuses, + nobody is at fault, and there is no evidence anywhere. The node must + **push** the grant. +5. **Gating every `file_req`.** Thumbnails, posters, cover art and audio + transcodes are served through the *same* `file_req` path + (`_try_serve_thumbnail`, `webrtc_server.py:3353`), and so are image and text + previews. Requiring a slot for all of them means opening a poster grid + queues behind a film. §4.4 handles this. +6. **Holding a slot while paused.** If pausing keeps the slot, one member can + pause eight downloads and close the node to everyone else. Pausing must + release. + +--- + +## 3. The design + +### 3.1 The lease + +Introduce **the transfer lease**: the node's record that a peer is transferring +something, held for the length of the transfer and released by name. + +``` +transfer_open { tr, kind: "download"|"upload", bytes, chunks, label_hint } + → transfer_state { tr, state: "granted" } + → transfer_state { tr, state: "queued", ahead: 3 } + …later… + → transfer_state { tr, state: "granted" } ← pushed, not polled +transfer_close { tr, reason: "done"|"cancelled"|"paused"|"failed" } + → transfer_state { tr, state: "closed" } +``` + +Six properties, each of which is load-bearing: + +- **`tr` is drawn by the client**, 16 random bytes hex, exactly like + `upload_id` (`transport.js:2208`). Re-opening after a reconnect with the same + `tr` is idempotent, so a reconnect cannot double-charge a member for one + transfer. +- **A lease is scoped to the connection, never to the account.** It dies with + the session, which is what makes the primary reclaim deterministic (§5). +- **A lease covers a job, not a file.** A directory zip + (`downloadDirectory`, `file-utils.js`) is dozens of files and is **one** + lease. So is a resumed transfer. +- **A lease is never persisted.** Not in `roster.db`, not on disk, not across a + node restart. A restart drops every session anyway; a lease that outlived the + process would be a slot nothing can release. +- **The node counts leases, not bytes.** What a slot protects is concurrency — + open file handles, disk seeks, and the channel buffer each transfer keeps + full — not bandwidth. +- **Every `file_req` for a real file carries its `tr`.** The exceptions are in + §4.4 and they are the delicate part of this whole design. + +### 3.2 Two caps, in one order + +| Cap | Scope | Default | Where it lives | +|---|---|---|---| +| `max_concurrent_downloads` | node-wide, all groups | 8 | `[node]` in `node.toml`, override in `roster.db` (§7) | +| `max_concurrent_uploads` | node-wide, all groups | 8 | idem | +| `max_downloads_per_member` | one group, one account, **across all their devices** | 2 | per-group, operator-signed (§7) | +| `max_uploads_per_member` | one group, one account, across devices | 2 | idem | + +Downloads and uploads have **separate pools**, and video streaming keeps its +own third pool (`max_concurrent_streams`) untouched. A member watching a film +is not charged a download slot, and a download does not make the next film +answer "server busy" — those are different resources with different costs and +merging them would make both caps meaningless. + +**Order of checks, and it matters:** per-member first, then node-wide. A member +at their own cap queues *behind their own transfers* and never consumes a +node-wide slot they would then hold while a second member has none. Reversed, +one member arriving first takes all eight. + +**"Per member" means per account, summed across their devices**, resolved with +`_sessions_of(user_id)` (`webrtc_server.py:3325`) — which exists for exactly +this reason, since device linking landed. Two browsers and a desktop client +signed in as the same person share the two slots. Anything else makes the cap a +function of how many tabs someone opens. + +### 3.3 The queue + +One FIFO per pool, per node, holding `(tr, session, user_id, group_id, +enqueued_at)`. + +- **Drained in one place.** `_release_slot()` is the only function that returns + a slot, and it is the only caller of `_pump_queue()`. Every path that ends a + transfer goes through it, in a `finally`. Two functions that both release + would be this repo's flow-control-accounting lesson (`CLAUDE.md`) one feature + later. +- **Head-of-line blocking is skipped, not waited on.** When a node-wide slot + frees, walk the queue and grant to the first entry whose *member* is under + their own cap. Granting strictly in order would let one member at their + personal cap stall the whole node. +- **A grant has an acceptance deadline.** 30 s. If no `file_req` bearing that + `tr` arrives, the grant is revoked (`state: "queued"` again, at the tail) and + passed on. Without this, a client that dies between the grant and the first + chunk holds a slot until the idle timeout. +- **The queue is bounded**: 32 entries per account per pool. Beyond that the + node answers `too_many_queued` and the client holds the rest in its own list, + sending `transfer_open` as its own transfers finish. Unbounded queues are how + a node runs out of memory politely. +- **Positions are pushed, throttled.** After each pump, recompute positions and + send `transfer_state {state:"queued", ahead:n}` only to the sessions whose + number changed, at most once every 2 s per session. + +### 3.4 What is *not* gated — posters, thumbnails, previews, navigation + +**Operator decision, 2026-09-08: browsing a group is never subject to a +transfer slot.** Not the poster grid, not the album covers, not the video +thumbnails, not the file list, not opening a photo or a PDF to look at it. A +member must be able to browse a group that is at capacity exactly as they +browse an idle one. This is a requirement, not a tuning parameter, and the +sections below are written to satisfy it structurally rather than by choosing a +lucky threshold. + +**Navigation proper never touches this path at all.** The file list, the +directory tree, the poster metadata and the album metadata travel as +`index_sync`, `index_delta`, `media_meta_req`, `music_meta_req` and +`link_preview_req` — different message types, sealed under the group key, with +no relationship to `file_req`. Nothing in this design can reach them. That half +needs no rule; it needs only to be verified by a test that fails if someone +later routes a listing through `file_req`. + +The `file_req` path carries three genuinely different things, and they are +distinguishable **structurally**, by what the id resolves to and by which +function asked: + +| What | Call site | Resolves to | Rule | +|---|---|---|---| +| Thumbnails, TMDB posters, cover art, cached audio transcodes | `MediaThumb` (`video-app.js:204`), and the Music/Photos grids through it | a **media-cache id**, not an index entry — `_try_serve_thumbnail`, `webrtc_server.py:3353` | **Never leased, never counted, never queued.** One chunk each, out of a bounded cache the node built itself | +| Looking at one file — a photo opened full size, a PDF, an image, a text file | `PhotoViewer` (`photos-app.js:171`), the Files preview modal (`files-app.js:610`) | a real index entry, fetched whole | **Not leased.** Bounded by §3.4.1 below, which no real viewer ever reaches | +| Downloading, and uploading | `downloadEntry`, `downloadDirectory` (`file-utils.js`), `uploadFile` (`transport.js:2192`) | a real index entry | **Leased.** These are exactly the three call sites that go through `transfers.start()` — the three that produce a row in the transfers widget | + +The last column is the whole rule, and it is worth stating as a sentence +someone can check by reading: **a transfer is something the transfers widget +shows. If it does not appear in that panel, it does not take a slot.** The two +sets are the same three functions, which is what makes this verifiable rather +than a matter of judgement at each new call site. + +#### 3.4.1 The bound on leaseless reads + +A `file_req` with no `tr` on a real index entry is served, subject to one +limit: **at most 2 distinct file ids in flight leaselessly per session**, with +no queue — the third is refused with `transfer_required`. + +Why this shape, and not the byte budget an earlier draft of this document +proposed: + +- **A viewer looks at one file.** The photo viewer shows one photo, the preview + modal one document. Two is already one more than any of them needs, and is + there so that prefetching the next photo stays possible. +- **A size threshold does not work here.** A RAW photo out of a camera is + 60–80 MB and is *browsing*; a 40 MB archive is a *download*. Size does not + separate them, and any threshold that let the photo through would let the + archive through too. What separates them is which function asked. +- **A byte-rate budget does not work either.** It would have to be large enough + for that same RAW photo, at which point it is large enough to be a download + channel. Concurrency is the thing being rationed, so concurrency is what the + exemption is expressed in. +- **What it costs.** A client that lies — labelling a bulk download as a view — + gets two files at a time per session instead of its member cap. That is the + residual, it is bounded, it is audited (`file_download` already goes to + `audit.db` on chunk 0), and it is the same class of statement as the cap + itself: **this is a fairness control among cooperating clients**, in the + company of `max_concurrent_streams`. It is not a defence against a member + determined to saturate the node's disk, and must never be described as one — + that member is a member, and the answer to them is `member revoke`. + +#### 3.4.2 The consequence for the interface + +A preview never shows "waiting", because a preview never queues. If the +2-in-flight bound is somehow reached, the modal reports a plain error and the +person tries again — it does not silently become a queued transfer in a panel +they were not looking at. + +### 3.5 Where the numbers are visible + +The node answers `transfer_state` with the current picture — `used`, `cap`, +`ahead` — so the client can say *"waiting — 2 of 2 of your slots are busy"* +rather than a bare spinner. The same counters go to: + +- the loopback API, `GET /api/transfers`, so the operator can see live leases + and queue depth from the CLI and the Node page, and +- a DEBUG line every 30 s: `transfer: d=3/8 u=1/8 queued=2 (skipped=1)`. + +That line is the `client_diag` lesson applied here: when someone reports a +transfer stuck at "waiting", this is the only thing that will say whether the +node ever had them in a queue. + +--- + +## 4. Wire protocol — MNP 3.0 + +### 4.1 New message types (`meshbay_common/protocol.py`) + +| Type | Direction | Carries | +|---|---|---| +| `transfer_open` | client → node | `tr`, `kind`, `bytes`, `chunks`, `from_chunk` | +| `transfer_close` | client → node | `tr`, `reason` | +| `transfer_state` | node → client | `tr`, `state`, `ahead`, `used`, `cap`, `scope` | + +`transfer_state` is the only reply, for granted, queued, revoked and closed +alike. One message type with a state field, rather than four types, because a +client that must switch on the type to find out it is still waiting is a client +that will get one branch wrong. + +`file_req` gains an optional `tr`. `file_upload` gains `tr` beside the +`upload_id` that is already there in clear. + +**Sealing.** `transfer_open` / `transfer_close` / `transfer_state` carry no +file names and no paths — `tr` is opaque, `bytes` and `chunks` are numbers — +so they stay in clear like `index_progress`, and for the same stated reason. +`label_hint` is **not** in the protocol: the client already knows what it named +the transfer, and putting a filename on the wire in clear to make a log line +prettier is exactly the trade `groupbox.py` exists to refuse. + +### 4.2 Why this is MNP 3.0 and not an additive 2.x + +The messages are additive; **the requirement is not**. A 2.0 client downloading +a 4 GB film sends no `tr`, is treated as a leaseless read, and is refused as +soon as it opens a third file (§3.4.1) — or, worse, is *not* refused and +transfers outside every cap. An opt-in switch — "enforce leases only for +clients that speak 3.0" — leaves a leaseless branch reachable on every node, +which is finding C6's lesson (`TCP accepted a bare JWT`) one feature later, and +it was already refused once for chat encryption on 2026-09-07. + +So: **`MNP_VERSION = "3.0"`, `MNP_MIN_SUPPORTED = "3.0"`**, and a 2.x peer is +refused at the handshake with `version_too_old` rather than admitted and then +mysteriously unable to download. Same play, same reasoning, and +`handshake.py`'s version range is the mechanism that already exists to pay for +it. + +**What that costs, stated plainly.** The SPA deploys with the hub, so browsers +get the new client. The **desktop client ships its own UI**, so an un-updated +app is locked out until its user updates — the version-skew consequence draft +v6 flagged under "Shipping the UI in a package". Before this ships, +`GET /v1/hub/version` needs its minimum-client-version field populated and the +client needs to show a real message ("this version can no longer connect, +update here") rather than a handshake refusal. + +### 4.3 Handshake ack + +Two fields inside the sealed configuration block (`_complete_handshake`, +`webrtc_server.py:852`), so the interface can draw correct numbers before +anything is transferred: + +```python +"transfer_limits": {"downloads": 2, "uploads": 2}, # this member, this group +``` + +Absent reads as "no limit known" and the client simply does not draw the +"1 of 2" hint — never as "unlimited", which would have the interface +contradicting the node. + +--- + +## 5. Nothing gets stuck + +The requirement the operator stated first, and the one worth over-engineering. +Two directions, and both must be closed: + +- **a slot the node never gets back** — the node fills up and everyone queues + forever; +- **a transfer the client shows as waiting when the node has forgotten it** — + one person's widget lies while the node is idle. + +### 5.1 Every way a slot can be lost + +| How it ends | What reclaims the slot | When | +|---|---|---| +| Transfer completes | Client sends `transfer_close{done}` | Immediately | +| …and the client forgets to | **Node auto-closes on serving the last chunk** — it knows `entry.size` and `CHUNK_SIZE`, so it knows the final index | Immediately | +| User cancels | `transfer_close{cancelled}` | Immediately | +| User pauses | `transfer_close{paused}` | Immediately | +| Tab closed, browser quit, app killed | `shutdown_tasks()` / `_unregister_peer()` releases every lease of the session | On WebRTC `connectionstatechange` — the same hook that already ends streams | +| Network drops, no clean close | Idle reclaim: no `file_req` under this `tr` for 120 s | ≤ 120 s | +| Client granted a slot and never used it | Acceptance deadline | 30 s | +| …and it *is* using it, but nothing said so | **`touch()` on every `file_req` carrying the lease.** Missing from the first build: the pool had the method, the tests covered it, and no caller existed — so `used` stayed False for every download and the acceptance deadline revoked each grant at 30 s while the file transferred at 20 MB/s | — | +| A grant revoked, requeued, granted again, revoked again | **Bounded at three misses, then the lease is closed.** Also missing at first: the requeue was a permanent cycle and the node logged the same reclaim every 30 s until it restarted | — | +| Client crashes mid-zip between two files | Idle reclaim (the gap between files is milliseconds) | ≤ 120 s | +| Node restarts | Leases are in memory only | Immediately | +| Group detached / member revoked mid-transfer | The existing revocation path drops the sessions, which releases their leases | Immediately | + +The first two rows are the answer to *"a user must not stay blocked once their +downloads have been done"*: the client says so, **and** the node concludes it +independently. Neither is trusted alone. + +### 5.2 Every way a client can be left waiting + +| Failure | What fixes it | +|---|---| +| Node granted a slot, the push was lost | Client watchdog: no state change for 60 s while `queued` → re-send `transfer_open` (idempotent on `tr`) | +| Reconnect: session gone, leases gone | `_onReconnected` (already in `transport.js`) re-opens the lease for every running/queued transfer, with `from_chunk` | +| Node forgot the lease (idle reclaim during a stall) | Node pushes `transfer_state{state:"revoked"}`; the widget shows **interrupted — resume**, not a silent hang | +| Node at cap and everyone is idle | Cannot happen if §5.1 holds; if it does, `GET /api/transfers` shows the leases and the operator can force-release one. Ship that endpoint | +| Queue entry for a member who left the group | Pump skips entries whose session is closed and drops them | + +### 5.3 The invariant, and how it is checked + +> **A new grant is never made past a cap, and every queue entry names a lease +> that exists. A lease is created in exactly one function and destroyed in +> exactly one function.** +> +> Not "granted leases never exceed the cap" — that was the first wording and the +> property test rejected it within a second of being written. Lowering a cap +> never interrupts a running transfer, so the count legitimately sits above the +> new value until those finish. The invariant is about what may be *handed out*, +> not about what is held. + +Two tests, not one: + +- a unit test per row of both tables above; +- a **randomised property test** — a few thousand random sequences of open, + close, drop, reconnect, pause, resume and cap changes, asserting after each + step that the counter equals the number of live leases and that no queue + entry references a dead session. The stuck-slot bug is a race by nature, and + "it works now" is not evidence against a race (`window_leak.mjs` is the + precedent in this repo). + +--- + +## 6. Pause and resume + +Cancelling already works client-side (`transfers.cancel`) and needs only to +send `transfer_close`. Pausing is new, and **whether it can be resumed depends +entirely on where the file is being written**. The interface must offer only +what the target can actually do — a pause button that quietly restarts the +download from zero is worse than no pause button. + +### 6.1 By download target + +| Target | Platform | Pause in-session | Resume after reload | Notes | +|---|---|---|---|---| +| File System Access (`showSaveFilePicker` / granted folder) | Chrome, Edge | Yes — keep the `writable` open, stop asking for chunks | **Yes** — reopen with `createWritable({keepExistingData:true})` and `write({type:"write", position})` | The full story. `downloads.js` already keeps the directory handle in IndexedDB | +| Electron native sink | Desktop app | Yes | **Yes**, once `main.js` gains a `save:resume` opening the file with `flags:'a'` and returning its current size | ~30 lines in `main.js` + `preload.js` (`main.js:797`, `preload.js:150`) | +| Service-worker stream | Firefox, Safari | **No** | No | Not on this target — see §6.5, which says what would be needed to change that. **Hide the pause button**, keep cancel | +| Blob in memory | Fallback | Yes (it is just an array) | No | Bounded by `BLOB_LIMIT` anyway | +| **Upload** (any platform) | All | **Yes** | **Yes**, via the sealed probe | A `File` is seekable and the *node* keeps the position, so there is no local target to consult. Missed entirely when 7a shipped — pause was built around the download path and uploads were refused it by the same guard that protects a transfer which cannot re-acquire its slot | + +**Corrected while implementing 7a.** `platform.capabilities` is the wrong home +for this: resumability is a property of the *target*, not of the platform. The +same Chrome yields a pausable target from a granted folder and an unpausable one +from the service worker, on the same page, for two files in the same batch. So +each target declares `pausable` itself, `prepare` carries it into the store, and +the widget renders from that record. The precedent to +follow — and the mistake not to repeat — is `_openDownloadTarget`'s silent +fallback chain (`CLAUDE.md`: *"a fallback chain reaches its floor silently"*). +Whatever the target ends up being, the transfer records which tier it got, and +the widget renders from that record. + +### 6.2 Pause releases the slot + +Stated again because it is a design decision, not an implementation detail: a +paused transfer holds nothing. Resuming rejoins the queue at the tail, and the +widget says so (*"resuming — 2 ahead"*). Anything else lets one member close +the node by pausing. + +### 6.3 Resuming the transfer itself + +Trivially available, because `pipelinedDownload` is already indexed by chunk: +resume is `nextRecv = Math.floor(bytesWritten / CHUNK_SIZE)` with the window +refired from there. Two constraints: + +- **`bytesWritten` must be a multiple of `CHUNK_SIZE`.** It is, because writes + are sequential and whole chunks — but a resumed file whose size is not a + chunk multiple (a partial write interrupted by a crash) must be **truncated + down** to the last whole chunk before resuming, never appended to. A silently + corrupted download is worse than a failed one. +- **The file must not have changed.** The entry id is its blake3 + (`GroupIndex`), so the resume record stores `{fileId, groupId, size, + bytesWritten, targetRef}`. A resume whose `fileId` is no longer in the index + fails with "this file has changed on the node", which is the truth. + +### 6.4 Resuming an upload + +The node already has the state (`self._uploads`, `.part` on disk). Make it +usable and stop it leaking: + +- **Corrected while implementing.** `transfer_open` travels in clear, and the + node identifies an upload by (member, directory, filename) — so asking there + would put the operator's filenames on an unsealed message, which is precisely + what sealing this path bought in MNP 2.0. The question is asked inside the + seal instead, as an ordinary `file_upload` with no bytes and + `UPLOAD_PROBE_INDEX` (-1); the node answers `resume_from` in the sealed ack, + writing nothing and reserving nothing. A node that predates it refuses the + index, which reads as "start from the beginning". +- Move `self._uploads` from the session to the **group context**, keyed by + `(user_id, rel_dir, filename)`, so a reconnect finds it. It is already + authorised per member; the session was never the right owner. +- **Reap orphans.** A `.part` older than 24 h with no live lease is deleted, on + a timer and at startup. This is a pre-existing leak (§1) and this work is + where it gets fixed. +- The no-overwrite rule, the filename allowlist, the size cap and the chunk + ordering are **unchanged** — draft v6 §2.1 names those four as what makes an + upload safe, and none of them is touched here. + +--- + +### 6.5 Firefox and Safari: OPFS, and what a pause would cost + +> **Corrected three times on 2026-09-08, ending here.** This section first said pause +> was impossible on Firefox; then that OPFS was the missing streaming target. +> **Both were wrong, and measurement is what settled it.** Firefox 154's OPFS +> quota is exactly 10% of the volume's size — ~12.8 GB on a 128 GB disk, hit +> *mid-download* — so OPFS cannot carry a film. The unbounded path is the +> **service worker**, which was already there and merely unreliable; it was +> fixed on 2026-09-08 (see `memory-audit-large-files.md` §6.5 and +> `test_streamed_download_reliability.py`) — and then fixed again the same +> evening, because "reliable" turned out not to include *staying alive*: an idle +> service worker is terminated after about thirty seconds and a streaming +> response does not count as activity, so every download longer than that lost +> its reader mid-file (§12.2). What survives below is only the pause question, +> for which OPFS is still the candidate, under its quota. + +Deferred by decision, so this section records **why**, because "Firefox cannot +pause" is not true as stated and would be a bad thing to leave in a plan. + +**Why the current target cannot pause.** On the service-worker path +(`downloads.js:openStreamedDownload`) the browser is already writing an HTTP +response to its own download folder. Three separate things break, and only the +first is about our code: + +- pausing means not writing to the stream, which leaves the browser's download + stalled in its own download manager — visible to the user, outside our + control, and cancellable by them in a way we never hear about; +- **a service worker is terminated when idle.** Firefox and Chromium both kill + one after tens of seconds with nothing to do. A pause that outlives that — + which any pause waiting on a queue does — takes the stream down with it; +- resume after a reload is genuinely impossible on this target. The response is + committed; no API reopens a browser-managed download to append to it. + +**The alternative that does exist: OPFS.** The Origin Private File System +(`navigator.storage.getDirectory()`) is a real, seekable, persistent +filesystem, private to the origin, and it **is** implemented in Firefox and +Safari — which is exactly what those two lack when they lack File System +Access. A download would be written there, resumable and reload-proof like any +local file, and handed to the person at the end: `handle.getFile()` returns a +`File` backed by disk, so `URL.createObjectURL` on it is a link the browser +copies disk-to-disk without loading gigabytes into the tab. + +So it is not a browser limitation. It is a **second download target to write +and maintain**, and it costs: + +- a fourth tier in `_openDownloadTarget`'s already four-deep chain — the + function whose silent floor cost this project a gigabyte of RAM per film + once already (`CLAUDE.md`); +- **quota.** OPFS is subject to storage quota and to eviction unless + `navigator.storage.persist()` is granted. A 4 GB film may simply not fit, and + finding that out at 90% is worse than not offering the path; +- **double disk usage and a real wait at the end.** The file exists twice while + the browser copies it out, and the copy is not instant for a large file; +- **cleanup.** Abandoned partial downloads sit in OPFS invisibly, consuming the + origin's quota, until something reaps them. That is a new janitor with its + own failure mode; +- **an API matrix that must be measured, not read.** The OPFS write path is not + uniform: `createSyncAccessHandle()` (workers only) and `createWritable()` have + had different availability across Firefox and Safari versions. This repo's + standing rule applies — *model the environment, never the code under test* — + so the answer comes from a probe in real browsers, not from a compatibility + table. + +**Recommendation.** Not in this work. Revisit it as its own change once transfer +slots are in production, and if it is picked up, it is worth more than pause +alone: it would give Firefox and Safari resumable downloads, reload-proof, and +retire the service-worker path and its iframe entirely. Ship it as a target +first and let pause fall out of it — the reverse order is how a fourth tier +gets bolted onto the chain in a hurry. + +## 7. Settings + +### 7.1 Node-wide — draft v6 §2.11's pattern, exactly + +Two new `[node]` keys, `max_concurrent_downloads` and +`max_concurrent_uploads`, default 8, positive int, `_positive()`-validated like +`max_concurrent_streams` (`config.py:332`). Then, mechanically: + +- `roster.py`: `SETTING_MAX_DOWNLOADS` / `SETTING_MAX_UPLOADS`, added to + `node_settings()` (`roster.py:943`); +- `ops.py`: entries in `get_node_settings` / `set_node_settings` + (`ops.py:1290`, `ops.py:1309`), written to `roster.db` **and** `node.toml`; +- **hot-swap through one helper.** `ops.set_node_settings` currently pokes + `webrtc._stream_sem`, which does not exist (§1). Replace all three with + `webrtc.set_capacity(streams=…, downloads=…, uploads=…)` on + `WebRTCTransport`, which resizes the live pools and pumps the queues. One + function, on the object that owns the state, with a test that changes a cap + on a running transport and watches a queued transfer start; +- the Node page grows two number fields beside "max concurrent streams" + (`node-page.js:1042`) and the CLI grows + `meshbay-node transfers show|set` — plus `meshbay-node transfers list`, + which prints live leases and the queue from `GET /api/transfers`. + +### 7.2 Per group, per member — `apps_enabled`'s pattern, exactly + +`OP_TRANSFER_LIMITS` in `adminop.py`, subject `"d=2,u=2"` so what the operator +signs names the outcome. `_do_transfer_limits` + +`_admin_exec_transfer_limits` in `webrtc_server.py`, stored by +`ops.set_transfer_limits` in `roster.db` group settings, broadcast as +`transfer_limits_ack` to the group's peers, surfaced in the group Settings tab +as a section beside the scan settings. + +**Absent means the default (2), not unlimited.** Deliberately unlike +`member_upload`'s "absent means allowed": a group that predates the setting and +came back unlimited would leave the node-wide cap as the only control, which is +the situation this work exists to end. Enabled by default, breaking, no opt-out +switch — the same call as MNP 2.0's chat encryption, for the same reason. No +release note and no operator prose: the project is in development and every +node is a test node. + +Bounds: 1–32. Zero is not "unlimited" and is not accepted; a member who may not +transfer at all is a member the operator revokes. + +--- + +## 8. The client + +### 8.1 `TransferStore` — three new states, one changed contract + +``` + ┌──────────┐ slot granted ┌──────────┐ + start() ─────▶│ queued │────────────────▶│ running │ + └──────────┘ └──────────┘ + ▲ │ │ │ │ + resume() │ │ cancel() pause() │ │ │ done + │ ▼ ▼ │ ▼ + ┌──────────┐ ┌────────┐ ┌──────┐ + │ paused │◀──────────────│ paused │ │ done │ + └──────────┘ └────────┘ └──────┘ + │ │ + cancel│ failed / cancelled + ▼ + ┌───────────┐ + │ cancelled │ + └───────────┘ +``` + +Statuses become `queued | running | paused | done | cancelled | failed`, and +`run({signal, onProgress})` becomes +`run({signal, onProgress, lease, resumeFrom})`: + +- `lease` — an object the store hands the runner: `await lease.acquire()` + blocks until granted, `lease.release(reason)` in a `finally`. The runner + never speaks to the transport about slots directly. One place opens a lease, + one place closes it — the same rule as the node's. +- **The slot is asked for after there is somewhere to write, and that ordering + is load-bearing.** Asking first reads better — the widget could draw a row + while the target is being chosen — and is wrong: opening a target takes tens + of seconds of streamed-download timeouts, or as long as somebody leaves a Save + As dialog open, and a grant not taken up in time is revoked. Tried the other + way during the build: three downloads started, one arrived. A test pins the + order now. +- The store gains `pause(id)`, `resume(id)`, `pauseAll()`, and a persisted + resume record for the tiers that support it. +- `_maybeRelease(transport)` must treat **paused and queued as busy** + (`transfers.js`'s `_busy`). A paused transfer whose transport was closed + because the group page went away can never resume — that is a one-line + regression waiting to happen, and it has a test. + +### 8.2 The widget + +`TransferWidget` (`app.js:141`) is a flat list of rows with a progress bar. It +becomes a small panel that can answer, at a glance, *what is happening and what +is my fault*: + +``` +┌─ Transfers ───────────────────── 2 running · 3 waiting ─┐ +│ ⏸ Pause all Clear finished │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ Some Saga (2019).mkv ⏸ ✕ │ +│ ███████████████████░░░░░░░░░░ 62% │ +│ 2.4 GB / 3.9 GB · 11.2 MB/s · 2 min left │ +├──────────────────────────────────────────────────────────┤ +│ ⬆ holiday-photos.zip ⏸ ✕ │ +│ ████████░░░░░░░░░░░░░░░░░░░░ 27% │ +│ 118 MB / 430 MB · 3.1 MB/s · 1 min left │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ A Different Show S02E04.mkv ✕ │ +│ ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ waiting · 1 ahead │ +│ Your 2 slots are busy │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ render.tar ▶ ✕ │ +│ ██████░░░░░░░░░░░░░░░░░░░░░░ paused at 21% │ +├──────────────────────────────────────────────────────────┤ +│ ⬇ notes.pdf Finished ↗ │ +└──────────────────────────────────────────────────────────┘ +``` + +Concretely: + +- **A summary in the header** — `2 running · 3 waiting` — and the nav badge + counts running *and* waiting, so a queued transfer is never invisible. +- **Waiting rows have a striped, indeterminate bar** (a CSS gradient animation, + no JS), visibly different from a stalled progress bar. `prefers-reduced-motion` + turns the animation off, not the state. +- **The reason for waiting is written out**, from the node's own counters: + *"Your 2 slots are busy"* vs *"The node is at capacity — 1 ahead"*. Someone + should be able to tell their own limit from the operator's without asking. +- **Pause / resume / cancel as icon buttons**, with real `aria-label`s and + `title`s; pause is **absent, not disabled**, where the target cannot resume + (§6.1) — and the row carries a quiet hint saying why, once. +- **ETA** from the existing windowed speed, hidden below 5 s of samples so it + does not flicker nonsense at the start. +- **Ordering**: running, then waiting (queue order), then paused, then + finished. A row does not jump when its neighbour finishes — group headers + rather than a re-sort on every emit. +- **Live region**: state transitions announce once (`aria-live="polite"`), + progress does not. +- Colours from the existing CSS variables, so light and dark both work without + a second palette. New rules go in `style.css` beside `.transfer-*`, measured + with `tests/harness/layout_probe.py` at 320 px — the transfers panel is + *precisely* what that harness was written for, after it hung 138 px off the + left of a phone. + +### 8.3 i18n + +New keys: `transfers.waiting`, `transfers.waiting_ahead`, `transfers.paused`, +`transfers.pause`, `transfers.resume`, `transfers.pause_all`, +`transfers.your_slots`, `transfers.node_busy`, `transfers.eta`, +`transfers.interrupted`, `transfers.no_pause_here`, `transfers.running_count`, +plus the settings labels. `en.js` is the source; `test_locales.py` holds the +other nine to its key set and will fail until all ten are translated. + +--- + +## 9. Regressions to watch + +Ranked by how quietly they would ship. + +1. **Browsing degraded by a busy node — the operator's stated requirement + (§3.4).** `_try_serve_thumbnail` shares `file_req` with real files, and the + Photos viewer and the Files preview modal fetch whole index entries through + `pipelinedDownload` just as a download does. Four tests, not one: with both + pools full and a queue behind them, (a) a Videos poster grid still fills, + (b) a Music album grid still fills, (c) a photo opens full size, (d) a PDF + preview opens. Each must also fail with the exemption removed. +2. **Video streaming charged twice.** `stream_req` must not touch the download + pool. Test: fill the download pool, start a film, assert it plays. +3. **Chat attachments — decided: they take an upload slot** like any other + upload, through the same `_do_file_upload` path. The regression is therefore + in the *interface*, not the rule: a paperclip whose file silently sits in a + queue is a chat that looks broken. `chat-app.js`'s composer must render the + same waiting state as the transfers panel — the attachment row says + "waiting, 1 ahead", not nothing — and a queued attachment must not block the + composer, which is the freeze already recorded in `CLAUDE.md` for chat + sends. Test it in `chat_send_probe.py`, which drives composer and transport + together for exactly this class of seam. +4. **`_maybeRelease` closing a transport with paused transfers on it** (§8.1). +5. **Zip downloads.** One lease for the whole job. Getting this wrong means a + 40-file zip takes 40 slots and deadlocks against its own cap — a deadlock, + not a slowdown, because the job cannot finish until it holds them all. +6. **Search across groups** (`search-page.js`, `getTransport`) opens transports + to several nodes; each node caps independently. Correct, but the widget must + not present another node's queue as this one's. +7. **`_free_name` and resumed uploads.** A resumed upload must reuse + `state["stored_name"]`, not draw a new one — otherwise a reconnect produces + `file (2).mkv` next to a half-written `file.mkv.part`. +8. **The 60 s cap in `_do_file_request`'s backpressure loop** interacts with a + paused reader: a paused transfer stops requesting, so nothing is waiting — + but check that a *slow* reader is not now reclaimed as idle. The idle timer + is reset on each `file_req` under the lease, not on each chunk *sent*. +9. **`test_security_regressions.py:786`** asserts `_transcode_sem` is in the + source. Renaming it during the `set_capacity` refactor will fail that test — + which is the test doing its job. Update it deliberately. + +10. **The one this list did not anticipate, and the one that happened:** none of + the eight defects the live pass found are in this list, because every entry + here is about *slots* and seven of the eight were about the download path + underneath them (§12.2). A regression list written from the change being + made will not cover the ground the change stands on. Before the flag day, + walk the download path itself — every write target, on every browser — + rather than the diff. + +--- + +## 10. Tests + +Following the repo's rule: measure the environment, never model the code under +test. + +**Node (`packages/meshbay-node/tests/`)** + +- `test_transfer_slots.py` — real `WebRTCPeerSession` objects against a fake + DataChannel (the shape `test_webrtc_transport.py` already uses): node cap, + per-member cap across two sessions of one account, queue order, head-of-line + skip, grant deadline, idle reclaim, auto-close on the last chunk, release on + `shutdown_tasks`, no double-charge on a re-`transfer_open` with the same `tr`. +- `test_transfer_invariant.py` — the randomised property test of §5.3. +- `test_transfer_exemptions.py` — the §3.4 requirement, as behaviour: posters, + covers, thumbnails, a full-size photo and a document preview all served with + both pools full and a queue waiting; the 2-in-flight bound refusing a third + concurrent leaseless entry; and a source-reading check that the only callers + opening a lease are the three that call `transfers.start()`. +- `test_navigation_not_leased.py` — `index_sync`, `index_delta`, + `media_meta_req`, `music_meta_req` and directory listing answered normally + with both pools full. Cheap, and it fails the day someone routes a listing + through `file_req`. +- `test_upload_resume.py` — reconnect mid-upload resumes at `next_index`; + orphan `.part` reaped; `stored_name` preserved. +- `test_node_settings.py` extension — `set_capacity` actually resizes a live + pool (the test `ops.py:1362` never had). + +**Hub / SPA (`packages/meshbay-hub/tests/`)** + +- `test_transfers.py` extension, under Node as today: queued → running, pause + releases, resume re-queues, cancel while queued, `_busy` counts paused. +- `tests/harness/transfers_probe.py` — mounts the real `TransferWidget` in + Chrome against a stubbed store and reads the rendered states back. The + precedent is `chat_send_probe.py`: the seam between store and widget is where + this will break, and neither source shows it. +- `test_layout_measured.py` extension — the panel at 320 px with a waiting row. +- `test_locales.py` — passes only when all ten catalogues have the new keys. + +**Live — `packages/meshbay-node/tests/transfer_probe.py`, in the repo** + +Not in `QE/` as this section first said: it found four defects nothing else +could, and `QE/` is not versioned, so it lived on one machine. It is not +collected by pytest (the filename does not match `test_*.py`) and still needs +`QE/deploy/e2e.py` and `demo.env` at run time, which it locates and explains +rather than importing blindly. + +`--want N` measures the cap, `--pull N [--parallel]` downloads real files to +completion on one connection, and `--operator` covers the two things only the +operator's CLI can answer: that a cap raised live starts what was waiting, and +that a vanished peer's slots are back before anyone asks. + +- ~~`QE/deploy/transfer_probe.py`~~ — opens N real MNP sessions as two accounts, + starts more transfers than the caps allow, and asserts the observed + concurrency and the order they complete in. The equivalent of + `stream_probe.py`, and the only thing that answers "is it the node or the + browser" in one run. +- A two-browser manual pass: cap 2, start 5, watch the queue drain; pause one, + reload the page, resume it; kill a tab mid-transfer and watch the slot come + back within 120 s. +- `QE/migration/reap_orphan_parts.py` — one-shot, node stopped, for the `.part` + files already on deployed nodes (§6.4). + +Per the QE rule in `CLAUDE.md`: any test against meshbay.org opens the UFW +port, tests, and closes the port and kills the processes **in the same block**. + +--- + +## 11. Order of work + +Each stage is shippable and testable on its own; nothing before stage 5 changes +what a member sees. + +| # | Stage | Contents | +|---|---|---| +| 1 | ✅ **Capacity, fixed** | `set_capacity()` on `WebRTCTransport`, replacing the dead `_stream_sem` poke. No new features. Ships the §1 bug fix alone, where it can be verified alone | +| 2 | ✅ **Leases, node-side** | Lease table, pools, queue, pump, every reclaim path in §5.1, `GET /api/transfers`, the DEBUG line. No client uses it yet; the node grants everything because no client asks | +| 3 | ✅ **Settings** | `[node]` keys, roster overrides, `OP_TRANSFER_LIMITS`, Node page fields, CLI verbs, ack field | +| 4 | ✅ **MNP 3.0** | Version bump on both sides, `MNP_MIN_SUPPORTED` at 3.0, **§3.4.1's bound on leaseless reads** (which did not exist), the desktop client's version gate, every package aligned on 0.13.0 | +| 5 | ✅ **Client leases** | `TransferStore` states and the lease contract, `file_req`/`file_upload` carrying `tr`, reconnect re-open, watchdog | +| 6 | ✅ **Widget** | The panel of §8.2, i18n, layout measured | +| 7a | ✅ **Pause / resume, in session** | Per-target `pausable`, the slot released on pause and re-asked on resume, `fromChunk` in the pipeline, the widget's pause button | +| 7b | ⬜ **Resume across a reload** | The persisted resume record, `save:resume` in Electron, `createWritable({keepExistingData:true})` with a position, truncate-to-chunk | +| 8a | ✅ **Upload state + reaping** | `uploads.py`, state in the group context keyed by member, the `.part` janitor | +| 8b | ✅ **Upload resume + pause** | The sealed probe chunk, `resume_from`, uploads asking for their own slot, `touch()` on an upload chunk | +| 9 | ✅ **Live pass** — done out of order, and §12 is its report. | `transfer_probe.py`, two browsers, the manual list | +| — | *Deferred, separate change* | **OPFS as a download target** for Firefox and Safari (§6.5) — resumable downloads there, and the retirement of the service-worker path. Not part of this work | + +Stages 1–3, 5 and 6 have landed. Stage 9 was run **before** stage 4 rather than +after, deliberately: while nothing is enforced, a defect in the machinery costs +nothing, and §12 is what that bought. Stage 4 is a flag day and should land only +on the evidence §12.3 describes. + +--- + +## 12. What the live pass found, and what it changes + +Step 9 was meant to confirm the machinery. It found **eight defects**, seven of +which no test in the repo could reach, and three of which are in code the +sections above describe as settled. Four more (§12.4 to §12.7) came out of +fixing those eight, and the last of them — a hard reload silently disabling the +only unbounded write path Firefox has — was the one actually being reported all +along. They are listed here because their pattern matters more than +any one of them: every one needed a real browser, a real node and a real hub to +show itself, and the last two were only reachable on Chrome. + +### 12.1 The three in the lease machinery + +| | Found by | +|---|---| +| **`touch()` was never called.** The node ignored `tr` on `file_req`, so `used` stayed False for every download and each grant was revoked at 30 s mid-transfer | the node's own log | +| **The requeue was a permanent cycle.** Revoked → queued → granted again → revoked, every 30 s, for as long as the daemon ran | the node's own log | +| **`transfers show` reported the module defaults**, so `transfers set 2 2` answered "applied now" and the next line said 0/8 | typing the command | + +The first is the one to learn from. `TransferSlots.touch()` existed, was +covered by its own test, and **had no caller**. The pool was right, the +handlers were right, and the call between them was missing — so neither side's +tests could see it, and the property test could not either: nothing drifted. +**A seam is not tested by testing both sides of it.** + +### 12.2 The five in the download path itself + +None of these are about slots, and all of them would have become "this download +is impossible" the moment leases were compulsory: + +- **three headers** decide whether the page may frame its own `/_mbdl/<id>` URL + — `frame-src`, `frame-ancestors`, `X-Frame-Options` — and all three were + wrong. Fixed one at a time over an afternoon; all three were visible in a + single `curl -I` against the deployed hub; +- **a service worker with no event for ~30 s is terminated**, and a streaming + response does not count as activity. The reader vanished mid-file and + `writable.write()` never resolved *and never rejected*: no error, no log, a + progress bar that stopped, and a node that stayed healthy throughout; +- **`encodeURIComponent` leaves `'` alone** and `'` is RFC 5987's delimiter, so + a 449 MB film arrived complete and correct under the name + `mtsshk9w-ohqty535`; +- **a browser grants one file picker per user gesture**, so downloading three + files at once failed two of them with a message about gestures; +- **two silent returns** meant a click with no connection produced nothing at + all — no transfer, no icon, no message. + +### 12.3 What this changes about step 4 + +Step 4 makes leases compulsory and refuses 2.x peers at the handshake. It is the +only irreversible step in this plan. + +Every defect in §12.1 and §12.2 was invisible to 1169 node and 814 hub tests, +and was found by a person clicking Download and pasting a console. Three of them +were introduced *by this work* on the day it was written. Making a mechanism +compulsory is a bet that it is well understood, and the evidence of one +afternoon is that it was not. + +**The recommendation was to wait for evidence rather than for time**: a week of +ordinary use with no freeze and no lost transfer, or the equivalent in deliberate +runs of `transfer_probe.py --pull 3 --parallel` and `--operator`. + +**Overruled by the operator on 2026-09-09, and the reasoning is better than the +recommendation it replaced.** The concern was that a flag day removes the +fallback exactly when defects are still being found; the answer is that the +fallback only has value while more work is coming down the same path, and none +is — the next work is a music application with playlists, which touches none of +this. The prerequisite this section named was satisfied first: +`GET /v1/hub/version` already carried `client.minimum`, and the desktop client +now reads it before connecting instead of meeting a handshake refusal it has no +vocabulary for. + +Nothing about §4's reasoning has changed. What changed is the confidence that +the thing being made compulsory works. + +### 12.4 The silent row — fixed + +**The transfers panel showed nothing while the target was being opened.** The +row was created after `_openDownloadTarget` returned, which can take tens of +seconds, so three clicks produced no icon at all and then several rows at once. + +It was "fixed" during the build by taking the lease first, which caused §12.1's +`not_taken_up` symptom and was reverted — on a wrong diagnosis, as it turns out: +the revocations were the missing `touch()`. The revert was right anyway (§8.1). + +The shape that worked is a `prepare` step in the store, distinct from `run`: the +row appears at the click, the target is opened, and the lease is asked for last. +The reservation about `showSaveFilePicker` needing a user gesture turned out to +be real, but not in the way expected — see §12.5. + +### 12.5 One dialog per file, and three downloads frozen behind it + +Reported from Chrome, after §12.4 shipped: selecting four files produced a Save +As dialog for the first, then — once that file had finished — a dialog for the +second, while the last two timed out. On a later attempt the three remaining +transfers simply froze. + +Three facts explain it, and only the third was a surprise: + +1. `for (const entry of selected) await downloadFile(entry)` used to serialise + the target openings **by accident**. Moving the opening into `prepare` + removed the accident and four pickers raced. Fixed by `_openTargetInTurn`, + an explicit queue on the targets — never on the rows, so every download + still appears at the click. +2. A browser grants one file picker per user gesture, and selecting four files + is one gesture. The code already recovered from the `SecurityError` Chrome + throws for a picker with no gesture behind it, by streaming instead. +3. **Chrome does not throw.** It shows the dialog anyway and waits for a human. + So the recovery in (2) was never reached, and the queue from (1) turned the + unanswered dialog into a head-of-line block: the third and fourth downloads + were not frozen, they were waiting correctly for a dialog nobody had been + shown yet. From the panel that is indistinguishable from a freeze. + +The fix is to stop asking. `_openTargetInTurn` marks everything that has to wait +its turn as `batched`, and a batched opening prefers the streamed path whatever +the download mode says. The first file of a batch — the one that actually holds +the gesture — still gets its dialog, so the preference is honoured where it can +be. For the rest there is no gesture left to spend, so nothing is lost by +streaming: the file still lands on disk, in the browser's own download folder, +written as it arrives. Only the choice of folder goes, and it was not on offer. + +If the worker does not answer, a batched download falls back to the dialog +rather than failing — asking is better than losing the file (§ "a preference +must not cost a capability", which applies to the fix as much as to the bug). + +Pinned by `test_targets_are_opened_one_at_a_time` (the queue, and that the first +opening is the only unbatched one) and by four cases in `test_memory_ceiling.py` +(the branch itself, both fallbacks, and that batching never pushes a large file +into memory). + +**What that left, and what was measured.** With the queue unblocked all four +files downloaded, but a dialog still appeared for each one — with MeshBay's own +setting on "save automatically", which should never prompt. That points at the +streamed path failing, so it was measured rather than reasoned about: a real +Chrome 152 driven over CDP against the deployed hub, running the actual flow +(a `TransformStream` posted to the worker, a hidden iframe on `/_mbdl/<id>`, a +wait for `mbdl-serving`). It is served in 2–3 ms on a normal load, after a hard +reload, and twice in the same document. Ctrl+F5 does leave the page +uncontrolled — `navigator.serviceWorker.controller` is null and no +`controllerchange` arrives — but the `mbdl-claim` recovery already in +`_claimController` gets control back inside 3 s. Both probes are in +`QE/`-style scratch scripts, not in the suite: they need the deployed hub. + +**And the queue itself became the next defect.** Serialising the openings was +new in the same commit, and on Firefox it regressed what had always worked: +four downloads that opened their targets at the same time began waiting on the +slowest, and all four sat at "preparing" — the node journal showing +`d=0/8(q0) u=0/8(q0)`, not one transfer opened, so the block was entirely in +the client before any slot was asked for. Measured on Firefox 154 against the +deployed hub: `register` and `ready` return instantly, the page is controlled, +and four serialised openings are served in 5–18 ms — so the streamed path was +not the delay either. The queue was. + +Two bounds fix it, and both are narrowings of the queue rather than of any +capability. Only an opening that could actually show a dialog joins the queue, +which on Firefox and Safari — no `showSaveFilePicker` at all — means none of +them do, restoring exactly the previous behaviour. And no opening waits behind +another for longer than `TARGET_QUEUE_BUDGET_MS` (90 s), because `_targetQueue` +is never reset and an opening that never settles would otherwise leave the page +unable to start a download again until it is reloaded. Releasing early is safe: +whatever is ahead is still the only unbatched opening, so the released one takes +the streamed path and opens no second dialog. + +The general shape, third time in this section: **a queue is a way to convert one +slow participant into several stuck ones.** Every queue added here needs to say +what it protects and refuse everything it does not protect, and needs a bound. + +### 12.6 The wait with no deadline + +Bounding the queue was not enough: Firefox still showed four rows at +"preparing", with the queue bypassed, so each opening was hanging on its own. + +`_claimController` had two waits with no deadline of any kind — +`navigator.serviceWorker.register()` and `navigator.serviceWorker.ready` — +while `SW_CONTROL_BUDGET_MS` bounded only the wait that comes *after* them. And +`_swPromise` is shared, so a single unsettled one of those left every download +on the page waiting on the same promise, for the life of the page. + +Measured on Firefox 154, on a local `127.0.0.1` site so no hub was involved: + +| worker | `register()` | `ready` | +| --- | --- | --- | +| that installs | 8 ms | 0 ms | +| **whose install handler rejects** | **7 ms** | **never settles** (still pending at 10 s) | + +That is the whole mechanism. `register()` resolves as soon as the registration +object exists — carrying nothing but an *installing* worker — and `ready` is +what waits for an active one. A worker that cannot install therefore produces a +registration that looks fine and a `ready` that never comes. + +Every wait in `_claimController` is now inside one budget, with two carve-outs +that exist so a deadline never costs a capability. A `ready` that times out +while `registration.active` is set is not fatal — `ready` may be waiting on a +newer worker that cannot install while an older one serves perfectly well. And +the `mbdl-claim` recovery keeps its own budget outside the deadline, because +giving up there would cost Firefox the only unbounded way it has to write a +download to disk. + +**A deadline alone would still not have been a fix**, only a better-explained +failure: a registration stuck with nothing but an installing worker does not +heal, and every later visit finds the same one. So when `ready` times out with +no active worker, the registration is discarded (`unregister()`) and asked for +once more with a fresh budget. The page repairs itself instead of needing +developer tools. + +Pinned by four cases in `test_streamed_download_reliability.py`, each checked +against the unfixed source: a worker that never installs, a registration that +never answers, a stuck `ready` that must not throw away a working worker, and +the discard-and-retry. + +So the streamed path is not what is prompting. Exactly three things can open a +dialog per file on Chrome in automatic mode: our `showSaveFilePicker`, the +folder permission bubble from `ensurePermission` (only when a folder was granted +and the permission lapsed, and it is not a Save As), and **Chrome's own +"always ask where to save each file"** in `chrome://settings/downloads` — which +applies to the worker's response because it carries `Content-Disposition: +attachment`, and which no download mode of ours can override. The `console.info` +added with this fix distinguishes them in one line: it is written before every +dialog we open, so a dialog with nothing in the console is the browser's. + +### 12.7 The one none of the above explained: a hard reload + +Every fix in §12.5 and §12.6 was real, and none of them was the defect being +reported. Downloads on Firefox failed with "the worker did not answer the +download within 15s", every time, for the operator, while the *same profile* +driven from this machine — headless and headed, on their display, with their +files, through the real interface — succeeded every time. + +The operator's own sequence found it, and it is worth writing down verbatim +because no automated run could have produced it: a freshly started browser +downloaded four files out of four, twice; one **Ctrl+F5** and every attempt +afterwards failed; restart, fine again; Ctrl+F5 before any attempt and the very +first one failed. + +**A document fetched by a hard reload is loaded with the service worker +bypassed.** It can still be claimed afterwards — so +`navigator.serviceWorker.controller` comes back and every check in +`_claimController` passes — but the navigations that document starts keep +missing the worker, and the hidden iframe a streamed download needs *is* a +navigation. On Firefox and Safari that is the only way to write a file too +large to hold in memory, so the download cannot happen at all, for the life of +that page. The same behaviour had already been measured on Chrome in §12.6's +work and its significance was missed. + +The hard reloads were on this author's instruction, after each deployment. The +SPA's HTML is served `no-store`, so an ordinary reload has always picked up a +new build and Ctrl+F5 was never needed for anything. + +**Why the measurements could not find it.** WebDriver cannot perform a hard +reload — the key event goes to the content, not the browser chrome — so every +automated run tested, over and over, the one case that works. A fix is validated +by tests; a diagnosis is not always validated by automation, and the operator's +manual sequence should have been asked for hours earlier. + +The remedy is to stop inferring servability from control and to ask instead. At +boot the client opens a four-byte stream and a hidden iframe, exactly as a real +download would, and tears both down. If the worker does not answer, the page +reloads **once**, ordinarily, which puts it back under the worker; the flag is +in `sessionStorage` because it has to survive the reload it triggers and must +stop rather than loop. Two delays then had to go, because the repair first +landed about thirty seconds late — long enough to click and watch four rows +hang: the claim is asked for *before* the control budget rather than after it +(a page uncontrolled beside an active worker is never claimed on its own), and a +download that starts while the self-test is running waits for it instead of +racing it. + +And the refusal message no longer tells a Firefox user to switch to Chrome for a +state an ordinary reload undoes. All ten catalogues say to reload first. + +--- + +## 13. Decisions — all settled (operator, 2026-09-08) + +Nothing in this plan is waiting on an answer. + +| # | Decision | Consequence | +|---|---|---| +| 0 | **Browsing is never subject to a slot** — posters, thumbnails, covers, previews, listings | §3.4, structural: a transfer is what the widget shows | +| 1 | **MNP 3.0, hard refusal of 2.x at the handshake** | No leaseless branch survives anywhere. Flag day, stage 4 | +| 2 | **Per-member default of 2 applies to existing groups** | No release note, no migration prose — the project is in development and every node is a test node | +| 3 | **Chat attachments take an upload slot like any other upload** | §9.3: the composer must show the waiting state, or the paperclip looks broken | +| 4 | **Pause on the service-worker path (Firefox, Safari) is deferred, not abandoned** | §6.5 says what it would actually take. It is cost, not impossibility | +| 5 | **The two big caps stay node-wide** | The resource is the machine's. A per-group variant can be added later with no protocol change | + |