MeshBay — Design
Status: the specification. This document states what MeshBay is, how it is built, and why each part has the shape it has. It replaces the draft series (
meshbay-draft-v5.md,meshbay-draft-v6.md,old-draft.md) and the design notes that grew around it; §16 maps every reference those documents and the code make onto a section here.The convention that is not negotiable: a claim in this document names the adversary it holds against. A property that holds against a passive hub and not an active one is written that way. A section that says "this buys nothing" is as load-bearing as one that says it buys something.
The second convention: this document states design, not history. Where a label like
C1,NS6orT3appears — and hundreds of code comments cite them — it names the invariant that holds today, not the incident that produced it. §13 is the register of those labels.Wire versions at the time of writing: MNP 3.4 (oldest peer accepted 3.0), MHP 0.1, packages 0.16.0. The normative source for the wire format is
MESHBAY_NODE_PROTOCOL.md; this document states the design the protocol serves, not its byte layout.
0. How to read this
0.1 Scope
| Read | For |
|---|---|
| this document | the architecture, the trust model, and the reason each decision is what it is |
QUICKSTART.md |
one machine to a working group, for somebody who has installed nothing |
USERGUIDE.md |
using a group and running a node, for the person who does either |
MESHBAY_NODE_PROTOCOL.md |
the MNP wire format, message by message |
transfers-v1.md |
the transfer system's failure-mode analysis, kept because a synthesis cannot carry "every way a slot can be lost" |
playlists.md |
the playlist design and its interface in full, with what building it corrected (§9.10) |
cast-smart-tv.md |
the DLNA/UPnP device backend — designed, not built (§11.4) |
WINDOWS-PORT.md |
the Windows port's audit and packaging detail (§11.2) |
PACKAGING-GUIDE.md, HTTPS.md, MAIL-SERVER.md, windows-build.md |
installation and server operations |
CLAUDE.md |
project conventions and the engineering lessons that govern how changes are made |
The user documentation is QUICKSTART.md and USERGUIDE.md (written
2026-09-18). The guide that preceded them described the system before per-node
identity, named roots and the sealed wire, and was retired rather than repaired
— a document a reader cannot tell the sound parts of is worse than none. The
replacements are written from this document and from the code, address a reader
who is not a developer, and cover Linux and the browser; Windows stays in
PACKAGING-GUIDE.md. They state what is not built as readily as what is
(§15.2, §15.3), which is what keeps them from drifting into the same condition
as the one they replace.
0.2 Reference labels
Code comments, tests and commit messages cite short labels — C1, H3, NS6,
T3, C5b, W2, E9, F1. Every one of them is defined in §13, stated as
the rule it names today.
Three label families collide, and the register keeps them apart:
C*/H*/M*/L*with no qualifier mean the second review's numbering (§13.3). That is the numbering the code uses. The first review'sC1/C2and the third review'sH1/H2/M1–M6/L1–L11are always written with the review named (§13.1, §13.4).M2a,M2b,M2candC5a,C5bare sub-items and are unique across the set.D1–D4are the client-architecture decisions (§14.2);D1–D12written as "Stage D*" are desktop-client build stages (§15.2).
0.3 The one-sentence version
The hub cannot read your content unless it ships you malicious client code — and against a native client it cannot do that undetectably.
Everything below either supports that sentence or states precisely where it stops.
1. What MeshBay is
1.1 Positioning
MeshBay is a platform for private, encrypted, self-hosted groups with an application store. A group is a set of people, a set of directories on somebody's machine, and a set of applications over them — chat, a file explorer, a video library, a music player, a photo album.
It is not a public file-sharing network. Public groups exist as an optional hub feature and are switched off on the reference deployment (§7.4). The design optimises for the private case throughout: the hub keeps no file names for private groups, registers no content hashes for them, and holds no key that opens them.
1.2 The three parties
┌─────────┐ accounts, group registry, signaling relay, ┌─────────┐
│ hub │ notifications, moderation, instance policy │ peer │
└────┬────┘ — never content, never a group key │ hubs │
│ └─────────┘
MHP 0.1 │ signalling (SDP/ICE, <1 KB), presence, revocation push MHP
│
┌────┴────┐ MNP 3.0 ┌──────────┐
│ node │◄──────── WebRTC DataChannel / QUIC ──────────►│ client │
└─────────┘ index, file chunks, streams, chat, admin └──────────┘
holds the files browser SPA or desktop
holds the group key client; both are the
is the content authority same UI source
- The hub is a registrar and a signaling relay. It is in the trusted path by choice, not by necessity (§14.2 D4), and it is never in the data path.
- The node is the daemon that holds a group's directories and its group key. It is the sole content authority: it decides who is served, what is served, and who may change anything about the group's content.
- The client is the browser SPA or the desktop application. Both are built from one source tree (§8.4).
1.3 The rule that settles arguments
Group-related server state lives on the node. Always.
Files, indexes, members' devices, pending device requests, invitations, chat, chat epoch keys, per-root availability, application settings, transfer limits — all on the node. The hub holds accounts, the group registry and membership, signaling, notifications, the moderation surface and instance policy, and nothing else about content.
This is decision E9, and it is the rule any new feature is measured against.
A feature that wants a row on the hub about a group's content is a feature that
has misunderstood the model. It has been re-verified at each content-model change:
SwarmSource carries a content hash, a node id and an endpoint — no paths, no
filenames — and private groups register nothing at all (H7).
2. Trust model
2.1 Adversaries
Every claim below is written against one of these, and they are the only ones the document uses:
| Adversary | What they can do |
|---|---|
| Passive hub | Read everything the hub legitimately stores and relays |
| Active hub | Also lie: forge tokens, invent accounts, substitute values it publishes, ship modified client code to a browser |
| Malicious node operator | Read and alter everything on their own machine, including the plaintext files they host |
| Malicious group member | Everything a member may do, plus anything the protocol fails to refuse |
| Network attacker | Observe and tamper with traffic between any two parties |
| Local attacker | Reach loopback services and files on a client or node machine |
| Registered hub user with no membership | Reach every hub endpoint that does not check membership |
| Federated peer hub | Push directory rows and revocations over MHP |
2.2 Security claims
| Claim | Passive hub | Active hub | Malicious node operator | Malicious member | Network attacker |
|---|---|---|---|---|---|
| Data never transits the hub | ✅ | ✅ | — | — | ✅ |
| File content is unreadable | ✅ | ❌ T3 (browser) · ✅ native | ❌ by design — the operator hosts the files | ❌ members share the group key | ✅ |
| The file index is unreadable | ✅ | ❌ T3 · ✅ native | ❌ | ❌ | ✅ |
| Chat content is unreadable | ✅ | ❌ T3 · ✅ native | ❌ — the operator is a member | ❌ | ✅ |
| Chat is unreadable off a stolen disk | ✅ | ✅ | ✅ without the keystore passphrase | ✅ | ✅ |
| Content cannot be modified | ✅ | ✅ | ❌ by design | ✅ | ✅ |
| The node cannot be impersonated | ✅ | ✅ | — | ✅ | ✅ |
| Client code integrity | ❌ T3, accepted (browser) · ✅ ships in the package (native) | ❌ T3 · ⚠️ native: detectable, not prevented | ✅ | ✅ | ✅ |
| The hub cannot obtain the group key | ✅ | ✅ except in an open-join group, where it can join legitimately (§7.3), and for an invitation the inviter asked the hub to mail, whose code it then holds (§3.4) | — | — | ✅ |
| Node content authority | ✅ | ✅ | ✅ sovereign | ✅ | ✅ |
| Devices cannot be added by the hub | ✅ | ✅ — the hub holds no user key and cannot countersign | ⚠️ a node adds a device only to itself, where it already reads everything | ✅ | ✅ |
| Chat senders are authenticated to each other | ✅ | ✅ | ⚠️ only for accounts the reader has already seen (§3.3) | ✅ | ✅ |
| Keypair bundles (C4) | closed for native devices | closed for native devices | ⚠️ open for any account that also signs in from a browser | — | — |
| Deleting your account erases you | ✅ hub-side | ✅ hub-side | ❌ files, pinned identity and bundle stay on the node (§7.7) | — | — |
| Your identity keys stay yours | ✅ | ✅ | ⚠️ offline attack on the bundle they hold — succeeds against a weak passphrase, and yields the identity used on that node only | ✅ | ✅ |
2.3 What the project must not claim
Three sentences are forbidden, each for a deliberate reason:
- "Everything is encrypted and unreadable by other parties, even the hub." A hub that ships the code can lift keys from the page regardless of protocol design (T3). That is an artifact-level attack, not a silent directory lie, and it is removed for native clients — not for browsers.
- "A native client makes the hub untrusted." It converts an undetectable, per-request, per-user attack into a persistent artifact that can be hashed and compared. That value is realised by reproducible builds and published hashes, not by the packaging format. A build signed with a key the hub operator holds relocates trust; it does not remove it.
- "C4 is closed." It is closed for a native device unconditionally, and open for any account that also uses a browser. An account is only as strong as its weakest client.
"End-to-end" here describes client ↔ node, never client ↔ client. Members and the operator read everything in their group; that is what a group is.
2.4 One boundary worth naming
An operator hosts your content by design. They should not be able to become you. They can still try — a keypair bundle sits on their disk and a weak passphrase gives it up — but what it gives up is the identity you use with them, which unlocks nothing they did not already hold. Reading what they host is by design; reading what other operators host is not, and does not follow (§3.2).
3. Identity
3.1 Accounts
An account lives on the hub: a username, an encrypted email address, a status and a role. A username is 8 to 64 characters, checked at registration only — accounts created under the older 3-character floor keep signing in. The passphrase never leaves the client. It derives two independent values, both salted by the trimmed username:
| Value | Derivation | Consumer |
|---|---|---|
auth_key |
PBKDF2-SHA512, 600 000 iterations, domain meshbay:auth:v1:<user> |
hub authentication — the hub stores an Argon2id hash of it |
bundle_key |
Argon2id 128 MB / t=3 / p=1, domain meshbay:bundle:v2:<user> |
AES-GCM key for the per-node identity bundle, held on each node |
This split is T1, and its consequence is structural: the hub never sees a passphrase, so the passphrase floor — 12 characters and roughly 60 estimated bits — can only be enforced client-side, and is.
The two values have different fates. The hub holds a verifier for auth_key and
can reset it from an email code. Nobody can reset bundle_key: the hub has
held no key material since the invite redesign, and cannot reach a
keypair_bundles row, which is served only over MNP to authenticated members.
That asymmetry is why passphrase change and passphrase recovery are two features
and not one (§3.6).
3.2 Identity keys are per node
A person's identity keypair is created at first contact with a node,
encrypted under bundle_key, and left on that node. It is never reused
elsewhere.
The reason is blast radius. The adversary is concrete: an operator holding their own node's disk, attacking a bundle offline at their leisure. What cracking one yields is the identity that person uses on that node — where the operator already holds the content, the index and every byte they serve. It is not a key anywhere else: each node gets its own, and a key one node pinned is a stranger to the next, which asks for a code like any first contact.
Two consequences fall out and both are wanted. Two operators cannot tell they
host the same person by comparing keys. And the hub stores and publishes no
user keys at all — users.pk_ed25519 / pk_x25519 are dropped, PUT /me/keys
does not exist, and /pubkeys returns an account id and the node's linking key.
There is no directory to substitute from, which is what H3 was.
Two further rules follow directly:
- Tokens carry no
pk_userclaim. A key chosen by the hub must never become the identity a node records for an upload, or whoever issues tokens decides who may delete a file. Attribution uses the roster pin. - There is no key rotation endpoint. Rotation is per node:
member unpinplus a fresh code, which already exists.
Registration therefore generates nothing, which has a pleasant side effect: a scripted signup produces a real account, and a wiped hub and node can be taken to a working demo without a browser.
3.3 Devices
One person may hold several devices on one node — a browser and a desktop client,
two laptops. identities is keyed by (user_id, pk_ed25519) with label,
added_at, added_by_pk, revoked_at and the evidence columns below. Pinning is
never INSERT OR REPLACE: a silent overwrite is a hole the moment a second key is
legitimate.
A new device is admitted by a key the node already pinned. The authority is therefore a key the node established locally, exactly as for operator authorisation — and the hub cannot produce it, because it holds no user keys. Device linking adds no hub-reachable authority.
The binding is a one-time code the new device generates and displays, hashed
together with the new keys: code_hash = sha256(code ‖ new_pk_ed25519 ‖
new_pk_x25519). The approving device asks the node for this account's pending
requests with their stored hashes and recomputes the hash for each until one
matches.
The code never reaches the node. That is what makes a substituted key impossible rather than merely detectable: a node offering fabricated keys would have to produce a hash over a code it has never seen.
The approval is deliberately not a human comparing digits. Safety numbers were evaluated and refused, permanently, under decision 19: no new code exchanges between people. A device-linking code is a code between a person's own devices, which is a different thing and was already accepted.
Bounds, all of them anti-abuse rather than the security boundary — the security comes from the hash binding:
| Bound | Value |
|---|---|
| Request TTL | 1 h, [node] device_request_ttl_minutes |
| Devices per account per node | 5 |
| Attempts per connection | 5, then a node-wide lockout |
| Filing a request | requires the account to have at least one pinned identity already |
member unpin <user> removes every device of an account, and is the only
form the CLI has (§6.7). Revoking one device is revoke_device over MNP,
driven from a device the node has already pinned, and it marks the row rather
than deleting it — a deleted row is a key the node would happily pin again.
Which device is on a connection is proved separately from which account. The
handshake proves the account; device_hello — signed over a transcript naming the
node, the group and this connection's nonce — proves the device, and is refused
unless the key is a live device of that account in the node's own roster. Without
it the node would fall back to the account's oldest key and record it as the
author of everything.
Members can verify each other's device keys (Tier 2). group_roster_req /
group_roster_resp, sealed under the group key, answers any member with, for
each live device of each active member: the key, the key that countersigned it,
and the signature, nonce and timestamp that prove it. The client walks the chain
itself — the node decides nothing, because the node is the party the property
holds against. A device the node lists but cannot evidence never enters the
verified set, so a fabricated key is not laundered in by being mentioned.
Three rules keep that honest:
- A root is a device that names no countersigner, not one that fails to produce a signature. Treating "no proof" as "root" would admit anything a node chose to write.
- First sight pins everything the node says, not the verified subset — an alarm that fires on legitimate second devices stops being read, and the budget for this whole feature is exactly one notice: "this account's key changed".
- A device pinned before the evidence columns existed is unevidenced and reads as such.
The property, stated exactly:
Once a member's client has seen an account, a node that later substitutes a key for it is detected. Nothing is gained at first sight, where the client has nothing to compare against.
That second sentence is not a caveat to be dropped. Closing first sight needs an attestation rooted outside the node — an operator-signed roster (Tier 3), which is deferred with nothing depending on it, and worth building only where the operator is not the machine.
The cost, which is real: the roster is member-visible, so every member of a group learns how many devices every other member holds and what their public keys are. It stays inside the group, the hub is not involved, and it is scoped to one group. A member who cannot see the keys cannot check them, so this is not avoidable.
Where device linking does not hold: an approval performed in a browser inherits T3 — the hub serves that browser its code and can read the typed code. The first browser-to-native link is the moment of highest exposure for an account, and it happens once. An account created natively does that first link in the safe direction.
3.4 Admission: invitations and pairing codes
The node wraps the group key, for a key the recipient proved possession of, over an authenticated channel, bound to an identity the operator admitted with a one-time code the hub never sees.
operator (SSH) meshbay-node member invite bob → CODE R3H8-TB6V
(or the same from the group's Settings tab, signed by the paired browser)
operator sends the code to bob out of band
bob opens the group; the client holds no group key
bob → node join_request {pk_ed25519, pk_x25519, code, sig} ← pre-proof window
node code valid for this account → pin the identity, admit to the group
node → bob the group key, wrapped for the X25519 key bob just proved he holds
Four properties, each load-bearing:
- No public key is ever fetched from a directory. The joiner's keys arrive
from the joiner, both signed together in one transcript (
meshbay:join:v1), so the identity key vouches for the encryption key. - The code binds a key to an account, and the hub never sees it — unless
the inviter ticks Send the invitation by e-mail (below). 40 bits,
Crockford base32 rendered
XXXX-XXXX, single use, valid for exactly one account in one group — except an invitation link's, which names its account when it is redeemed (below) — stored only assha256(code). A password KDF over 40 uniformly random bits would buy nothing. Guessing is bounded by 5 attempts per connection and a node-wide lockout, and every attempt is an audit event. - The node's roster is the authority, not hub membership. A hub that invents
an account, adds it to a group and mints it a token gets
not_authorized_for_group. - Wrapping happens on every connection. Nothing is stored per member, so key rotation propagates by itself and revocation actually takes effect. (Rotating the key after a revocation is still required — the ex-member holds the current one, and no protocol can take that back.)
- Revoking somebody cancels the code they have not redeemed yet, for that
group and no other. There is no membership until a code is consumed, so an
invitation sent to the wrong person is the whole of their access, and a
removal that left it usable would be a removal in name only. It is also the
case removal is asked for most: an invitation is undone before it is
accepted, not after. Nothing to rotate then — they never held the key, and
member_revokesays so by returning no reminder rather than by leaving the caller to work it out.
Node authority is established the same way, once per node: meshbay-node operator
pair prints a code, the operator types it into their own browser, and the node
pins that identity. It is never learned from the hub — a hub able to name the
operator's key could install itself as node administrator, which is NS4/M3.
Code lifetimes differ because the acts differ:
| Code | Default | Setting |
|---|---|---|
| Member invitation | 7 days | [node] invite_ttl_hours |
| Operator pairing | 24 h | [node] pair_ttl_hours |
| Device add request | 1 h | [node] device_request_ttl_minutes |
An invitation waits for someone to read their messages; a pairing code is typed during the SSH session that printed it.
Why a code and not something lighter — the question is what stops the hub from being bob on his first connection:
| Option | What an active hub can do | |
|---|---|---|
| Wrap for whatever key the peer presents | Forge a token for bob, present its own key, receive the key | worse than nothing |
| Bind to the key the inviter fetched from the hub | Substitute at invite time | H3, relocated |
| TOFU: first connection wins | Race the real bob with a forged token | small window, total consequence |
| Safety-number comparison | Nothing — but it needs two humans reading digits at the worst moment | correct, unusable as a default |
| One-time pairing code | Nothing: the code never reaches the hub | adopted |
Invitation links. A link lets somebody who has no account yet register and land in the group without typing a code. The code above cannot do that: it names an account when it is issued, and someone who has not registered has none. So a link carries two secrets with two jobs:
| Half | Secret | Checked by | Grants |
|---|---|---|---|
| Hub ticket | 128 random bits, stored as sha256 |
the hub, against the redeeming account's verified address — the blind index of the address the inviter typed | membership, i.e. the right to reach the node (§7.1). Nothing about keys |
| Node code | the ordinary 40-bit code, with kind = "link" |
the issuing node only | pinning and the group key, as for any code |
https://<hub>/#/invite?v=1&g=<group_id>&t=<ticket>&n=<node_pk, base64url>&c=<CODE>
Everything is after #, so no part of it is sent to the hub in a request; the page
reads it and rewrites the address to #/invite at once. g lets the client check
that the group the hub answers is the one the inviter meant; n names the node that
issued the code, which is the only node holding it.
At the node the code is therefore a bearer code, and everything else about it is
fixed: a member of one named group, once, never an operator, not spendable by
someone already an active member of that group, at most twenty outstanding per
group, cancellable by a handle unrelated to the code (MESHBAY_NODE_PROTOCOL.md
§8.6). kind is a column rather than an empty account id, so a row bound to nobody
can never be read as bound to anybody (AV1).
The rules the client follows, each of which a test holds:
- it sends the code only to the node the link names, and only once that node has signed its challenge with that key (§5.2). Without the signature the code went to whichever peer answered signaling, and in a group hosted twice, to whichever host answered first;
- it never sends the code to the hub (unless the inviter asked the hub to mail it —
below), and it keeps the pending invitation in the tab's
sessionStorageonly, cleared on success, refusal, expiry and sign-out; - joining is one explicit click after the invitation is shown — group, inviter — never automatic, or a link would add anyone to any group without asking;
- a link whose origin is not this hub is refused (the desktop client accepts one pasted in), so a ticket is never sent to another hub.
What it holds against, per the convention at the top. A third party holding the
link — a messaging service that previews it, a forwarded mail — gets nothing
without the invitee's mailbox. An active hub is not stopped: it verifies the
addresses it compares, and could already be anybody. A hijacked signaling path
no longer receives the code on a native client; a browser is still T3, exactly as
for a typed code. What it costs: the hub learns that an owner invited an address
(as a blind index and a masked form, until the link is used or expires). Addresses
are compared case-insensitively and nothing more, so +tags and dots must match.
Mailing the code is the inviter's choice, and it gives up the row above.
The Members tab can ask the hub to mail the code to the invitee's address on file
(invite-notify); that request carries the code, so an active hub holds it and
can join in the invitee's place. It is offered because a code that arrives on its
own is worth more to most groups than the property, and it is stated rather than
hidden: the box reads "Send the invitation by e-mail (may land in spam)", is
checked by default, and is remembered per account (the invite_email
preference), so an operator who unticks it once is not asked to again. Unticked,
the hub is never called and the table above holds exactly. The CLI mails nothing.
The same box sits under the link form, sharing the same preference. Ticked, the hub mails the link to the address the ticket is bound to — no second address, and the hub writes the link itself from its own origin, the ticket it has just minted, and the node key and code it is handed, each checked against its exact shape, so a hub-signed mail cannot be made to carry any other URL. Because that mail reaches an address the hub may have no relationship with, at the request of anyone who owns a group, it is also counted per sending account (AV29). A refused mail does not cancel the link, which is shown either way.
Delegation is designed and deferred. invite_create is authorised as a role
check against the roster rather than an equality test against the operator, and
the delegate role value is reserved, so a group admin who does not run the node
becomes a roster row and a CLI command — no protocol change, no migration.
3.5 Open-join groups
A group whose join_policy is open pins on first contact (TOFU) and wraps the
key immediately. A code there protects nothing — the hub can create an account,
join through the front door, and be a legitimate member — so it would be pure
friction.
Stated plainly, per the convention: in an open-join group the hub can obtain the group key. That is a property of open joining, not a defect of this design. Content in such a group is protected from the network and from non-members, and from nobody else.
Note the axis. visibility (public/private) controls discoverability and swarm
hash registration (H7). join_policy (open/request/invite) controls
admission. Only the second decides whether a code is required: a public group with
join_policy = "invite" keeps the code, because being findable is not being open.
join_policy is read from node.toml, never from the hub. A hub able to
declare a group open would be handed its key. An unknown group reads as invite.
3.6 Passphrase change and recovery
Changing a known passphrase re-wraps every reachable node's identity bundle
from the old bundle_key to the new one before touching the hub — if the
fan-out fails, the account is unchanged. Only then is POST /v1/users/password
called with the old and new auth_key. Nodes that were unreachable are named to
the user, with the operator fallback (member unpin plus a fresh code) as the way
to fix each one. Every refresh-token family is revoked.
Recovering a lost passphrase splits into what each key can reach:
| Recovered by | |
|---|---|
Hub login (auth_key) |
an email code alone |
| Per-node identity keys — group key unwrap, provable upload ownership, chat identity, device countersigning | the recovery key, per reachable node |
| An identity on a node with no recovery-wrapped copy, or offline at recovery time | operator member unpin plus a fresh code |
The recovery key is a full-entropy 32-byte secret R generated by the client,
rendered as a grouped mnemonic. recovery_key = HKDF-SHA256(R, info =
"meshbay:recovery:v1:" + username) — HKDF and not Argon2, because R has 256 bits
and there is nothing to brute-force. Every time an identity bundle is written to a
node, a second copy is written beside it wrapped under recovery_key
(bundle_enc_recovery, additive on the wire). R is a pass-through: offered in
the registration email by default, never written to any database, never logged.
The reset endpoints are built to leak nothing. POST /v1/users/password/reset-request
requires the username and the email on file as a pair, checked against a blind
index and never decrypted; a mismatch, an unknown username and a non-active
account all take the identical no-op path and return the same 200
{"status": "sent_if_exists"}. So it cannot be used to spray reset mail at an
inbox from a username alone.
A reset deletes every UserDevice row on the hub. Three different things are
called "device" here and only one is touched:
| What it is | A reset | |
|---|---|---|
user_devices (hub) |
an Ed25519 key that lets a client skip the passphrase prompt on launch. A hub-login convenience — no group key is wrapped for it | deleted |
per-node identity (identities on each node) |
the keys that unwrap the group key, prove upload ownership and sign chat — this is group access | recovered from the recovery copy, or via the operator fallback |
| the roster pin | which identities a node has admitted | untouched |
Deleting user_devices costs one passphrase prompt per client, which is the
point: after a "control may be lost" event, a laptop still carrying a stored
hub-auth key must stop signing in on its own.
3.7 The keypair bundle, and what it is worth (C4)
A bundle carries one node's identity keys, encrypted under the owner's passphrase, stored on that node. It is what lets a second browser open the same account there — the ordinary expectation, and the only mechanism available to a browser, which keeps nothing durable of its own.
Why Argon2id. PBKDF2 is compute-only, which is exactly what a GPU is good at. Measured: PBKDF2-SHA512 600k costs 241 ms per guess on one core, Argon2id 128 MB / t=3 costs 88 ms — the defender pays less — but only one of them forces an attacker to find 128 MB per guess.
The honest size of the gain. On a single high-end card the ceiling moves from roughly 8k guesses/s to roughly 2k: a factor of four, not a thousand. What it really buys is the cost of scale — 128 MB per lane caps a 24 GB card near 187 concurrent guesses and makes custom hardware unattractive, where SHA-512 silicon is cheap.
The passphrase decides this, not the KDF. At ~2k guesses/s a dictionary-and-rules run of 10⁹ candidates takes about six days on one card. Four random words (~52 bits) outlasts the sun. No parameter choice saves a weak passphrase; it only moves it from hours to days.
Operational facts that constrain changes:
- Argon2id runs in WebAssembly, vendored under
static/vendor/with its provenance. The CSP forbids external hosts and must keepwasm-unsafe-evalinscript-src. - Never change these parameters in one place.
keyderive.js,keyderive.py, the desktop client and the test harness are held byte-identical bytest_bundle_kdf_parity.py. A mismatch does not look like an error — it looks like an account nobody can open. - Bundles carry an
MBK2marker; an older PBKDF2 form is still readable and is re-encrypted on the next backup. - Cost is paid once per sign-in (≈650 ms bundle + ≈239 ms
auth_key). Reloading a page derives nothing: the key lives in IndexedDB. - The pre-proof window that serves bundles is bounded (4 fetches) and audited.
C4 is reduced, not closed. Bundles still sit on disks their owner does not
control. It closes for a native device unconditionally, because that device's key
is in no bundle anywhere. It closes for an account only when no browser needs a
bundle on that node — which needs device_policy {allow_bundle: false}, signed
by a pinned key so the decision is the user's and never the hub's (open item
O3).
4. Cryptography
4.1 Key hierarchy
User identity key Ed25519 signing, authentication — per node (§3.2)
User exchange key X25519 key agreement — per node
Group encryption key AEAD 256-bit content and index encryption — the group secret
Chat epoch key 32 bytes per group, per epoch — node-generated (§4.5)
Session keys X25519/HKDF per-connection, from DTLS/TLS
Every private key lives in an encrypted keystore on the machine that owns it. The hub never sees one.
Domain separation is consistent and mandatory. Every derivation uses a
distinct info string, and the AES variant adds an :aes suffix so two ciphers
can never derive the same key from one group key. This is a small detail that
prevents cross-protocol key reuse, and it is checked rather than assumed.
4.2 Group key wrapping (ECIES)
wrap: sk_eph, pk_eph = X25519.generate() # fresh per bundle
shared = X25519(sk_eph, pk_recipient)
wrap_key = HKDF(shared, salt=pk_eph, info="meshbay:gek_wrap:v1", len=32)
wrapped = AEAD(wrap_key).encrypt(nonce, gek, aad=pk_recipient)
bundle = pk_eph ‖ nonce ‖ wrapped
unwrap: shared = X25519(sk_recipient, pk_eph) # same derivation
Three properties are why this shape:
- The ephemeral keypair is fresh per bundle, so the same key to the same recipient produces different ciphertext every time.
- The AAD binds the bundle to its recipient, so a bundle reused for a different member is rejected by the tag rather than by a check somebody has to remember to write.
- A wrong private key fails at the AEAD tag — an immediate, unambiguous refusal.
The node produces every copy of the key itself, from its own CSPRNG. Nothing
arriving over MNP can activate a group key (C5b). Read that precisely: it
targets key material arriving from outside, not the instruction. An
operator-signed gek_rotate where the node generates the key is a different shape
and is allowed. The initial gek-init stays local, because with no key there is
no completed session to carry a signed op.
4.3 On-the-fly encryption
Files are stored in plaintext on the operator's disk and encrypted at read time. This avoids double storage and makes key rotation feasible without re-encrypting terabytes.
disk (plaintext) → compress → per-chunk AEAD under a group-derived key → transport → client
- Chunk size 1 MB: amortises AEAD overhead and enables seeking, because each chunk is independently decryptable.
chunk_key = HKDF(GEK, salt=None, info="file:" ‖ blake3(file) ‖ ":chunk:" ‖ index). The salt is omitted deliberately: the group key is CSPRNG output and already uniform, so the file and chunk context belongs ininfo, which is the correct HKDF usage (M5, first review).- Chunk authentication is the AEAD tag, not a per-chunk signature. The tag authenticates the ciphertext under a key only members hold, which is what the signature was for.
- Compression precedes encryption, because compression is ineffective on ciphertext.
- Upload chunk size is 48 KB, which is what fits the SCTP limit after msgpack overhead.
Crypto is not the bottleneck: encrypt-and-send costs single-digit milliseconds per megabyte against tens to hundreds for the network.
4.4 The group envelope
meshbay_common/groupbox.py, mirrored by sealGroup/openGroup in
static/crypto.js, is one envelope with purpose-separated subkeys derived
from the group key. AAD is "<msg_type>|<group_id>".
| Purpose | Info string | Carries |
|---|---|---|
index |
meshbay:index:v1 |
index_sync, index_delta |
ack |
meshbay:ack:v1 |
the handshake ack's configuration payload |
upload |
meshbay:upload:v1 |
file_upload / file_upload_ack |
chat_keys |
meshbay:chat_keys:v1 |
chat epoch key delivery |
roster |
meshbay:roster:v1 |
group_roster_req / resp |
Never reuse the chunk key with a pseudo-file for this. The purposes are what keep two message classes from sharing a key.
Three things about the sealed messages must stay straight:
- Sealing the ack line is integrity, not confidentiality. The handshake
transcript names no ack field, so
is_node_admin,enabled_apps, the roots table and the rest were authenticated by the channel alone. The AEAD tag comes from a key the hub does not hold. - Sealing the index is defence in depth against our own next bug, of a class that has shipped twice (C1, C6). It buys nothing against an observer, the hub or a member. That is the whole claim.
- A payload that does not open ends the session, never a default. An
unopenable
enabled_appsreads as "the operator disabled every app" and an unopenable index as "the group is empty" — both indistinguishable from legitimate states.
What stays in clear, and none of it is content: the handshake itself; type, v,
group_id and the ack's node_pk/proof/sig, because a receiver must route and
authenticate before it would trust a decryption; upload_id and chunk_index,
because the node routes and orders on them; index_progress, which is counters
only, every 2 s; transfer-lease fields, which are an opaque id and two numbers
(§5.5); and the media-metadata replies.
4.5 Chat encryption
Chat is encrypted with one key per group, per epoch, per device, derived by name from an epoch key the node generates and delivers wrapped under the group key:
epoch_key 32 random bytes, generated BY THE NODE (C5b)
device_key(d) = HKDF(epoch_key, info="meshbay:chat:dev:v1|"+group_id+"|"+d)
where d = base64(device pk_ed25519), the roster's own identifier
Every member derives every device's key from the epoch key, so nothing is distributed per device and nothing is stored per device.
There is no mutable sending state at all. That is the point, and it is a stronger guarantee than per-device chains rather than a weaker one: a shared chain advanced by two clients produces key and nonce reuse (first review C1, one level down), and a design with no sending state cannot have that hazard.
Two clients of one account normally share a device key. A second browser recovers the existing identity from the keypair bundle rather than minting a new one; device linking is the exception, not the rule. What makes that safe is the nonce, not the derivation: 96 random bits, never a counter. Two independent senders under one key collide only on the birthday bound, which at chat volume is unreachable; two senders advancing one counter collide immediately.
Each message is sealed and then signed over the ciphertext with the device key
the node pinned, so a receiver verifies before decrypting and anyone holding the
roster can verify. The AAD binds the group and the epoch, so a ciphertext cannot
be replayed into another group or attributed to another epoch. sender_name lives
inside the sealed payload — as a wire field it was free to spoof.
sender_id stays a clear field set by the node from the authenticated session
(NS6). It is what the store keys on and what the UI groups by; it is not what
authenticates the message. The device claim is checked against the connection's
own device_hello, or a member could sign as anyone. Replay is refused by a
unique (device, nonce) — a replay is a validly signed copy, so nothing about the
signature refuses it.
Epochs. A new epoch is opened when, and only when, the set of devices that may
read future messages shrinks: member revoke, member unpin, revoke_device,
gek_rotate, or an explicit chat rotate. Epoch 1 is opened at group load — a
group with no epoch is a group nobody can speak in.
Old epochs are kept and still delivered. That is what keeps history readable
to everyone who could already read it, and it is why rotating the group key is a
re-wrap rather than the destruction of the whole archive: the archive is not
encrypted under the group key, only the epoch keys are wrapped with it in transit.
Nothing anywhere deletes an epoch. Epoch keys are stored ECIES-wrapped to the
node's own X25519 key in bundles.db, never raw — a plaintext key store beside
chat.db would collapse the threat model silently, and it is the obvious thing to
write.
Why not a ratchet. Under group-key distribution and server-served history, the node must retain each chain's earliest key, and a chain key at iteration i yields every message key from i onward by pure HKDF. Forward secrecy is therefore zero either way. What a ratchet was left buying is a large amount of stateful client code with silent failure modes, three of which are concrete: any member could sign as any other, a second device dropped the first's chain, and the skipped-key cache grew without bound (§13.6, F1–F3). Forward secrecy is given up deliberately and on the record. If it ever becomes a real requirement it belongs in 1:1 DM, where there is no server-side history to contradict it.
Neither a sender-key nor a ratchet implementation exists in the tree. Both
were written, neither was ever called, and both are deleted — the reasoning that
ruled them out lives at the top of chatbox.py, the module that replaced them,
where it stands on its own instead of pointing at a file to compare against.
Kept code that nothing calls is worse than absent code. It reads as an alternative somebody may reach for, its green tests read as evidence of a protection that is not in the product, and it has to be maintained past every refactor to stay compiling — maintenance spent on a decision already made. If forward secrecy ever becomes a real requirement, it belongs in 1:1 DM, where there is no server-side history to contradict it, and it starts from the requirement rather than from a module somebody left behind.
What chat encryption protects against, in the words the user-facing docs should use: someone who obtains the node's storage without the keystore passphrase — a hosting provider imaging the machine, a leaked backup, a seizure where the passphrase is not surrendered. It does not protect chat from the operator or any current member (they hold the group key, and the chat key is delivered under it); from anyone holding any one device of any member; from a former member, for messages sent before the epoch changed; from the hub as regards metadata; or from the node as regards links posted, which it fetches to unfurl.
Deliberately not encrypted, stated so nobody reads more into the feature than
it does: sender_id, timestamps, message sizes and the fact of a message are in
the clear to the node, which is the relay and cannot route otherwise. The hub
learns per message the group, the time and the sender's account id, so it can skip
the author when creating notifications — it can build a social graph with timings
without reading a word, and that is a known metadata leak rather than a solved
problem. Attachments are ordinary files on a root and stay plaintext on disk; the
reference to one is inside the sealed payload, but the file and its name are in
the index.
4.6 Parameters
| Parameter | Value |
|---|---|
| Node keystore KDF | Argon2id 256 MB, t=3, lanes=4 — recorded per envelope, so raising it does not orphan existing keystores |
| Hub password verifier | Argon2id 64 MiB, t=3, lanes=4 (pw_version 4), over the client-derived auth_key — RFC 9106's second recommended setting. An older hash is verified at its own version's parameters and rewritten at the current ones on the next sign-in |
| Browser bundle key | Argon2id 128 MB, t=3, p=1 |
Browser auth_key |
PBKDF2-SHA512, 600 000 iterations |
Why the hub verifier is 64 MiB and not more. What it protects against is an
offline attacker holding the database; online guessing is bounded by the sign-in
lockout (§7.7), where Argon2's cost plays no part. That attacker pays, per guess,
the client's 600 000 PBKDF2-SHA512 iterations and the hub's Argon2id, because
auth_key is 256 bits and cannot be searched directly. Above 64 MiB the memory
multiplies the attacker's cost by a constant factor — at most 16 at 256 MB, less
once PBKDF2 is counted — which moves a weak passphrase from cracked-soon to
cracked-later and a strong one from out of reach to out of reach. On the hub the
same memory is paid at every sign-in, one derivation at a time (§13.5b, AV9):
measured on meshbay.org, 450 ms at 256 MB against 105 ms at 64 MiB. The
passphrase floor is what separates the two cases, not the verifier.
The stored pw_version always names the parameters the hash was made with.
Verification reads them from it, so hash_password takes the version it is
hashing for; the legacy raw-password scheme stays at version 2 and the
auth_key scheme is every version from 3 up.
| Chunk cipher | AEAD, 1 MB chunks, per-chunk key by HKDF |
| Chat nonce | 96 random bits per message, never a counter |
| Invite / pair codes | 40 bits, Crockford base32, single use, stored as sha256 |
5. The node protocol (MNP)
5.1 Transports
| Listener | Role | Status |
|---|---|---|
| WebRTC DataChannel | primary, browser and desktop client | the path everything is tested on |
| QUIC | LAN, port-forwarded, hub-less group:// |
off by default ([node] quic_enabled) — nothing ships a QUIC client yet, and it does not implement chat (below) |
| ~~TCP + TLS~~ | — | does not exist (C6) |
| ~~HTTP file API~~ | — | does not exist (C1) |
QUIC carries no chat, deliberately. It implements neither the per-device sealing nor the device identification the chat rules require, and a message reaching a group's archive without them would be a plaintext row in an encrypted history — indistinguishable from one somebody actually wrote. The handler was removed rather than gated: refusing there would mean maintaining a second, weaker set of rules for a transport with no client, which is how a bypass branch survives. An unimplemented message type is logged and dropped, like every other one this transport does not have.
That is the general form of the parity rule. A transport either implements a content rule fully or does not serve that content at all. "Implements the authentication but not the authorization" is the shape of finding C6 and of the third review's M2, twice.
ICE/STUN is the NAT traversal mechanism, for native clients too — via
aiortc in Python. punch_nat() is a direct-connection helper, not a traversal
stack: one UDP probe to one address, no STUN client, no candidate gathering, no
dual-stack fallback, and it requires the client to already know its own external
address. It was validated on one ISP and one NAT type. ICE has been validated
across two ISPs, two browsers, IPv4 STUN and IPv6 direct, and 4G CGNAT
(§11.1) — no TURN relay is needed.
Several STUN servers, two providers deep, because a single server is a silent single point of failure that adds the full gathering timeout to every connection when it is slow:
stun:stun.l.google.com:19302
stun:stun1.l.google.com:19302
stun:stun.cloudflare.com:3478
Both sides carry the same defaults and gather independently; neither learns which server the other used. A STUN server learns the querier's public IP and NAT mapping — that is its purpose. No content, no credentials and no group metadata passes through it; adding one is trusting its operator to learn your NAT topology, nothing more.
The node's list is editable three ways (Node page, meshbay-node stun,
node.toml) and hot-swapped on save. The browser's list is hardcoded and not
configurable: the hub relays SDP, not ICE policy, and no mechanism exists — or
is wanted — for a node to push ICE configuration to a browser.
ICE interface filtering (ice_interfaces) is node-side only, and controls
which local addresses the node offers as candidates. Two modes: auto (excludes
virtual and VPN adapters by heuristic) and manual (an explicit whitelist). It
exists because VPN clients add virtual interfaces whose .local mDNS candidates a
remote node spends seconds failing to resolve. The browser's own gathering is
governed by the engine and is outside this application's control.
5.2 The handshake
One implementation, meshbay_common/handshake.py, called by both transports. Two
implementations of one security check is C6 waiting to happen.
client → node handshake {token, group_id, nonce_c, v, v_min}
node authorize_token() JWT · scope · denylist · group_id · membership · hosting
node → client handshake_challenge {nonce_s, node_pk, sig} sig: Ed25519 over the challenge (3.4)
── pre-proof window: bundle fetch, join ──
client → node handshake_response {proof}
node verify HMAC(GEK, client transcript)
node → client handshake_ack {proof, sig, node_pk, sealed configuration}
client verify HMAC(GEK, node transcript) + Ed25519(node_pk),
and that ack.node_pk is the key announced above
The transcript is length-prefixed, domain-separated and role-bound:
"meshbay:mnp:handshake:v1" ‖ len‖role ‖ len‖group_id ‖ len‖nonce_c ‖ len‖nonce_s ‖ len‖binding
Every field is length-prefixed so the concatenation is unambiguous, and the role is bound in so a client proof can never be replayed as a node proof (L4).
Since 3.4 the node signs its challenge, with the key it announces, over
"meshbay:mnp:challenge:v1" ‖ len‖group_id ‖ len‖nonce_c ‖ len‖nonce_s ‖ len‖binding.
A join is sent in the pre-proof window, before the ack that proves the node, so
without it an invitation code went to whichever peer answered signaling. What it
buys, per the convention at the top: a client that knows which node it means to
reach can refuse to send a code anywhere else — against a hijacked signaling path
and against a second host of the same group. It proves a key, not the right
one: it helps only a client that already knows which key to expect. A wrong
signature is refused; an absent one is an older node, discovered from its answer.
Channel binding is mandatory and an absent one is refused — never degraded to nonce-only, which would silently drop MitM detection:
| Transport | Anchor |
|---|---|
| WebRTC | both DTLS certificate fingerprints |
| QUIC | SHA-256 of the server certificate. On a resumed session, which carries no certificate, the anchor travels with the session ticket — sound, because the ticket derives from the handshake where the certificate was presented |
Authentication is mutual (C3). The node proves possession of the group key
over a client-chosen nonce and signs the transcript with its long-term key.
The client verifies both, refuses a bare ack, and TOFU-pins pk_node per node,
refusing a changed key outright with a deliberate reset path in Settings for a
legitimate rotation.
node_pkis announced in the challenge because joining needs it before the ack: a first-time member signs a transcript naming this node and has no group key to complete a handshake with. It is unverified at that point and is never a substitute for the ack.
The pre-proof window is three messages, and that is a bound to defend. Only
the two bundle fetches and join_request are served before the proof, because
each is something a caller needs in order to prove possession of the group key:
a returning browser has to recover its identity, and a new member has no key to
prove with. Nothing else qualifies. Device messages are authenticated-only —
a device request is countersigned later by a device already pinned, so requiring
the caller to finish its own handshake first costs nothing and keeps the surface
at three. The bundle fetches are additionally bounded (4 per connection) and
audited, because that window is the neighbourhood C4 and C5b came from.
Pinning is defence in depth, not the primary control. A substituted node already fails the key proof. Pinning covers the case where an attacker holds the group key — an ex-member, a leaked key — and swaps the node underneath, which the proof alone cannot distinguish from the genuine node.
Authorization rules:
group_idis mandatory. Omitting it once skipped the membership check and fell back to the node's first group (M1).scope == "user"is enforced by default; node-scoped daemon tokens are refused on the client path (M9, NS7).- The denylist is consulted for user,
jtiand group. - The node refuses connections when it holds no group key — there is no
gek_required: falsebypass (NS8).
Refusals carry a code, not only a sentence, because a client can act on a code.
not_a_member in particular is usually a token issued before the person was added
to the group — groups is baked in at sign-in and the hub pushes no updates — so
the client refreshes once and retries rather than telling someone who was invited a
minute ago that they are not a member.
5.3 Correlation and liveness
Every reply carries the request id it answers (req_id). Matching by arrival
order is a guess that fails silently and asymmetrically: the victim is never the
request that was answered wrongly, it is the unrelated one now waiting for a reply
already delivered elsewhere. The node stamps req_id on the reply from _send,
via a ContextVar so a handler's spawned work still answers under the right id, and
never on a broadcast, which answers nothing.
PING/PONG is liveness on an already-open channel and never a discovery
mechanism — opening a connection costs a full ICE/DTLS handshake, measured at
0.6–7 s.
5.4 Operator-signed operations
Destructive and privileged operations require an Ed25519 signature over a structured transcript, never a JWT. The hub controls JWT issuance and can therefore never establish node-level authority.
"meshbay:admin:v1" ‖ len‖op ‖ len‖node_pk ‖ len‖group_id ‖ len‖subject ‖ len‖nonce ‖ len‖ts
TTL 120 s. The client reconstructs the transcript from announced fields and refuses to sign if the operation or subject is not what the user asked for (H5) — a challenge of opaque random bytes signed blind is an unbound signing oracle. The transcript's subject names the outcome, not the operation: what the operator is shown before signing has to be what happens.
Verification is against roster.operator_pks(), rebuilt from node state, never
from anything in the response.
| Operation | Authority |
|---|---|
file_delete |
the operator, or any non-revoked device of the uploading account |
dir_delete |
the operator alone, and only on an empty directory |
invite_create |
the operator (or a delegate, when delegation ships) |
invite_link_create, invite_cancel |
the operator |
gek_rotate |
operator-signed; the node generates the key itself |
initial gek-init |
local admin API or CLI only |
root add/remove/update/eject/plug, apps_enabled, app directories, transfer limits |
operator-signed |
~~gek_bundle_store~~ |
the message does not exist. No member ever hands the node key material |
gek_bundle_store was deleted rather than gated. The operator's X25519 public key
is announced in the handshake ack, so any member could wrap a key of their choosing
for it; a path that does not exist cannot be mis-authorised (C5b).
Authorisation is against the account, not the key. With several devices per
person, verifying against the exact uploading key would refuse a person's desktop
the right to delete what their phone uploaded. uploader_pk stops being the
authorisation key and becomes the audit record of which device acted. This
remains roster-rooted, not token-rooted: a hub minting a token that claims to
be someone holds no key the node pinned for them, so the signature fails.
Ownership is recorded by the node, from the authenticated session, and it is
durable. The record is written when the last chunk lands and the file reaches
its final name; the entry is stamped from it when the indexer creates the entry,
which is a later moment — the entry does not exist while the upload is still
arriving, so an attribution written against the index at the end of the upload
matches nothing, silently, and leaves every uploaded file owned by nobody. It is
stored beside the hash cache rather than on the entry alone, because the index is
rebuilt from disk at every start and an owner the node forgets on restart is a
right quietly taken away. It is validated against a live stat(), so whatever
later occupies that path inherits nothing.
It is asserted by the node, not proved by the uploader. A member verifies nothing here: they are told who uploaded a file, by the node that served them the index — the same trust they already extend to every other field in it. The overwrite path where an attacker could become a file's recorded uploader (C5a) is closed by the no-overwrite rule (§6.4), not by a signature. An upload transcript the uploader signs — over node, group, root, path, content hash, account and timestamp, stored with the entry — would make it verifiable by any member instead. That is an open item (§15.3), not something the product has.
One implementation, several front doors. The meshbay_node/ops/ package holds
every operation, one module per domain, all re-exported by ops/__init__.py. The loopback API, the CLI and the signed MNP handlers all call these
functions; they take the daemon state, raise OpError, and know nothing about
HTTP. Two implementations of one operation with two authorization checks is
C1/C6 one size down.
5.5 Transfer leases
A download used to be invisible to the node: a client sent eight independent chunk requests and reassembled the answers, and nothing said a transfer had started or ended. There was nothing to count and therefore nothing to cap.
The unit is the lease — the node's record that a peer is transferring
something, held for the length of the transfer and released by name
(transfer_open / transfer_close / transfer_state). Six properties are
decisions:
tris drawn by the client. Re-opening after a reconnect with the sametris idempotent, so a reconnect cannot charge a member twice for one transfer.- A lease is scoped to the connection, never to the account. It dies with the session, which makes the primary reclaim deterministic.
- A lease covers a job, not a file. A directory zip is dozens of files and one lease.
- Nothing is persisted. A restart drops every session anyway; a lease that outlived the process would be a slot nothing can release.
- Leases are counted, not bytes. What a slot protects is concurrency — open file handles, disk seeks, the channel buffer each transfer keeps full.
- Per-member first, then node-wide. A member at their own cap queues behind their own transfers and never holds a node-wide slot a second member has none of. Reversed, whoever arrives first takes everything.
| Bound | Default | Why it exists |
|---|---|---|
| Node-wide concurrent transfers | 8 per kind | operator's machine. Downloads and uploads are separate pools (max_concurrent_downloads, max_concurrent_uploads, §6.8), because the two cost different things and one queue for both makes each cap mean nothing. Streaming is a third pool again (max_concurrent_streams): a member watching a film is not charged a download slot |
| Per account, per group | 2 | absent means this, not "unlimited" |
| Grant deadline | 30 s | a grant nobody takes up is a slot nobody can use |
| Idle timeout | 120 s | catches a peer that vanished without the connection noticing |
| Queued per member per kind | 32 | an unbounded queue is how a node runs out of memory politely |
| Missed grants before closing | 3 | without a bound the requeue is a permanent cycle |
Browsing a group is never subject to a transfer slot. Not the poster grid, not the album covers, not the video thumbnails, not the file list, not opening a photo or a document to look at it. A member must be able to browse a group that is at capacity exactly as they browse an idle one. That is a requirement, not a tuning parameter, and it is met structurally rather than by choosing a lucky threshold.
Navigation proper never touches this path at all: listings and metadata are their own message types, sealed under the group key, with no relationship to a chunk request. That half needs no rule — only a test that fails if someone later routes a listing through the chunk path.
The chunk path itself carries three genuinely different things, and they are distinguished structurally, by what the id resolves to and by which function asked:
| What | Resolves to | Rule |
|---|---|---|
| Thumbnails, posters, cover art, cached audio conversions | a media-cache id, not an index entry | Never leased, never counted, never queued. One chunk each, out of a bounded cache the node built itself |
| Looking at one file — a photo full size, a document, an image | a real index entry, fetched whole | Not leased, subject to the bound below |
| Downloading and uploading | a real index entry | Leased |
The rule stated as a sentence someone can check by reading:
A transfer is something the transfers panel shows. If it does not appear there, it does not take a slot. The two sets are the same three call sites, which is what makes this verifiable rather than a judgement call at each new one.
Which of the two a request is, is the node's answer and not the client's.
tr is drawn by the client, so it is a claim: the node resolves it against its
own record and serves the chunk as a leased transfer only when that record is a
granted lease of this connection. A lease it holds but has not granted is
refused with a code — a member reading while queued is the cap not applying — and
an id it has no record of is treated as leaseless and bounded below, because that
is also what a reconnect looks like from here, where the session's leases died
with the old connection and the client is re-opening them. Read as a bare
presence check, the field made every cap on this page decoration: any non-empty
string skipped the ceiling, and the queue held only the clients that chose to
wait.
Why the exemption is expressed in concurrency and not in size or bytes, which is the durable part of this decision:
- A size threshold does not separate the two. A raw photo out of a camera is 60–80 MB and is browsing; a 40 MB archive is a download. Any threshold letting the photo through lets the archive through too.
- A byte-rate budget does not either. It would have to be large enough for that same photo, at which point it is large enough to be a download channel.
- Concurrency is the thing being rationed, so concurrency is what the exemption is expressed in.
The bound is therefore a ceiling on how many distinct files one session may read leaselessly at once (12, with a 60-second idle expiry). Per session and not per member, because this is a ceiling on what one connection can do while claiming to be browsing, not a resource pool — a member with three tabs open is browsing in three tabs. Two rules keep it from becoming a bug: a file already being read is always admitted, whatever the count, because refusing a chunk halfway through a photo is worse than never having admitted it; and entries expire on idle, because a viewer closed mid-file stops asking and says nothing, and dead entries would eventually refuse every later preview.
A preview never shows "waiting", because a preview never queues. If the bound is somehow reached, the request is refused with a stated reason and the person tries again — it does not silently become a queued transfer in a panel they were not looking at.
What the residual is, stated plainly. A client that lies — labelling a bulk
download as a view — gets that bound's worth of files at a time instead of its
member cap. It is bounded, it is audited, and it is the same class of statement as
the cap itself: this is a fairness control among cooperating clients, in the
company of the stream cap. It is not a defence against a member determined to
saturate the operator's disk, and must never be described as one — that member is a
member, and the answer to them is member revoke.
The numbers are visible or the queue is unprovable. transfer_state carries
used, cap and ahead so a client can say "waiting — 2 of your 2 slots are
busy" rather than showing a bare spinner; the same counters reach the loopback API
for the CLI and the Node page, and a periodic debug line. When someone reports a
transfer stuck at "waiting", that line is the only thing that will say whether the
node ever had them in a queue at all.
The lease module is free of asyncio and of the transport: it decides, and the caller does the I/O. A queue that reveals itself only through a DataChannel is a queue nobody can prove things about.
5.6 Versioning and flag days
MNP and MHP version independently of the package version. Every wire message
carries v; both peers declare v and v_min on the handshake and refuse each
other with a code (version_too_old / version_too_new / version_unreadable).
A mismatch is a refusal, not a field that turns up missing. A stated refusal is a bug report; a feature that quietly does not work is a support case.
Additive changes are MINOR and cost nothing. A change to what a peer must be able to do is MAJOR even when the messages are additive — a peer that cannot ask for a transfer lease is either refused, or not refused and transferring outside every cap the operator set.
There is no compatibility switch, by policy. An opt-in flag leaves the old
branch reachable on every node, which is C6's lesson one feature later. Where
a break is required, MNP_MIN_SUPPORTED moves with MNP_VERSION and the
deployment is coordinated: the hub serves the SPA, so a browser picks up the new
client on reload; the desktop client ships its own UI, which is why
GET /v1/hub/version carries client.minimum and the client checks it before
connecting and says "this version can no longer connect" rather than showing a
handshake refusal nobody can act on.
Every requirement is true of every peer the client can reach. The floor moves
with each MAJOR, so check_version refuses at the handshake any peer that cannot
meet one: an upload is sealed or it is not sent; a transfer has a real lease or it
does not run; there is one app-directories op and no wrappers behind it. The client
records the version its peer declared, for diagnostics, and branches on none of
it.
A capability flag on a peer whose floor already guarantees the capability is a branch that can only ever take one path — until somebody lowers the floor, at which point it silently takes the other. A field kept "just in case" is how the branches come back.
The floor is not the current version, and MINOR additions are why. It is
MNP_MIN_SUPPORTED in handshake.py, it equals the last MAJOR, and 3.1, 3.2,
3.3 and 3.4 have all been added above it without moving it. So a peer can be reachable and
still not do something the current version can, and the client has to cope with
that — by reading the peer's own answer, never by comparing version numbers.
3.2's audio tracks are the worked example: the node lists them in stream_init,
the client draws its selector from that list, and a node that sends no list gets no
selector. 3.3's subtitles repeat it exactly, and add the case where the list is
narrower than the file: only the tracks the node can convert to WebVTT are named,
so what the menu offers is what will actually appear, and a track's ordinal is
therefore not its position in the list. That is not the branch the box above refuses. The branch it refuses is a
flag the client sets from a version it parsed; this is the node stating what it
has, in the same message the feature already needed, and it takes exactly one path
per peer because the peer said which.
Where a break leaves data behind, a migration runs with the node stopped, backs
the database up first and is idempotent. But a migration that has to be
remembered is a migration that does not happen, so anything that can be a
read-time fallback is one instead: the roster reads an older settings key when the
new one is unset and leaves it behind on the first write, and node.toml keeps
being read in its older spelling. Only a transformation nothing can infer — two
settings that disagree, where only the operator knows which they meant — is
allowed to need a step somebody has to run.
6. The node
6.1 Node authority
The node is the content authority. Its authority comes from its own roster, established locally by pairing, and from nowhere else — never from the hub, never from a config file, never auto-pinned from the keystore (NS4). A configuration that still names a legacy admin key is warned about at startup and never obeyed.
The division of trust: the hub certifies identity; the node authorises content operations. Hub membership lets someone reach a node; the node's roster decides whether it wraps anything for them.
6.2 Roots
A group's content is a set of named roots, each mapping to a local directory, forming one union virtual root:
/ (group virtual root)
├── Films/ → D:\Media\Films
├── Music/ → E:\Audio (external drive, removable)
└── Documents/ → C:\Users\me\Share
The name is the chosen directory's basename, derived once at add time and stored. Never recomputed from the path: renaming a folder on disk would otherwise silently re-identify a whole library and break every stored reference to it. Four rules make basename naming safe:
- A duplicate basename is refused, case-insensitively. Collisions are common in
practice (
D:\FilmsandE:\Films). Refusing is correct; an explicit alias is the escape hatch (open item O11). - No root may contain another, compared case-insensitively and after canonicalisation. Two nested roots would index the same bytes twice under two identities.
- The basename becomes a path segment every member sees, so it must itself pass the portability rules (§10) — a root Windows cannot write to is a root nobody on Windows can download from.
- Every index path carries a root segment, uniformly, including in a single-root deployment. One code path, not two.
Five consequences, none optional:
- The root name is part of a file's identity, so renaming a root rewrites every path under it. Renaming is explicit and warned, never a cosmetic setting.
- Availability is per root. With one directory an unplugged disk was a hazard; with named roots it is a supported state — one root freezes, the others carry on.
- Free space, quotas and capacity are per root — different volumes. Anything the UI says about space names which root it means.
- Path resolution is per root, in one place: a request names
<root>/<path>,RootSet.resolve()resolves it against that root's canonical path and refuses.., absolute segments, symlinks and anything escaping the root. kindis a view hint (generic/video/audio/photo), nothing more.
Each root is read-only or read-write.
writable = false(the default) means read-only for everyone, including the operator. A published library that quietly accepts writes from whoever holds admin authority is not one, so refusing the operator is the point rather than the defect.writable = truemeans any group member may upload there.
Several roots may be writable and none need be — a fully read-only group is valid. The operator toggles this with a signed op.
There is one answer to "may this member write", and it is the root. A single flag over the group cannot express "this library is published read-only and that folder is a drop box", which is the ordinary arrangement — so the group-wide switch that used to exist is gone entirely: the message, the signed operation, the field on the handshake ack, and the
uploadalias each root used to carry besidewritable. Two sources for one question is one too many: whichever the code consulted first decided it, and a client falling back to the group flag whenever a root omittedwritableis exactly that bug with a compatibility justification. Two names for one boolean is the same fault one size down.A control that writes must name where. With two writable roots the node cannot choose without guessing, and a guess sends a member's file to a disk the operator did not intend. The client names a root, never a path; everything below the root is decided by the node.
A root that goes away must freeze, not empty. The indexer runs a watchdog observer and rebuilds on change; unmounting a volume either emits deletions for the whole tree or presents an empty directory to the next scan. Both propagate as though the owner erased their library. So a root has two independent runtime states:
ejected— operator-controlled, persisted inroster.db.available— computed asnot ejected and is_live(). This is what clients and the indexer see.
The distinction matters: between clicking eject and physically unplugging,
is_live() is still true, and without ejected the availability sweep would
immediately flip the root back.
Eject stops that root's observer, marks it unavailable, freezes its entries and propagates the availability change to connected peers — the operator can then safely unplug. Plug checks the path is accessible first, then rescans: the plan called for a reconciliation, but a device people carry around can come back arbitrarily different, and the hash cache means unchanged files are not re-read.
ejected is persisted and restored at startup, because a restart is exactly
what an operator does after noticing a drive fell off, and an in-memory flag would
let the following scan read the empty mount point as an erased library. It lives in
roster.db and not in node.toml: it is runtime state, and an operator's
hand-written config must not be rewritten because a USB drive was unplugged.
Auto-eject is the safety net. If a removable root's path disappears, the
availability sweep sets ejected as though the operator had clicked it, and
reports it so the daemon persists it. Nothing is deleted: index entries, cached
metadata, thumbnails, chat history referencing those files and app directory
configurations all survive, the last flagged as temporarily invalid rather than
wrong.
6.3 Indexing
The index is content-addressed: GroupIndex is keyed by blake3, so the same
bytes at two paths inside one group are one entry. This is why a scan can
report ten files and index nine, and it decides how reconciliation must work:
Anything comparing disk against index must compare ids, not paths. Comparing paths makes the second path of a duplicated file look like a missed event, every sweep, forever — rewriting the entry, bumping the version and pushing an index update to every connected peer.
Hashing is partial above 40 MB. A hash exists for content identity, and a 4 GB file does not need 4 GB of I/O to be identified with overwhelming probability:
| Size | Method | hash_version |
|---|---|---|
| ≤ 40 MB | full read | 1 |
| > 40 MB | blake3 over the first 20 MB ‖ last 20 MB ‖ 5 MB at the midpoint | 2 |
Head and tail catch container headers, trailers and files that differ only at one end; the mid-sample catches files sharing a header and trailer. Below the threshold a partial read would sample the whole file anyway, so the full path is simpler and produces the same value — which is what keeps small files cross-comparable between nodes of different versions. A large file indexed by a node of each version produces different ids and does not merge in cross-group search; that resolves itself when both upgrade, and is the accepted cost of not reading 4 TB to build a library.
hash_version is an additive index field with a default, so an entry written
before it deserialises correctly and needs no protocol bump. The hash cache
carries the column and auto-migrates on open; large files are re-hashed lazily on
the first scan after an upgrade.
Periodic reconciliation is mandatory on every platform, not a backstop:
ReadDirectoryChangesW drops events under load on Windows, and inotify is
unreliable on a FUSE-mounted volume on Linux (§10).
Progress accounting covers the real-time path too. A file's size joins the total the moment its debounce timer is first scheduled — not on every re-trigger, or a cancelled-and-rescheduled timer double-counts — and joins the scanned total when its hash finishes. The scanning flag clears only when no timer is pending and no hash is running, because the in-flight hash of a large file is the entire reason to show progress.
index_progress is deliberately not sealed: counters only, every 2 s.
6.4 Uploads
Five protections, and they are the substance:
- a filename allowlist;
- no overwrite — a colliding name gets a free one. The check is
Path.exists(), andstat()is itself case-insensitive on NTFS and exFAT, so this already holds there; - strict chunk ordering;
- a size cap — 8 GB per file by default, and the operator's to set
(
max_upload_gbin node.toml, on the Node page, ormeshbay-node transfers max-size), because the disk that fills is theirs and a Pi and a film server are not the same machine. It is read from the transport context on every chunk, so a change applies to an upload already running. A file past the ceiling is refused at the chunk that crosses it and its.partis deleted. There is deliberately no aggregate quota yet, and that gap is named in §15.3 rather than left to be discovered: a member can still fill the operator's disk one capped file at a time; - the target root must be writable and available, enforced by the node.
There is no quarantine subdirectory. A folder appearing beside the operator's
library because somebody sent a file is the node deciding how their disk is
arranged. What made a quarantine worth having was never the subdirectory — it is
the four rules above, and they are unchanged. The client now names the destination
folder, which is safe for exactly one reason: it is resolved through
RootSet.resolve() (§6.2). A member answers "which of this group's folders",
never "which path on the operator's disk".
If the named root is unavailable the upload fails with a stated reason and never falls back to another; if the group has no writable root, uploads are refused rather than guessed.
The node enforces it; the interface merely stops offering it. Each root's
writable flag rides the handshake ack and the index payload, so a client knows
whether to draw the Upload button and the chat paperclip, and changes are broadcast
to everyone connected. None of that is the control: a member on an old tab, or one
speaking MNP directly, is refused by the node.
Files can be dropped onto the Files tab, files and folders alike, under the
Upload button's rule (a folder on screen, in a writable root, in a group — not in
Search). A drop is decided whole before anything is sent: a name already in the
folder refuses it, compared without case, because the node's no-overwrite rule
would otherwise store a colliding file under a free name nobody asked for and
refuse a colliding folder half-way through; and a name outside the filename
allowlist refuses it too, from a client copy of the rule that a test holds to the
node's answers. A folder is rebuilt one dir_create at a time, parents first, and
its files are fed to the transfer store a few at a time so the member's queue
(§5.5) never reaches its cap. None of this is a control — the node still enforces
every rule above; it is what keeps a drop from ending in a partial copy.
In a group with no writable root the interface says so plainly rather than picking one — a fallback that chooses whatever comes first only moves the failure to send time, where the person has already chosen the file.
Archives are not the node's business. A member downloading a folder as a zip fetches the same encrypted chunks as any other download and assembles the archive in the browser. The node serves no bundles, holds no temporary files, and cannot be asked to compress anything — one fewer place where a request turns into work on someone else's disk.
Directory creation is not privileged. Directory removal is, and is refused unless the directory is empty. The emptiness rule is the safety property: whatever the caller intended and whatever the client sent, the operation cannot destroy content. It is checked twice — before the challenge is issued and again after the signature returns — because a file can land during the round trip to the operator's browser.
6.5 Derived data and enrichment
The governing rule:
Enrichment happens on the client, from data it already has. What the client cannot compute, the node produces — and where the node produces it for everybody, it caches it in its own
data_dir, never in a shared root.
The client half is unchanged and is why a chat image thumbnail costs the node nothing: the browser already decrypted the image and scales it itself.
The node half was decided against the alternative and the reasoning is worth keeping, because it looks like a violation of "no derived state" and is not:
- A third-party API quota is per credential, not per device. A token shipped inside every install and called from every client scales with the number of devices in existence. A node making the calls on behalf of its own members makes the number of nodes the denominator, and one lookup per unique title serves every member indefinitely.
- Thin clients benefit from a node that does more. A phone should receive a small ready-made image and a JSON blob, not decode video or hold its own multi-gigabyte cache.
- Keeping the credential server-side is a strict improvement over shipping it
to every renderer, and it preserves the desktop client's tested invariant: it
issues no request outside
/v1/and the signaling socket.
The cache lives in the node's own data_dir — beside chat.db, audit.db,
bundles.db — and never inside a shared root. That is not a compromise, it is
strictly better, for reasons independent of sovereignty: a shared root is
routinely a read-only backup mount or a share the node cannot write to; a
dot-prefixed folder is not hidden on Windows and would appear as an ordinary
folder full of previews of a private group's content to anyone who plugs the drive
into another machine; and it would need filtering out of every listing path
consistently, forever.
Delivery reuses the chunk path. A thumbnail is addressed by its own blake3 exactly as a file is by its id, so the chunk handler resolves a requested id against either a real file or the media cache. Same transport, same group-derived encryption, same backpressure — no parallel mechanism, and no new authorization surface.
Cache lifecycle is tied to the index. A file's thumbnail and its third-party match are pruned by the same event that removes its index entry. Third-party metadata is cached per external id with its own refresh window, since several files of one show share one fetch.
A visible node-wide toggle turns third-party calls off entirely, independently of any application being enabled, for an operator who wants zero third-party network traffic. This is genuinely new node behaviour — egress to a third party, and a disk-resident cache with a real deletion obligation — and it is stated rather than left implicit.
Anything that shells out to a media tool obeys three rules, all of which this codebase has paid for:
- its own small bounded pool with a short timeout, never the streaming pool — a grid of fifty videos would otherwise exhaust every streaming slot on the node, since a stream slot is held for the length of a film;
- drain the pipes, then wait with a timeout, and release the slot regardless — a process that outruns a paced reader cannot finish closing while its stdout is full, SIGKILL or not;
- a background task must be held, or the loop may collect it mid-flight and the slot is lost for good.
Link previews are a further instance. A URL pasted in chat is unfurled by the
node: the browser cannot (a strict img-src/connect-src, and CORS), and a
direct fetch would leak every reader's IP to the linked host on each render. The
card text lives in a bounded in-memory TTL cache; the image rides the same
blake3-keyed store as any other thumbnail. The new surface is SSRF, because the
URL is a member's choice and it triggers an outbound request from the operator's
machine: http(s) only, no credentials, a port allowlist, every resolved address
must be globally routable, redirects followed by hand so each hop is re-checked,
the connect address re-checked against the checked one, a response-size guard, and
a per-member rate limit. The operator can switch previews off per group.
Every new outbound or cross-trust surface needs a bound and a named adversary in the same commit. That is the standing rule this section exists to enforce.
6.6 Chat storage
One SQLite database per group at data_dir/<group_id>/chat.db. Rows carry the
sealed ciphertext, the epoch, the sending device, the nonce and the signature, with
a unique (device, nonce) refusing replays.
Paging is backwards — get_recent / get_before / has_before — because a
chat opens at the newest page. A forwards pager is not what a chat opens with.
meshbay-node chat prune <days> deletes messages only, never an epoch key. An
epoch with no messages is harmless; an epoch key deleted while messages still need
it is an unreadable archive.
A message is bounded in size and in rate, like every other member-supplied
write. Sending one costs the operator a row that nothing expires, every other
connected member a relayed copy, and every member of the group a notification —
so the two bounds answer the two halves: 64 KB of ciphertext for what one
message may cost, and 60 a minute per account per group for how often one
member may impose it. Both are checked before anything is stored or relayed, and
a refusal names itself (chat_too_large, chat_rate_limited) and is audited.
The numbers are meant to be invisible. The sealed payload is the text, a thread id, a display name and a timestamp — an attachment is a file on a root and travels as a reference (§4.5) — so 64 KB is some sixty thousand characters, and sixty a minute is far above a person typing. The rate is keyed by account, not by connection: a second tab does not make anyone type faster, and keying on the session would hand a script one budget per socket it opens.
There is deliberately no node-wide chat ceiling beside the per-account one, and the contrast with link previews is the reason. A preview spends the node's egress and its third-party quota, which is one shared thing and deserves a shared bound; a chat message spends the sender's own group. A node-wide ceiling would let a busy group silence a quiet one — the same defect this bound closes, one level up.
6.7 Operator surface
Two personas need different tools, and the headless one is the normal deployment:
| Operator | Reaches the node via |
|---|---|
| Desktop | the desktop client's Node page |
| Headless / SSH | the CLI |
Every operation is reachable over SSH with no browser on the host, with one
exception named in §15.3 rather than left to be discovered: revoking a single
device of a member is an interface action, and the CLI's only lever is
member unpin, which removes all of that account's devices at once. status
deliberately reads the keystore and config directly so it works while the daemon is
stopped — the state an operator is most often in, since the daemon will not stay up
before its key is linked or a group exists.
meshbay-node status
meshbay-node group list|add|remove
meshbay-node root list|add|remove|set|eject|plug
meshbay-node gek init|rotate
meshbay-node operator pair
meshbay-node member list|invite [--link]|cancel|revoke|unpin
meshbay-node file list|rm
meshbay-node video rematch
meshbay-node chat status|rotate|encrypt-history|prune
meshbay-node transfers show|set|max-size|per-member
meshbay-node denylist show|clear
meshbay-node stun list|add|remove|reset
meshbay-node reload|restart-daemon
This block is a list of verbs that exist. It carried
member device list|revoke for as long as device linking has been built, and no
such verb was ever written — device falls through to the username resolver and
fails. A command list in a specification is read as a promise by whoever writes
the next document from it, which is exactly how it was found (§15.3).
member revoke/unpin resolve a username against the roster and refuse an
unknown one rather than acting on nobody — a typo must not look like success.
Revocation tells the operator what it does not do: the ex-member stops receiving
the key on their next connection but still holds the current one, so the message
ends with the command that rotates it.
The node's local control API is JSON only, on loopback, behind a per-run session
token (X-MeshBay-Token, printed at startup, file mode 0600). "Localhost only"
is not authentication: any local process can reach it, as can a page in the
operator's browser via DNS rebinding — and this API re-initialises group keys,
issues invitations and reads the audit log. There is no server-rendered dashboard;
the desktop client's Node page and the CLI are the two consumers, and each
operation endpoint is one _op(...) line onto ops (§5.4).
Over MNP, the node's own controls need a proved operator device. The
node-wide surface — node_status, which lists every group on the machine with
each root's absolute path, plus node_settings_set, roster_read,
denylist_read, denylist_clear and node_reload — is reachable when two
things hold: the account is the one the node belongs to, and the device on the
connection has proved (device_hello, §3.3) a key the roster holds as an
operator. The first alone is a claim in a token the hub issued, and NS4 does
not allow it to be authority: a hub that can name the operator is a hub that can
be one. The second is what it cannot forge, since it holds no user keys and
cannot countersign a device — the same property device linking rests on. A
browser that has never been paired therefore reads nothing here, exactly as it
can already sign nothing (§5.4).
The accepted cost, recorded as a choice: on a headless server the only admin path is the CLI. The CLI covers every operation, so this is acceptable — but it is a real capability reduction, not an oversight.
MNP is the path that must exist; loopback is the fallback. The operator of a node is not necessarily sitting at it. Any operator-facing control needs its MNP route first, or it renders for nobody on the web.
6.8 Node settings
Eight [node]settings affect what the node does rather than how it starts, and
their value is invisible until something goes wrong — so they are surfaced on the
Node page:
| Setting | Default | What it controls |
|---|---|---|
invite_ttl_hours |
168 | how long a member invitation stays valid |
pair_ttl_hours |
24 | how long an operator pairing code stays valid |
device_request_ttl_minutes |
60 | how long a device request waits for approval. Comfort, not security: the code is bound to the keys by its hash |
max_concurrent_streams |
8 | simultaneous video streams. One process per viewer, ~50 MB each; a slot is held for the length of a film, so this counts viewers |
max_concurrent_downloads |
8 | node-wide download leases (§5.5) |
max_concurrent_uploads |
8 | node-wide upload leases. A separate pool from downloads, because the two cost different things and one queue for both makes each cap meaningless |
max_upload_gb |
8 | the largest single upload, §6.4. Fractions are allowed, and it is read per chunk so a change reaches an upload already running |
transcode_incompatible_video |
true | whether browser-incompatible video is transcoded during streaming. Unlike remuxing this costs real CPU per viewer |
hardware_video_encode (default true) governs whether that transcode may use the
GPU. It is node.toml only and not on the page: the capability is established by
encoding and reading the result back (hwaccel.py), so the setting means "use it
if it works" and there is nothing for an operator to decide from a panel.
Turning transcoding off does not mean the same thing for every source. A codec with an MSE codec string falls back to a copy and the viewer's own decoder decides; a codec with none has nothing to fall back to and the stream is refused, naming this setting.
Settings are persisted in both roster.db and node.toml: the database for
immediate effect with no restart, the file so the value survives a wipe or a fresh
install. On startup the file is read and a database override wins. The TOML write
is a targeted line replacement, never a round-trip through a writer — that file is
hand-written, full of comments recording decisions, and a setting changed from a
panel must not rewrite the operator's file.
Transfer limits (§5.5) follow the same pattern, with per-group per-member caps as an operator-signed op.
7. The hub
7.1 Role — chosen, not minimal
Hub minimisation was considered and deferred, and may be dropped (decision D4). The hub keeps serving the web UI and remains in the trusted path by choice. That is a legitimate product call; what follows from it is carried deliberately rather than by accident (§2.3).
Stores: accounts (username, encrypted email, status, role), the group registry and membership, IP logs (one year, legal retention), node registrations, refresh tokens, notifications, the moderation blocklist, instance policy, and per-account device keys for hub login.
Does not store: file content, file names, private-group indexes, message content, private keys, group keys, keypair bundles, user identity keys, node IPs beyond ephemeral signaling.
Knows, unavoidably: who is a member of what, when nodes connect, and when a chat message was posted in which group and by which account id. The last is a stable identifier the hub needs in order to skip the author when creating notifications; it carries no content, and it is a known metadata leak rather than a solved problem.
Decides nothing about keys. Hub membership lets someone reach a node; the node's roster decides whether it wraps anything for them.
user_devices is not the key directory that was H3: nothing reads it but the
hub, nothing wraps a group key for it, and it is a different key from the per-node
identities. What it does cost is metadata — the hub knows how many devices an
account has and when each last signed in.
7.2 Node registration and signaling
Registration on the node socket requires a node-scoped token, verifies the node
record against the token subject, and derives group claims from the database: a
node may narrow the set to what it hosts but cannot widen it, and cannot displace a
live registration (C2). Narrowing goes all the way down: an empty
claim is a claim on nothing, never on everything. Reading it as "all of this
account's groups" made an unconfigured node a registered source for groups it could
not serve — including other members' — and since /v1/groups/{id}/nodes answers in
registration order, one such node reaching the hub first made a group unopenable for
everyone in it (2026-09-11). The ceiling applies to every message that changes the
set, not only to the registration: a node that may narrow on connecting and widen on
reload has no ceiling.
The socket is accepted before anyone is known, so the auth message must arrive within ten seconds or the socket is closed with 4001: an unbounded first read is a connection any stranger holds open for free. The node sends it on connecting.
A client must therefore treat that list as candidates rather than a ranking, and try
the next node on a not_hosted refusal (MESHBAY_NODE_PROTOCOL.md §6.3).
The node authenticates to the hub with an Ed25519 signature over a domain-separated
timestamped message — no password and no auth key on a node — and receives a
scope: "node" token that is refused for group management. The operator manages
groups from a client (NS7).
Signaling is rate-limited, SDP-size bounded, capped per user, and the caller must share an active group with the target node. Otherwise any authenticated user could make a third party's machine allocate peer connections on demand (H6). The address in a NAT-punch request must match the caller's source address.
The offer ceilings bound a member and never an ordinary one (AV30). They are sized against an account with twenty groups on three devices, because a refused offer is indistinguishable, to the reader, from a node that is down:
- Per account and per node, a budget: a burst of 120 offers, refilled at two a
second (
OFFER_BURST,OFFER_REFILL_PER_S). This is what bounds a member's cost to one machine, and it spares the others — an account that has spent it on one node still reaches every other. It is counted by account, not by address, because a mobile carrier puts many subscribers behind one IPv4 address. - Per account, 32 offers pending at once, across devices and nodes — an offer is pending for the node's round trip, or up to the fifteen-second answer timeout when a connected node is silent.
- Per address and per node, 600 a minute: a coarse guard in front of authentication, and nothing more.
- Every refusal carries
Retry-After, and a client sends the same offer again on 429, 502 or 503 — the hub busy or restarting — with growing, jittered waits (postOfferintransport.js). Any other failure, 404 for a node that is not connected above all, fails at once, so a dead node costs no time.
The node bounds its own total separately (MAX_PEER_SESSIONS in
webrtc_server.py), which is the limit that protects the machine whatever the
number of members.
A node registered for no group shares one with nobody, and is refused rather
than exempted. Written as "check membership if the node claims any group", the
rule skipped itself — membership, group status and the public-group gate
together — for precisely the node that AV1 made commonplace: the unconfigured
one, hosting nothing, which is also the one least able to absorb the work. No
legitimate connection is lost, because such a node refuses the handshake anyway
(group_id is mandatory, M1, and a node holding no group key refuses,
NS8); the refusal simply stops happening at the operator's expense.
X-Forwarded-For is honoured only from a trusted proxy, rightmost hop (M7),
through one helper so the behaviour is defined in one place — including for the
rate limiter, whose keying otherwise collapses to a single global bucket behind a
loopback proxy (third review L10).
7.3 Groups
A group's identity is its UUID, everywhere: the route, the node's configuration,
membership. A group name is unique per owner account, case-insensitively and
trimmed, enforced by a functional unique index; two different owners may each have
a photos. Names are displayed as name@owner, which is a label plus a create-time
check and not an addressing scheme. The handle is hub-local: the same
name@owner on two federated hubs are different groups, and a federated row shows
its source hub rather than an account.
visibility and join_policy are the two independent axes described in §3.5.
join_policy is read from the node's own configuration, never from the hub.
Public group creation is quota'd — ten live public groups per owner account, staff exempt. Public groups are the ones that cost other people something: they appear in the directory and are brokered to strangers. The check is at creation only, which is correct because the update endpoint refuses to change visibility.
An invitation link's hub half is a ticket, and it grants membership to one
account. group_invite_links holds sha256(ticket), the invitee's address as
the same blind index users has, and a masked form for the owner's list — never
the address. Redeeming requires a signed-in account whose verified address has
that index; anyone else holding the ticket is told only that it is not for this
account, and an unknown, used, expired or cancelled ticket, or a group no longer
active, is one uniform refusal. Creating a link says nothing about whether the
address has an account (M1). At most twenty outstanding per group, as on the
node; a lifetime clamped to thirty days; a node token may create one for its own
operator's group (the CLI) and may not ask for mail. The list answered to the
owner holds the links nobody has used yet, which are the ones there is still
something to do about: a redeemed one has become the member row it produced, and
showing both says the same thing twice. The row itself outlives the list by
thirty days, so the account that used a link is answered on a reload or in a
second tab rather than refused. What the binding holds
against, per the convention: third parties — a messaging service that
previews the link, a forwarded mail — and not this hub, which verifies the
addresses it compares and could already be anybody.
Which nodes host a group is answered to its members. For a public group that is everyone, which is what public means; for a private one it is the membership row and nothing else. Answering any authenticated account — as it did while only the public case was checked — hands whoever knows the group id the identities of the machines hosting it, and an ex-member knows that id for ever. Nothing needs it before joining: an open join writes the membership row first, and an invitation registers the invitee's when the code is created.
7.4 Instance policy
hub_settings is a key/value table an admin edits at runtime. It is instance
policy, not group content: it says how this hub behaves and holds nothing about
any group's files, index, membership or keys, so §1.3 is untouched.
The first entry is allow_public_groups. Switched off, server-side and read live on
every path the hub mediates: creating a public group is refused (staff included —
the way back is to re-enable, not to slip past), the public directory returns
nothing local and federated, open-joining is refused, a non-member is handed no
node to connect to, the "this node hosts an open group, admit anyone" signaling
fallback is dropped, and the federation export advertises nothing.
It is a directory-and-brokering control, not a remote kill. Existing members
keep their membership and their access. A node whose operator set join_policy =
"open" still pins and serves whoever reaches it directly over MNP; what the switch
removes is the hub-provided ways to find and reach such a node.
What the hub answers without an account is a reviewed list.
test_unauthenticated_surface.py walks every route and fails on one that takes no
authentication dependency and is not listed there with its reason; routes that
authenticate in their own body (a signature, an e-mailed code, an MHP token) are
listed with what they check. The hub publishes no API description — no /docs,
/redoc or /openapi.json — in the code, not in a proxy rule, so a packaged
install behind any proxy publishes none either.
The mail bounds (mail.*) and the sign-in lockout (login.max_failures,
default 4, and login.lockout_minutes, default 60 — §7.7) live in the same table
for the same reason: they are what an operator changes while the hub is serving,
from the panel, without a restart. Each value is clamped to published bounds, and
max_failures = 0 turns the lockout off.
7.5 Moderation
Two verbs on a group, and they are distinct things:
suspend |
revoke |
|
|---|---|---|
| Hub | status = "suspended" |
status = "revoked" |
| Node | nothing | signed revocation broadcast → denylist + live sessions dropped, persisted across a restart |
| Reversible from the panel | yes | no |
The client shows the real state, not a blanket one. Revocation is honoured by nodes and the denylist survives a restart (H4); signaling refuses a group that is not active.
Moderator is not administrator. The user-patch handler is split by field: a
moderator may act on the fields moderation needs and may not write role.
Content reporting requires authentication, distinct reporters and a rate limit, and is refused when public groups are off. An unauthenticated endpoint that blocklists a content hash after two reports is a network-wide censorship and DoS primitive for anyone who learns a public file's id.
The exact-hash CSAM check is structural, not yet functional — production databases are perceptual — and is stated as such so it is not relied on operationally.
7.6 Federation (MHP)
Federation is closed in the code, and every MHP route refuses with a stated 503.
federation.FEDERATION_ENABLEDis the only thing that decides it — a constant rather than a setting, because a switch in an admin panel invites an operator to turn on something that has never worked between two machines./v1/hub/inforeports it, since themhp_versionbeside it would otherwise be a claim this hub does not honour.The reason is not the design below. It is that nothing has ever run it: two hubs have never completed one authenticated request between them (AV14 — the issuer signed with a key bound before it was loaded and named itself after the reference deployment whatever it was called, and the verifier named no audience for the
audthe issuer sets). Both were found by reading, and both stood for a month behind a green suite, because a second implementation of a peer proves the protocol and nothing about two machines — the sentence §12 already writes about a second implementation of the client. It re-opens when a second hub has been stood up and the exchange run both ways.What the closure does not touch: the public directory still reads whatever
federated_groupsholds, which is nothing, because nothing can arrive.
Peer hubs exchange directory rows and revocations. The trust rules, which hold when it re-opens:
- a pushed row's source is bound to the signer, not taken from the payload;
- the token audience is checked;
- a push is capped, and replays are rejected;
- revocation acts on the peer's own directory entries — it does not reach nodes, and nothing local hosts a federated group.
7.7 Account lifecycle
A user can delete their own account from Settings behind a passphrase re-entry — a live token may be a borrowed laptop, and the bar for something with blast radius is proof of the passphrase. An admin can delete one too.
The row is tombstoned rather than dropped: username released, email and password hash cleared, node linking key dropped, memberships, notifications, refresh tokens, node registrations, device keys and public-swarm sources removed, active tokens refused at once by a status check rather than left to expire. Device keys go because the desktop client keeps its half: left on the tombstone, the key would refuse that installation to the next account created from it.
Two things survive on purpose:
- The IP log, for its legal retention period, and it stays attributable — detaching it would keep the data and lose the only thing it is for. The name is copied onto those rows as the account goes, since the join that used to supply it would answer with the tombstone.
- Everything on a node. Files, the pinned identity and the keypair bundle live on machines the hub does not command — the same sovereignty that makes admission work. Deleting a hub account is not an erasure request to the operators who host you; the operator surface is where that happens, and the docs must say so.
Groups the account owns decide between the two routes:
- The owner's own deletion is refused while the account still owns groups, rather than cascading into other people's data — the owner can hand them over first.
- An administrator's deletion deletes them with the account. It is the route
an erasure ordered by an authority takes, and it cannot wait on the person it is
about. Everything on the hub that references those groups goes too
(
db/purge.py, which finds the referencing tables from the schema, so none is left to fail a foreign key on PostgreSQL). Then a signed revocation for the account and for each group goes to every connected node: an access token already issued stays valid on a node until it expires, and the revocation is what makes the nodes refuse the account and close the groups' sessions now. A node that is offline misses it.
Registration is gated by a CAPTCHA whenever one is configured — unconditionally, not only when some other field is absent, or the real client's ordinary request skips it. The desktop client renders the widget too.
Sessions end three ways, all admin settings in hours (session.*, §7.4):
- A browser signs itself out after
browser_idle_hourswith no input and no<video>/<audio>playing. The page measures it, because the hub cannot: it hears a renewal from any open tab, attended or not, and nothing while a film plays. The last-active time is shared by the browser's tabs and survives the browser being closed. The desktop application is exempt — its owner's machine, which signs back in with its device key, also after a renewal fails. - A refresh token unused for
refresh_idle_hoursstops renewing, never below the access token's life plus an hour, and no session renews pastmax_hoursafter its sign-in. - Signing out revokes the refresh token on the hub, not only in the browser, and "sign out everywhere" revokes every one the account holds. Access tokens already issued run out on their own.
Passphrase sign-in locks per username. After login.max_failures wrong
passphrases (§7.4) the name is refused with 429 account_locked and a
Retry-After for login.lockout_minutes, without the passphrase being checked.
The per-IP rate limit bounds one address, and IPv6 hands every subscriber a /64
of them; an online guess targets an account, so the account is what is counted.
The rules that make this safe:
- Counted by the name as typed, existing or not. An unknown name locks exactly
like a real one, so
loginstays uniform (M1). The key is a hash: people type passphrases into the username field. - The attempt is taken before the check, in one statement — an
INSERT … ON CONFLICT DO UPDATE … WHERE … RETURNING— so a concurrent burst gets no more attempts than the limit. A request that checked no passphrase gives its attempt back. - Every path that checks the passphrase counts on the same row: sign-in, passphrase change and account deletion. A right passphrase clears it; failures older than the window age out.
- A lockout refuses passphrase sign-in and nothing else. Open sessions, token
renewal and device sign-in continue, and a reset code sent to the address on
file clears it — so a stranger who locks a public username costs its owner at
most a new sign-in (AV26). A session learns its own lockout from
/v1/users/me, because a passphrase change re-wraps every node's bundle before the hub accepts the new passphrase and must not start when the hub would refuse.
8. Clients
8.1 Two clients, deliberately
| Hub-served web SPA | Desktop client | |
|---|---|---|
| Distribution | served by the hub | installed, signed release |
| Code integrity | T3 accepted — the hub can inject | detectable if reproducible builds ship |
| Key storage | IndexedDB / sessionStorage, plus a bundle on each node | OS-protected local storage; keys never bundled |
| Transport | WebRTC | WebRTC + QUIC (sidecar, for hub-less group://) |
| Downloads | to disk where the engine allows it | native, streamed, unlimited |
| Positioning | convenience tier — zero install | recommended for sensitive use |
The SPA is not deprecated and stays. It is the zero-install path, and the objective is explicit: a native client must not prevent web use. It must be labelled honestly — the application page states that the hub serves this code, and the docs never claim end-to-end integrity for that path.
Several browsers, one identity per node. A browser keeps nothing durable the user controls, so the identity it creates for a node is left with that node, encrypted under the passphrase. Any other browser recovers it there with the passphrase alone: same identity, same pin, no second code. Joining a different node creates a different key and needs that operator's code — the first contact it has always needed. This is what makes the product behave the way people expect, and it is also C4, with a blast radius of one node.
8.2 The desktop client
Electron, with an optional Python sidecar for hub-less group:// over QUIC.
The shell choice follows from what the interface actually depends on: not "the web"
in general but engine-class APIs — RTCPeerConnection, WebCrypto X25519/Ed25519,
MSE, service workers, File System Access. Keeping the engine keeps the transport,
crypto, key-derivation, download and player modules as the client. They are not
browser workarounds to be deleted once native; they are the implementation.
The non-negotiable: UI assets ship inside the package and load from disk. A shell pointing at the hub's application URL is a browser with a different icon and fixes nothing.
What running it establishes, and what each fact costs:
- A CSP in a meta tag silently drops
frame-ancestors. It is sent as a header by the protocol handler. - Service workers do not work on a custom scheme. The application has none and uses the native save dialog; the worker stays for the browser.
- A secure context is what makes crypto exist at all — without it the whole subtle-crypto surface is undefined, AEAD included.
- The renderer cannot call the hub. Its custom-scheme origin is refused by CORS, and the hub deliberately has no CORS middleware — its API is reachable from no web origin. Every hub call leaves from the main process, which refuses any origin that is not the signed-in hub. A script served by the hub is refused by the policy, which is T3's mitigation demonstrated rather than asserted.
- The device's hub key lives in the main process, never in the renderer. Generated, stored and used there; the interface asks for a signature and is never handed a key. Same rule as the save dialog, and for the same reason: the renderer parses decrypted content from nodes, which is attacker-controlled input.
- OS-backed secret storage is real on a desktop and honest without one. With a keyring it is keyring-backed; headless, the same code reports unavailable and refuses to store rather than downgrading silently.
- Installation places files, never secrets. No key generation in a package's post-install step or an installer custom action — a golden image would give every machine the same key.
- Hardware video decoding is asked for and then verified. Chromium ships VA-API
off on Linux, so the client enables it where a render node and a driver are
present, and then asks
navigator.mediaCapabilitieswhether the codec the node streams really decodespowerEfficiently. A no moves to the next GL backend on the next launch, and an exhausted list drops the switches — a feature name that a Chromium release renamed must not pass for a feature that is on, and--ignore-gpu-blocklistmust not survive on a machine it did not help. The package recommends the drivers; nothing requires them.
Two guards apply to anything the hub can display inside the application, which is a phishing surface: plain text or a very restricted markup subset, never raw HTML; and a visually distinct region labelled as a message from the hub operator, never a modal that can imitate application UI.
The download page is a security page: it publishes the release key fingerprint, and a hostile hub serves that page too. The fingerprint must also be published somewhere the hub does not control, or the relocation of trust is circular.
8.3 One UI source
packages/meshbay-hub/src/meshbay_hub/static/ is the interface, for the web and
the application alike. The client's build copies it and CI fails if the copy drifts.
Never edit the copy by hand.
Shipping the UI in a package creates version skew for the first time. Today the
SPA and the hub deploy together, so a response shape and its caller change in one
commit. Once the UI is installed rather than served, the hub API is a compatibility
surface — which is why GET /v1/hub/version carries a minimum client version, and
why the client checks it before connecting (§5.6). Cheap now, awkward later.
A second copy of the hub address is what breaks the application, not the
protocol. A module that decides where the hub is — "empty string, same origin" —
is true of a page the hub served and false of one loaded from a package, where a
relative API call hits the application's own protocol handler and sign-up and
sign-in fail. There is one seam, platform.hubBase(), and a test refuses any file
that decides where the hub is or fetches the API relative to the page origin.
8.4 URL space
| Space | Served by | Seen by the desktop client |
|---|---|---|
| landing, about, downloads, news | the static site overlay | ❌ never |
| the application | the hub (SPA) | ❌ never |
| the versioned API and the signaling socket | the hub | ✅ only this |
The desktop client issues no request outside the API and the signaling socket. That is a testable invariant, and it is what any feature adding third-party egress must preserve — which is one of the reasons enrichment is node-side (§6.5).
8.5 Downloads and streaming
Downloads go to disk, never through RAM, on every platform. There are three mechanisms — a granted directory handle, a service worker streaming a response, and a blob as the floor — and which exist depends on the engine.
A fallback chain reaches its floor silently. Where the first two do not exist, every download went through memory and the only visible symptom was a save dialog at the end instead of the start. When a chain degrades, check what the floor costs on every platform that will reach it.
Specific rules the download path is built on:
- A service worker being active is not the page being controlled. An uncontrolled page's requests never reach the fetch handler, so the stream is handed over and never asked for. Require the controller, and have the worker confirm it served the request.
- An idle service worker is killed, and a streaming response is not "something to do". The page pings the worker while it writes and the worker answers, because receiving a message is the event that resets the timer.
- Every await on a download path is bounded and says which chunk it gave up on. An unbounded await is a freeze nobody can report.
- A filename in a content-disposition header needs both forms. The RFC 5987
encoding uses
'as a delimiter and the standard escaper does not escape it, so a plain ASCII fallback rides alongside — the next surprise loses accents rather than the whole name. - Three headers decide whether a page may frame itself and they must agree — the frame source policy, the frame-ancestors policy and the legacy frame option. Same-origin framing is what a streamed download needs; refusing every foreign origin is unaffected.
Pause and resume, and where resumability actually lives. A pause button that quietly restarts a download from zero is worse than no pause button, so the interface offers only what the target can do.
Resumability is a property of the target, not of the platform. The same engine yields a resumable target from a granted folder and an unresumable one from a service worker, on the same page, for two files in the same batch. So each target declares it itself and the record travels with the transfer; the widget renders from that record. This is the fallback-chain rule (above) applied one level up: whatever tier a transfer ended up on, it records which, and nothing infers it from the platform.
Two design decisions rather than implementation details:
- A paused transfer holds nothing. Resuming rejoins the queue at the tail and the interface says so. Anything else lets one member close the node by pausing.
- A resumed file is truncated down to the last whole chunk, never appended to. Writes are sequential whole chunks, so a size that is not a chunk multiple means an interrupted write — and a silently corrupted download is worse than a failed one. The resume record also stores the file's content id, so a resume whose file is no longer in the index fails with "this file has changed on the node", which is the truth.
Uploads resume through the seal, not around it. The node already holds the partial state; the question "how much do you have" is asked as an ordinary sealed upload message with no bytes and a probe index, and answered inside the sealed ack. Asking on the lease message instead would have put the operator's filenames on an unsealed message — precisely what sealing the write path bought. The partial state is keyed by member, directory and filename in the group context rather than the session, so a reconnect finds it, and an orphaned partial with no live lease is reaped on a timer and at startup. The four upload protections (§6.4) are untouched by any of this.
Video streaming is fragmented-MP4 remux (or transcode where the codec has no MSE string) fed to a source buffer, with the node holding one slot per viewer.
-
The re-encode runs on the GPU wherever one works — VA-API on Linux, Quick Sync or NVENC on Windows. Which one is established by encoding 1080p and reading the file back, never inferred from the hardware or from ffmpeg's encoder list, and nothing is accepted that does not produce the exact profile and level
stream_initannounces: an encoder that wrote another level would make the node's own codec string a lie, and the client checks that string before it trusts a byte. Every mode falls back to libx264 — per source codec, because a GPU that decodes HEVC may have no decoder for MPEG-4 Part 2 and only asking it finds out. This is what lets a low-power node keeptranscode_incompatible_videoon: the setting refuses a capability, and hardware encoding is the mechanism that makes refusing it unnecessary. -
Flow control is a window, not a debt. Read-ahead is bounded by time past the playhead, with a small window of segments in flight topped up as they land, driven by a clock and by playback and never by arriving data. Granting a credit per append pulls at network speed, fills the buffer ceiling, and then goes silent for the length of the accumulated balance.
- The read-ahead bound is a byte budget, stated in seconds. What a browser limits is bytes, so a bound written in seconds has to be sized for the highest-bitrate file in the library and every ordinary file then holds a fraction of what the same browser would have taken — measured against a 100 MB ceiling, a 1.0 Mbit/s film reached its ninety seconds on 20.7 MB and a 2.3 Mbit/s one on 45.6 MB. On a mobile network that unspent ceiling is the whole of the margin there is for a dead spot. The budget is therefore in bytes, converted with the file's own average bitrate, and the fixed bound is its floor and never its ceiling: nothing pulls less than it used to.
- A ceiling that varies by engine and device is walked up to, not declared. The two ways to find a limit nobody publishes are to overshoot it or to approach it, and overshooting costs a refused append every time it is tried: a flat budget against a smaller ceiling measured 1386 refusals on a film that had none. So the budget never exceeds one step past the most read-ahead that SourceBuffer has actually accepted, and a refusal takes the ceiling down below where it broke and leaves it there — backing off and climbing again is a sawtooth with a refused append at every peak.
- The budget only grows while the film is playing. What it buys is a film that survives a dead spot, and a film nobody has started cannot be interrupted. Left to grow on the loading screen it filled a small engine's whole buffer — 40 MB where the fixed bound took 8 — with film on a connection that is paying for it, and reached a ceiling the fixed bound never came near. Autoplay is blocked on a phone more often than not, so that is the ordinary first state of every playback. Pausing does not shrink it: the lead already paid for is the viewer's.
- A reconnection carries the read-ahead across; it does not spend it. The old stream dies with the connection, so something must be asked for again, and the question is where. Restarting at the playhead goes through the reinit that empties the source buffer — which a seek needs and a reconnection does not: the viewer navigated nowhere, the film kept playing out of the buffer for the whole outage, and emptying it at the moment the link returns is what finally stops the picture. So where there is read-ahead worth keeping, the stream carries on from the end of the buffer and what is there is left alone. The two modes are not interchangeable — resuming keeps the buffer and must not move the playhead, seeking discards it and must — and the choice is one pure function so it can be tested by being run. Below a few seconds of read-ahead, and for a stream that already ended, the seek is still what happens: there is nothing to save, and it is the proven path. The node lands on a keyframe at or before what is asked for, so the new material overlaps the tail of the old and the coded frame processing replaces it; a start that somehow fell past the buffer would leave a gap, which is a permanent silent stall, so that case falls back to the reinit rather than trusting the reasoning.
- An append refused for quota is not retried until something frees room. It will be refused again for as long as the answer has not changed, and the pipeline's clock calls the append path several times a second: the retry alone cost minutes of refusals while the film played far enough for eviction to have anything to drop. Eviction is the only thing that gives a SourceBuffer room back, so its success is what lifts the hold — not a timer, and not the playhead, which moves long before the answer changes. This cannot deadlock: a full buffer is one holding film the playhead has not reached.
- Credit follows the buffer, decided in one place. The append path grants nothing, because the buffer's update event fires for evictions too — crediting from it pays the node for the player's own housekeeping.
- Flow-control accounting comes before every early return. A segment that arrived is no longer in flight, whatever is then done with it.
- A new stream starts from a known state, reset at the start of the stream and never in the teardown of the one before, which is skippable.
- A viewer holding credit deliberately must say so, or the node's stall timeout ends a film that is merely paused.
- Seeking restarts the source with an index seek before the input, clamped away from the end and echoed back; the client supplies the timestamp offset, because copying timestamps does not preserve position.
stream_init.startnames where the picture begins, never where the viewer dragged, and on the copy path that position is measured, never predicted. The client buildsSourceBuffer.timestampOffsetout of this number and a subtitle cue carries the source's own absolute time, so every second of disagreement puts a line on screen a second away from the voice saying it. Reading the key frames and taking the last one at or before the request answers a different question twice over: Matroska's Cues index only some keyframes, so an index seek backs off to an indexed one that can be much earlier, and the landing point moves with which streams are mapped, because the container is positioned where every mapped stream has data — one real seek answered 4909.863 s from the frame list and delivered 4907.236 s. So the node runs the same seek with the same mapping, copies one frame under-copyts, and reads the answer back: 0.06–0.07 s, cheaper than the scan it replaced. Only the copy path needs it — re-encoded video begins exactly where it is asked to.- A seek trims what it can, and what it can differs per stream — which is a desync, not an inconvenience. Accurate seeking cannot trim copied video, which must begin on a keyframe, but it does trim re-encoded audio to the exact request. The output then carries video from the keyframe and audio from the request, a whole GOP apart, with a hole between them. So accurate seeking is off wherever video is copied and on wherever it is re-encoded, which is the only place the two can agree on where to begin.
Every timestamp was correct while this was happening. First PTS per stream, durations, spans, and the browser's own audio/video delta through MediaSource all agreed, because both streams genuinely sat where the container said. Only the content at a given instant was displaced. A property that every timestamp-shaped check confirms is not thereby true: this one needed the output decoded and compared against the source, frame against frame and envelope against envelope. - The viewer picks the audio track, and picking one is a seek. One ffmpeg carries one audio track, so there is nothing to switch inside a running stream: the node is asked again at the current position and the source buffer is reset the way any seek resets it. It costs nothing extra — audio is transcoded on every stream anyway — and mapping the first track unconditionally, which is what this replaced, made a dubbed library playable in one language only. The node reports the tracks it found and the one it used; the client draws its selector from that list and from no version number, which is what keeps the addition MINOR. - Losing a peer must stop its work, not merely forget it — anything holding a resource is shut down on the way out, or a closed tab transcodes for the length of the credit timeout. - Resume positions are per file, per device, in local storage. No protocol, and nothing new learns what you watch.
9. Group applications
9.1 The plug-in architecture
A group's UI is a set of pluggable applications, not one page. Two things motivated the split: one file had become the thing every unrelated change touched, and the roadmap wanted several more group-level surfaces — none of which need a protocol change, because the indexer already classifies files as video, audio or image and they read the same index, chunk and stream messages the explorer already uses.
group-page.js ─┬─ the shell: the connection, the file index, the tab bar,
│ the video/preview modals — nothing app-specific
├─ apps.js ─── the registry: [{ key, icon, labelKey, Component, Settings? }]
├─ chat-app.js · files-app.js · video-app.js · music-app.js · photos-app.js
└─ group-settings.js — not an app, always present, never toggleable
Settings is not an app and cannot be disabled — it is the one way back if everything else were turned off.
Shared infrastructure lives in its own modules (icon.js, file-utils.js,
hub-client.js, settings-ui.js, folder-tree.js, source-merge.js) rather than
being re-exported from the shell, because the shell importing an app that imports
the shell is a cycle, and ES modules answer that with a temporal-dead-zone error
at first render: the component simply does not appear, with nothing in the console
to say why.
9.2 What every application receives
The shell builds one props object per render and spreads it into whichever application is active. Every registered component gets the same context and destructures what it needs — a new application does not get a bespoke prop list.
| Prop | Why it is here rather than local state |
|---|---|
entries, nodeDirs, nodeRoots |
the group's file index. Chat needs it too, for image attachments — lifting it avoids two copies going stale against each other |
applyIndex(...) |
anything that mutates files calls this, so every application sees the result |
onPreview(entry) |
opens the shell's modal; an application does not own modal state |
transportRef, gekRef |
refs, never state, so a reconnect does not re-render every application |
deviceReady |
the exception, and why it is a prop. A ref not re-rendering is right for a transport reached into on demand and wrong for a fact about the connection an application renders from |
mayUpload |
computed once; a second derivation would eventually disagree with the first |
An application that needs local state owns it. One pattern is worth carrying: any notion of "current location within the group" resets on group change, because a directory from the group just left rarely exists in the one just entered.
9.3 Enablement and settings
enabled_apps is a per-group setting on the same pattern as everything else the
operator decides: stored on the node (roster.db — not the hub, not
node.toml, for the reason in §1.3 and §6.8), changed by a signed operator
instruction, and enforced by the node refusing an unrecognised or empty set.
Validation happens before a signature is ever requested. The whole set is signed in one message rather than one op per application, so ticking several boxes costs one signature. The subject is the sorted, comma-joined list, built identically on both sides so the two arrive at identical bytes.
The node's allow-list is the server-side enforcement — a client that names an application this node does not know is refused, and an application the node refused could not demonstrate anything. That entry and the client's registry line are the whole of what adding an application costs.
A group that has said nothing gets Chat and Files. files is always enabled
and not toggleable, and is added to the list at every writer so the two agree.
The ability to hide it was misleading: the protocol permits root exploration
regardless, so hiding the tab only ever misled.
A group with no stored set gets the default, never the whole registry. Falling back to everything registered would turn on an application nobody chose — including one shipped behind a development flag.
Changes are broadcast to everyone connected, so a disabled tab disappears without waiting for a reconnection. A client that has not yet received the list shows everything registered — a node that predates an application hides nothing.
An application's directories are the same shape one level down: one generic
signed op (app_directories) keyed by the application's own registry name, stored
under <key>_directories, one MNP message, one loopback route. Adding an
application adds no function, no message type and no route — which is what
"plug-in architecture" has to mean to be worth the phrase.
The per-application variants are gone, and the reason is the interesting part. Three messages, three signed ops, three handlers and three
opswrappers were the same instruction three times, differing only in the key they wrote and whether they carried a string or a list. That shape is what made adding an application mean adding a message type, an op, a handler and a widget — and it meant three validation paths, of which the older ones validated nothing: a typo was stored, matched no entry, and the application showed an empty tab with no way to tell "misconfigured" from "no files yet". One op has one validation path, and an unknown application name is refused rather than stored.What stays is a read fallback: the roster still reads the older per-app keys out of its settings table, because that is a key on an operator's disk rather than on the wire, and a node upgraded into this must find its own configuration (§5.6).
Each application's settings pane is its own file, named in its registry entry, and every pane takes the same props and nothing else. The split is the point:
What every application has, the page does generically; what one application alone has, the pane does itself. Pointing an application at folders goes through the shared saver; a third-party credential or a per-group switch is the pane's own business, made with the transport it is handed.
An application that only needs directories therefore touches neither the settings page nor the shell.
The folder picker asks the node for nothing. The tree is derived from paths the client already holds, so it shows what the group's index contains and no more. There is no folder-browsing protocol and this does not add one.
9.4 Adding an application
<name>-app.jsexporting a component with the standard props shape.- Register it in the registry.
keyis the wire identifier: it must match the node's allow-list and it is the row the application's directories are stored under. One identifier per application, everywhere. Register the component — and its settings pane — throughlazy(() => import('./<name>-app.js'), 'Name')(lazy.js), not a static import: the module is then fetched the first time the tab is opened instead of with the first page, andtest_first_load_is_lean.py(add the file to itsON_DEMANDset) refuses a static import that would bring it back. <name>-app-settings.jsif it has anything to configure. Do not import the settings page — that is the cycle in §9.1.- Add the key to the node's allow-list, and — if the application keeps
directories — to
NodeDaemon.APP_DIR_KEYS, the one list a group's context is built from. Nothing else on the node may name an application. - i18n: at minimum a tab label key in all ten catalogues. A settings key added to the client must be added ten times; write the table and generate the insert.
- Nothing to register for caching. Every file under
static/feeds the/a/<hash>/fingerprint, subdirectories included, so a new or changed module moves the URL by itself — which matters for a module reached through the registry, since nothing imports it by name. - Add the file to the test file-set lists — hook ordering, sticky headers, transport contracts. A file missing from those lists is never checked, which fails silently rather than loudly.
- Toolbars pin. A toolbar is a direct child of the page root and is opaque, or content scrolls visibly through it; if anything pins below it, it must publish its own height, which is never a constant because it wraps on a phone. An application with no toolbar renders none — an empty band still holds a strip of the page open.
No protocol change, no hub change, no daemon change. Steps 4 and 7 are the only node-side and test-side touches, and both are allow-lists.
A list of application names is only ever kept in one place, and everything downstream is derived from it. There were three. The daemon built a group's context from one; the handshake ack was assembled from a copy that had already lost an entry — the reference application's, so the one application that exists to prove a new one needs no special-casing was the single one whose directories never reached a client; and the client shell named three applications by hand while the live-update path beside it was already generic.
The fix that holds is removing the copies, not syncing them: the ack emits whatever
<app>_directoriesthe context carries, and the shell reads the ack's own keys. Neither can drift, because neither has anything of its own to drift from.Where the one list lives matters too. It is on the daemon, which is what wires a group's context; the roster, the operator ops, the config and the root set must name no application at all, and a test holds them to it. That is the property the reference application exists to demonstrate, and it is the reason a first attempt at this fix — moving the list to the roster, where the directory storage lives — was wrong and was caught.
A reference application exists in the tree behind a development flag. Every other test of this architecture reads source for the absence of application names, which proves nobody wrote a special case — not that a new application works. Writing a real one immediately found two places the claim was only nearly true.
9.5 Files
The explorer: roots, folders, sorting, selection, upload where the current root is writable and available, download, folder-as-zip, and the eject/plug control beside each removable root. Entries from unavailable roots are filtered out.
9.6 Chat
Messages, threads, attachments, and link previews (§6.5). The composer gates on the connection having identified a device (§3.3) and on a writable root existing for attachments; the attachment directory is a single writable folder chosen by the operator, and the paperclip is disabled with a stated reason when it is not usable.
Own-ness is decided from the account id, in one place, never by comparing display names — and the optimistic local echo carries an explicit flag rather than inventing an identity for itself.
9.7 Videos
A poster browser over the video files in the group's configured folders.
Two modes, both driven by the index: a poster grid with third-party metadata and artwork, falling back to a thumbnail card with the cleaned filename when the lookup returns nothing or a low-confidence match; and a flat, folder-driven list with no third-party dependency, which keeps working with the service switched off node-wide. The mode toggle is a per-device display preference in local storage — it has no authority implication, so it needs none of the signed-op treatment.
A library is read a page at a time, in Videos and Music alike, on the group
page and in Search: the pinned toolbar carries previous/next arrows and the range
shown, and a page is a slice of exactly what the mode would otherwise draw, in the
same order (pager.js). The size — 50 by default, 10 to 200 in steps of 10 — is a
per-account preference on the hub (media_page_size, behind the same
allowlist as the others), because it describes the reader and not the group's
content (§1.3). Show merging (below) runs before the page is cut, so a merged show
counts as one card.
Scoping to folders is not a display preference, because it decides what every member's tab shows. It is a per-group setting changed by a signed op, broadcast to connected members, and validated before a signature is requested: a candidate path is resolved against the group's real root set and must name a real, readable directory, so a stale path never reaches the operator's browser as a signing prompt.
Grouping is by unit, not by file: one card per film, one per show — expanding to seasons and episodes. Seasons carry their own text where the service supplies it, falling back to the show's. An operator can correct a wrong automatic match, and correcting one applies to the unit rather than to a single file (for shows) or to the one file (for films), because those are the units each actually is.
The filename parser is a fallback, and directory context is what bare-filename parsing cannot supply. Matching is a scored ladder rather than the first candidate to clear a threshold, with a year-exact rescue for a weak top hit.
Thumbnails, probes and metadata are node-side (§6.5). Index-time probing runs in its own bounded pool after a file is first seen: the file appears in the index immediately with size and hash, and an index delta fills in the technical and parsed fields once ready. No scan is blocked waiting for enrichment.
9.8 Music
An album browser and a player over the audio files in the group's configured folders.
Metadata mostly already exists in the files themselves, which is the real difference from Videos. The order of trust is embedded tags, then filename and folder parsing for what tags do not supply, then a third-party lookup for canonical spelling, a missing field, or cover art where none is embedded — node-side and cached like any other enrichment, and needing no credential, unlike Videos.
No playback protocol change is needed at all. A track is a few megabytes, so playback reuses the ordinary download-and-decrypt pipeline and hands a blob to an audio element. No streaming request, no transcode pool, no stream slot, nothing added to the node's streaming machinery. The one exception is narrow and extension-gated: two container formats tag perfectly and decode in no mainstream engine, so for those the node performs a one-shot whole-file conversion, cached under its own content hash and served through the ordinary chunk path.
That conversion has a size ceiling, and it is the cache's rather than memory's. The result is one row in a 512 MB least-recently-used store sized for thumbnails, posters and subtitle tracks; a source hours long would be a single row that evicts most of the cache to fit and is evicted again by the next few thumbnails. Whole-file is the right shape for the few-megabyte sources this exists for, and the wrong one past that — serving audio of that length means streaming the conversion, which is a different feature and not this one.
The player is persistent across tabs, at shell level: closing the tab must not stop the music. The next queued track is prefetched while one plays — client-side only, a small in-memory cache evicted as the queue moves.
Music is scoped to folders on the same mechanism as Videos. The reason is not cost — tag reads are cheap — it is mixing: a shared tree with more than one kind of thing under it puts everything into one undifferentiated view with no way to narrow it.
9.9 Photos
An album browser over the image files in the group's configured folders. It is smaller than Videos and Music, deliberately, in two ways:
- One album-grid view, no mode toggle, because there is nothing to fall back from.
- No third-party service at all — there is nothing to match a photo to. It already is what it is, per its own folder and filename.
"Several root folders rather than one" used to be the third, and is not a difference any more. The generic
app_directoriesop (§9.3) gave every application a set of folders, so what Photos needed — a library scattered across directories — is what Videos and Music get as well. The pane that chooses them is the same shared widget inmultimode for all three; Chat's is the onlysingleone, because an attachment is written to exactly one place. It is worth recording because the per-application shapes converging is the whole point of §9.3, and this section had been left stating the difference it removed.
An album is a directory, exactly as a season is a folder in Videos. Thumbnails are node-side, orientation-corrected, and delivered through the same chunk path.
EXIF is read locally on the node and narrowed on purpose: a capture time and a camera, and never GPS, anywhere, in any field a client receives. The claim this supports is precise, and the one it must not make matters more:
GPS is in the file, for most phone photos, in the original bytes any member with file access can already download. What this design controls is what the application computes and surfaces — not what the underlying file contains.
9.10 Playlists
Playlists are the first feature to need per-account state that spans several
groups on several nodes. Built 2026-09-16; full design, and every place the
design was wrong before it was built, in playlists.md.
A playlist belongs to one account and is never shared with other members. That scope is what keeps the merge problem small.
Where the state lives. Not the hub — and the operative rule is narrower than "the hub stores nothing about a user", because it already holds small per-account preferences behind an allowlist. The rule is:
No content metadata on the hub.
A playlist is literally a list of content hashes of private-group files, plus the titles needed to render while nodes are offline. That is the exact object H7 removed from the hub, and it is the same rule that keeps resume positions local: nothing new learns what you watch. An encrypted blob on the hub is technically trivial and is still refused.
It lives on the node instead, as opaque per-account blobs in bundles.db — the
same shape as the keypair bundle, which the node already stores and cannot read.
No new trust boundary: the node is not asked to hold a kind of thing it does
not already hold for that same account.
One blob per playlist, plus a small manifest — not one blob for the collection, and the reason is write amplification rather than size. Under a single blob, starring one track rewrites and re-uploads the whole collection to every node reached; split, it rewrites that one playlist. The manifest — names, revisions, tombstones, counts, a few KB — is also the only thing every menu needs, so "add to playlist" draws instantly with every node offline, and a body is fetched only when its playlist is opened or played. Blobs are compressed before sealing and padded after: the payload is repetitive enough to be worth a factor of three, and the padding is what stops a ciphertext length from counting somebody's tracks. The node caps each blob and the account's total, and refuses rather than truncates — a truncating cap silently loses tracks, which is the failure the whole design exists to prevent.
The key is the one thing that must not be got wrong. Identity keys are per node
(§3.2), so a blob encrypted under one is unreadable from every other node — the
precise opposite of the requirement. The only secret an account holds everywhere
is the bundle key, so playlist_key = HKDF(bundle_key, info =
"meshbay:playlists:v1"): one derivation at sign-in, two handles, no second
Argon2 run, and a purpose-separated subkey rather than the bundle key reused with a
different AAD (§4.4's rule). The nonce is 96 random bits and never a counter, for
exactly the reason chat's is (§4.5): two devices of one account derive the same
key, which is the point. The AAD names the blob's kind, so one playlist's body
cannot be served in place of another's.
Merge is the hard third, and the granularity is what makes it tractable. The unit is one playlist, not the collection — which the per-playlist blob now makes true of the storage as well, so two devices editing two playlists do not even write the same row. Revision counters order writes, never the wall clock; and a deletion is a tombstone, never an absence — an absence is indistinguishable from a device that has not seen the addition yet. A node that is offline for a month therefore cannot corrupt anything: it holds an older revision of some playlists and is overwritten per playlist, not wholesale.
The interface is a menu inside Music, not a page. Music's player is already persistent at shell level and already holds a queue that crosses groups; loading a playlist replaces that queue, and below that call a playlist and an album are indistinguishable — so auto-advance, shuffle and prefetch are unchanged by construction. The one genuine code change is that the queue can today only be replaced: "play next" and "add to queue" require it to become appendable, which makes it a small reducer rather than three pieces of component state. Everything else is a context menu on a cover or a track row, and one button in Music's sticky toolbar. Because Music is mounted by both the group page and the Search page (§9.11), a playlist built inside a group is managed from the consolidated view with no second surface and no application-registry entry.
Unavailability is answered at play time, not at add time. Whether a group is reachable is only knowable by dialing, and refusing to add a track because its node is off tonight loses the user's intent permanently to a condition that lasts an evening. So adding never dials; playback skips, distinguishing a file that will not decode (a property of that file) from a group that does not answer (a property of that group, whose tracks are then skipped together).
It adds no dialing and no new streaming path. Sync rides connections the client already makes, and playback is unchanged (§9.8).
9.11 Cross-group search and source merging
The Search page mounts the same media applications across every group the reader belongs to. The difference lives entirely on the entries, in fields the group page never sets: which group serves an entry, that group's transport and key, a connection generation to use as a refetch key, and the merged source list.
A file shared by two groups is one entry, not two. Identity is the content hash: two entries with the same id are the same file, whatever group announced them and whatever their path. Within one group this cannot arise, because the index is already keyed by hash; the duplication is created by concatenating N independently keyed indexes, and by nothing else.
One source is chosen per logical unit — a film, a show, an album, a photo album — not per file, so a show's episodes never stream from two different nodes. A group hosted by the local node wins; otherwise the pick is deterministic and pseudo-random, stable for one reader and spread across readers. If the chosen source is unreachable the unit fails over. Which source was picked is never shown; the badge names the group when there is one source and counts them when there are more.
Two rules for a new application here:
- Use the shared source tag rather than an entry's group name — a merged entry has several groups. Pass it the whole unit, not the entry the card was drawn from, which is usually chosen for its thumbnail and would under-report.
- Never re-derive unit keys. Call the application's own exported grouping function. A copy keeps agreeing until one of them changes, and the symptom is a show whose episodes stream from two different nodes.
A group's index is never kept in browser storage. The browser holds keys and playlists; it does not hold a copy of what a group contains. Such a cache existed, for a cross-group search that read it instead of dialling, and it outlived that search by weeks — writing a cleartext file listing that nothing read and no sign-out removed. Its only remaining use would be to draw a group's files while its node is unreachable, and that is refused on its own merits: a listing that cannot be opened is worse than an honest absence.
A group whose node is down is the normal case, and nothing waits for it. A node is a machine in somebody's house, so with a handful of groups one of them is always off. Each index is drawn the moment it arrives rather than when its neighbours finish; the limit on how many groups are dialled at once is a ceiling on concurrency and never a batch, because a batch waits for its slowest member and a silent node is always the slowest there is; and a group that did not answer last time is dialled last, remembered per browser, because a ceiling alone still lets silent nodes hold every place at once. The ordering is advisory — storage that refuses, a private window or a first visit all fall back to the hub's own order, which costs that one sweep and then corrects itself.
Search reaches each group with one connection, through one pool. The sweep,
the warm-up and a tile asking for its group all go through ConnectionPool,
which negotiates at most six at once (MAX_IN_FLIGHT) and keeps what the sweep
opened for the tiles, up to its size; a group past that connects when a tile of
it scrolls into view. So a visit costs one offer per group, a refresh costs none
for a connection that answers a short ping, and a connection whose index is being
read is never the one evicted. Separate ceilings for each of these add up behind
each other's backs, past what the hub admits per account (§7.2).
Search dials many nodes at once, so its deadline measures stalling rather than elapsed time. Opening one group from the sidebar waits out the transport's own budgets; in Search a dead group holds one of the pool's places for the deadline — so the deadline is what a page of unreachable groups costs in spinner, and raising it to accommodate a slow link taxes every dead one. A hub that refuses an offer for load is answering, and each retry counts as progress. A node that is not there reports no progress and fails in one window. A node that answers ICE, then opens a channel, buys another window at each step, to a ceiling, because a deadline that only ever resets has none. A flat budget cannot tell a slow link from a node that is not answering, and set low enough for the fan-out it cuts off connections that are still arriving: the same group then fails in Search and opens from the sidebar, against the same node.
The Files explorer is deliberately not merged, and not "mostly not": there each group is a top-level folder, and merging would remove a file from one of them. A test refuses a build that changes this.
A group can be left out of Search — search_listed, a per-group setting on the
node, signed by the operator and carried in the sealed ack. Search reads it after the
handshake and stops there: no index is asked for, cached or merged, in any of the four
views. The case it exists for is a family album that should not turn up in the middle
of a film library.
It is a listing preference and it protects nothing, against anyone. The node cannot tell Search's request from the group page's and serves the same index to both; every member lists the whole group by opening it; a client that ignores the flag lists the group in Search too. It must never be described as "private" or "confidential". What it costs is one handshake per unlisted group, because only the node knows the setting — a hub-side flag would save that and put group state on the hub (§1.3).
One consequence is deliberate and is not a bug: albums are keyed by directory, so two groups whose roots have different basenames put the same photo into two differently-named albums, and the merge — scoped to a unit — leaves it in both. They are two albums.
10. Filesystem portability
exFAT and NTFS on Windows are the common case, not an edge case. Most users are expected to share from an external drive. The consequences below are correctness requirements, not compatibility notes.
| Property | What has to be true |
|---|---|
| Case-insensitive, case-preserving | The index needs a canonical identity and a case-folding collision check at scan time, reported to the operator rather than resolved silently. One directory indexed as two roots is the same problem one level up |
| Unicode normalization | A name written on one platform in decomposed form and on another in composed form are different byte strings. Normalize to NFC for identity and comparison; preserve the original bytes for display and for opening the file |
| Reserved names and characters | A group indexed on one platform can hold names another cannot create. The client sanitises on save and says so; the upload allowlist is the intersection across platforms, or some files are simply undownloadable |
| Path length limits | Use extended-length paths on Windows, in the node and the client alike |
| Coarse timestamps, local time | mtime alone is not a change detector. Size plus mtime with tolerance, and rehash when in doubt |
| Watcher reliability | Change notification drops events under load on Windows, and inotify is unreliable on a FUSE mount. Periodic reconciliation is mandatory on both platforms |
| No symlinks, no POSIX permissions | Simplifications: nothing to defend against, and the node runs as the user anyway |
Case folding is for comparisons the code makes itself — index identity,
collision reporting, root names, nesting checks. It is not needed for the
no-overwrite rule, where the filesystem's own case-insensitive stat() already
gives a colliding upload a free name.
A volume that disappears freezes its root's subtree and never empties it (§6.2). Emptying propagates deletions for a whole library as though the owner had erased it.
Never assume POSIX, systemd, or case sensitivity. One platform-specific trap with no counterpart elsewhere: a service unit with filesystem-protection options gets its own mount namespace, so a volume mounted on the host after the service started is invisible inside it — the directory reads as empty with everything else configured correctly.
11. Platforms
11.1 NAT traversal, measured
QUIC native path — residential ISP A to a hosted VPS: port-restricted cone NAT, direct connection established from the server socket's own probe.
WebRTC browser path — mobile and laptop clients to nodes behind two different residential ISPs:
| Path | Result |
|---|---|
| LAN, IPv6 direct | OK, ~100 ms |
| Mobile data, IPv6 inter-network | OK, ~600 ms |
| Mobile data, IPv4 only, STUN hole-punch | OK, ~650 ms |
| Laptop → second ISP's node, IPv6 inter-network | OK, ~7 s |
| Laptop → second ISP's node, IPv4 only, STUN hole-punch | OK, ~6.9 s |
Two ISPs validated, both without TURN. The hub relays under a kilobyte of signaling; the data path is peer to peer.
11.2 Windows
The bulk of the codebase is portable because the portability rules in §10 were treated as correctness from the start. What the port needed is registered as W1–W9 (§13.7) and is done; packaging is built and awaits a clean-machine run.
Two Windows-specific design points worth stating here:
- Autostart has two modes, chosen at install and switchable afterwards from the Node page: a per-user startup launcher (the default) and a scheduled-task service mode. A per-user default is right for the desktop persona; a service is what a machine that must serve while nobody is logged in needs.
- A user service unit cannot carry a system unit's user directive. Two unit templates exist, held apart by a test that parses directives rather than searching the file — searching matched the comment explaining why the directive is absent.
The hub stays Linux. macOS is not planned. Windows on ARM and store packaging are out of scope.
11.3 Android
A client, not a host. The platform is hostile to hosting a node — background execution, storage, battery — and fine as a client, which is one of the reasons enrichment is node-side (§6.5).
11.4 Casting
An HTTP relay in the desktop client serves a standard fragmented-MP4 stream that any LAN renderer can play; the relay is device-agnostic. Chromecast discovery and control ship. DLNA/UPnP is designed and not built: it is a second device backend beside the first, not a second relay.
Subtitles are rebased onto the relay's clock before they are sent. The node
extracts a track whole, so its cues carry the film's timeline, and the player
can use them unchanged because its SourceBuffer is given timestampOffset =
start. The relay has no such offset: it forwards the node's fragments, which
begin at zero at the seek point. A receiver is therefore sent the cues shifted
by -start, recomputed at every restart of the relay, and cues that end before
the stream begins are dropped rather than clamped to zero.
The subtitle is served from the relay's own port, behind the same token as the
stream, and with CORS. A receiver fetches a side-loaded track with XHR from
its own origin rather than handing it to a media element, so without
Access-Control-Allow-Origin it fails as a network error and the film plays on
with no subtitles and no message. The stream carries the same headers, because a
receiver given a side-loaded track reads the media through the same checked
path: on one and not the other, the load fails whole. They widen nothing the
token does not already govern. The subtitle URL carries a version because a
receiver caches a track by address: changing the cues behind a fixed URL leaves
the previous language on screen.
12. Testing posture
Security tests are negative assertions — "this attack does not work" — and are verified to fail against the pre-fix source before being trusted. A suite that tests only that features work will happily pin a vulnerability in place as expected behaviour, and a refactor that accidentally fixed one would be reported as a regression. That has happened here, to four findings at once.
Rules that follow from what has actually escaped this suite:
- A syntax check validates names not at all, and a module syntax check must force the module parser or it accepts template syntax pasted into an object literal and reports success.
- A second implementation of the client proves the protocol and nothing about the client. An end-to-end harness written in the right order by construction cannot see ordering or lifecycle faults. Source-reading tests are weak evidence and are sometimes the only evidence available; prefer ones that re-derive a value from the source over ones that restate it.
- A test that models a fix agrees with it by construction. Lift the real functions out of the source as text and execute them; model the environment, never the code under test.
- A test that reads source and inspects "the first" occurrence of a call stops guarding anything the moment a new call is inserted before it — and keeps passing.
- Measure, do not read. A stylesheet does not tell you where anything lands; a browser measuring the real stylesheet does. Assert on geometry, and check the test fails with the fix removed.
- A fixture narrower than real data tests the fixture. Names are the one thing a file browser cannot be given short.
- Pace a probe like the real thing. A stress probe fast enough to finish inside a timeout hides every time-based fault.
- "It works now" is not evidence against a race. Force the worst case.
- Exercise every browser's branch of a shared path before shipping a fix for one of them, and finish with a live pass: launching the real thing finds what source reading cannot.
- A passing suite over code nothing calls is evidence about that code, never about the product. Two full protocol implementations sat green and unreferenced for months (§13.3 L7). Before trusting a suite, check that production imports what it tests.
- A comment that contradicts the constant beside it is worse than no comment — one of them is wrong and the reader cannot tell which. The same holds for a document: a number restated away from its definition is a number that will drift.
13. Register of labelled findings and decisions
Each entry states the rule the label names today. Where the label originally named a defect, the subject is given in one clause so a code comment citing it can be understood, not so the incident can be retold.
13.1 First review (design review)
| Label | The rule it names |
|---|---|
| C1 | Group chat has no shared mutable sending state. A pairwise ratchet shared across a group produces key and nonce reuse; the design that shipped has no sending state at all, which is stronger than partitioning it per device (§4.5) |
| C2 | Tokens carry a groups claim and the node verifies membership before serving content (§5.2) |
| S1 | Every admin endpoint has an authorization check |
| S2 | Email addresses are encrypted at rest, with a blind index for lookup |
| S3 | Token revocation reaches nodes over the hub socket, and the denylist persists (§7.5) |
| S4 | AEAD nonces are 96-bit, per NIST SP 800-38D |
| S5 | Refresh tokens rotate, one-time-use, with family-based reuse detection |
| M1 (first review) | Account enumeration is a known, bounded property of the account-management endpoints; login, device auth and reset-request are uniform |
| M2 (first review) | Node TLS certificates are transport confidentiality only; identity is the Ed25519 key checked at the MNP layer |
| M3 (first review) | Rate limits on key-adjacent endpoints |
| M4 (first review) | Delegation is designed and deferred; the role check is written so it drops in (§3.4) |
| M5 | Chunk key derivation uses info, with salt=None — correct HKDF usage, because the group key is already uniform CSPRNG output (§4.3) |
| M6 (first review) | Argon2id production parameters are applied, and recorded per envelope so they can be raised (§4.6) |
| N1–N5 | Notes, no action: the forward-secrecy model is appropriate to the deployment; login error messages are correct; the hub's legal exposure model is well-positioned; the NAT probe payload is fine; the web/CLI derivation difference is by design |
13.2 Node sovereignty
| Label | The rule it names |
|---|---|
| NS1 | The client proves possession of the group key in the handshake, and the node verifies it |
| NS2 | Admin operations are Ed25519 challenge-response over a structured transcript, never a token (§5.4) |
| NS3 | The node never serves the group key in plaintext. There is no request message for it; the constants are gone from the protocol |
| NS4 | Operator authority comes from the node's roster and from nowhere else. No auto-pin from the keystore, no resolution through the hub, no config key — a config naming one is warned about and never obeyed (§3.4, §6.1) |
| NS5 | The proof is bound to the transport channel (DTLS fingerprints / certificate hash), so a signaling relay that substitutes its own cannot produce it (§5.2) |
| NS6 | sender_id is enforced from the authenticated session, never the wire. It is what the store keys on; it is not what authenticates a message — the device signature is (§4.5) |
| NS7 | The node authenticates to the hub with Ed25519 and no password, and its token's scope is refused for group management (§7.2) |
| NS8 | The node refuses connections when it holds no group key. There is no bypass switch |
13.3 Second review (code review) — the default numbering
Critical
| Label | The rule it names |
|---|---|
| C1 | All content travels over the authenticated protocol. The node exposes no unauthenticated HTTP surface; the per-group file API that served private indexes and plaintext files on all interfaces was deleted rather than repaired, because it duplicated MNP without any of its controls |
| C2 | A node's signaling identity is resolved against the database and derived from it, never taken from the client's first message (§7.2) |
| C3 | Authentication is mutual: the node proves key possession over the client's nonce and signs the transcript, and the client verifies both and pins the key (§5.2) |
| C4 | Keypair bundles are per node, Argon2id-protected, and closed for native devices — and open for any account that also uses a browser (§3.7) |
| C5 | The two halves below, cited together where a comment means "a member must not be able to write what the node then trusts" |
| C5a | Uploads cannot overwrite, are allowlisted, ordered and capped, and ownership is signed by the uploader (§6.4) |
| C5b | No key material arrives from outside. The member-supplied bundle message does not exist; the node generates every copy of a group key itself (§4.2) |
| C6 | One handshake implementation, shared by every transport. A transport that accepts a bare token is the standing example of what an opt-in compatibility branch costs (§5.2, §5.6) |
High
| Label | The rule it names |
|---|---|
| H1 | Per-group isolation on a multi-group node: the chat store, the peer registry and the broadcast set are per group |
| H2 | Every value that originates outside the node — filenames chosen by members, usernames originating at the hub — is escaped where it is rendered. A CSP contains exfiltration but cannot prevent injected inline script, so escaping is the actual fix |
| H3 | No public key is ever fetched from a directory to wrap a group key for. The node wraps for a key the recipient proved possession of, bound to an account by a code the hub never sees (§3.4) |
| H4 | Revocation reaches nodes, drops live sessions, and persists across a restart (§7.5) |
| H5 | An admin challenge is a structured, domain-separated transcript naming the operation and subject, and the client refuses to sign anything that is not what the user asked for (§5.4) |
| H6 | Unauthenticated work a node will do is bounded: a small pre-handshake buffer, a transcode semaphore, per-user pending-offer caps, and a membership check on signaling (§7.2) |
| H7 | Only public groups register content hashes with the hub. Private groups register nothing, and the swarm route requires authentication |
Medium
| Label | The rule it names |
|---|---|
| M1 | group_id is mandatory on the handshake — there is no fallback to the node's first group (§5.2) |
| M2 | The node keystore uses the full Argon2id parameters, recorded per envelope (§4.6) |
| M3 | Operator authority is the roster pin, established locally by pairing. Asking the hub for the operator's key — the obvious-looking fix — would let the hub install itself as node administrator (§3.4) |
| M4 | Every reply carries the request id it answers. Arrival-order matching is a guess that fails silently and asymmetrically (§5.3) |
| M5 | The index has one protection level, not one per transport (§4.4) |
| M6 | Audit rows are attributed to the row's own subject, never backfilled across rows |
| M7 | Client addresses are taken from a trusted proxy's rightmost hop, in one helper (§7.2) |
| M8 | A node announcement requires proof of possession of the key it announces |
| M9 | Node-scoped tokens are refused on the client path (§5.2) |
Low
| Label | The rule it names |
|---|---|
| L1 | The wire contract carries no constants for messages that do not exist |
| L2 | MNP negotiates versions explicitly and refuses with a code (§5.6) |
| L3 | Errors returned to a peer name no filesystem path and no exception detail |
| L4 | Transcripts are length-prefixed, and an empty channel binding raises rather than degrading the proof to nonce-only (§5.2) |
| L5 | The hub serves a CSP and security headers on the application it serves |
| L6 | Registration validates the email field it declares |
| L7 | A module nothing imports is not a protection, and it is not kept. The sender-key and ratchet implementations this finding named were unreferenced for months and are now deleted; a green test suite over uncalled code is evidence about that code, never about the product (§4.5) |
| L8 | An uploader record identifies a file by its id, not by a name at a root |
13.4 Third review
| Label | The rule it names |
|---|---|
| H1 (third review) | Moderator is not administrator. The user-patch handler is split by field so a moderator cannot write role (§7.5) |
| H2 (third review) | Content reporting requires authentication, distinct reporters and a rate limit, and is refused when public groups are off (§7.5) |
| M1 (third review) | The registration CAPTCHA gate is unconditional when a captcha is configured — never conditioned on a field the real client always sends (§7.7) |
| M2a | sender_id comes from the authenticated session (NS6). Now guaranteed by there being one chat implementation: QUIC does not carry chat at all (§5.1) |
| M2b | Chat broadcast is per group (H1), on the one transport that carries chat |
| M2c | No transport runs a synchronous media process on the event loop, and every one is capped |
| M3 (third review) | Link-preview SSRF is gated: rate limit, port allowlist, globally-routable check, per-hop re-check, connect-address re-check, size guard (§6.5) |
| M4 (third review) | Federation binds a pushed row's source to the signer, checks the token audience, caps the push, rejects replays, and scopes revocation to the peer's own entries (§7.6) |
| M5 (third review) | A CSP and security headers apply to the hub-served application, verified against the running app — a mis-tuned CSP shows as a blank page |
| M6 (third review) | Withdrawn. It misread the node registering a hub membership during the CLI invite flow — which is deliberate — as authorization drift |
| L1–L11 (third review) | Opportunistic hardening: relay-registry proof of possession; delete orphaned modules rather than leaving them to be rewired; decide and document account enumeration; an aggregate upload quota; header-only control-API tokens; a freshness bound on revocation replay; state that the exact-hash content check is structural; validate group-name length and charset; require exp and bind an audience on token decode; key the rate limiter through the same client-address helper as everything else; keep diagnostic logging truncated |
Two structural recommendations from that review stand as rules:
- Make transport parity a test, not a habit. Shared helpers in common code with a test that fails if a transport calls a chat or stream path that bypasses them.
- Every new outbound or cross-trust surface needs a rate limit and a named adversary in the same commit (§6.5).
13.5 Standing trust limits
| Label | The rule it names |
|---|---|
| T1 | The password split. The hub never sees a passphrase; it holds a verifier for a client-derived auth_key. The passphrase floor can therefore only be enforced client-side (§3.1) |
| T2 | The hub was the key directory. Closed by admission redesign, not by safety numbers: the invite path reads no directory at all (§3.4). Reclassified as H3 |
| T3 | The hub serves the SPA. Accepted permanently for browser users. It is the only remaining way an active hub reads content, it is an artifact-level attack rather than a silent lie, and it does not exist for a native client — whose value is realised by reproducible builds, not by packaging (§2.3, §8.2) |
13.5b Availability between members (AV)
Added 2026-09-12, after the finding that produced it, and extended the same
day when users.py and admin.py were read under it. The first three
reviews asked who can read what, who can impersonate whom, and what a hostile
node can forge. None asked what a legitimate but misconfigured or careless
member costs everyone else — and that is the question a group platform lives
on, because every member was invited by someone who trusted them and none of
them is an attacker. C2 had asked "can a node claim a group its owner is not
in?" and the answer was correctly no; nobody had asked what happens when a node
claims one its owner is in but which it does not host, which is how a group
went dark for all of its members on 2026-09-11 with its real host online
throughout.
The lens, for anything reviewed from here: a participant supplies input; who else bears the cost? Where the answer is "someone other than the sender", there must be a ceiling, and it must apply on every path that writes the state — not only the one where the ceiling was first thought of.
Two shapes recur, and are worth naming because each accounts for several
entries. A limit written on one of several equivalent paths — three
endpoints send mail and one of them had neither a rate limit nor a captcha;
every list in admin.py is bounded and the two outside it were not; the group
claim was bounded at registration and not on reload. And a bound that counts
the wrong thing — a per-IP rate limit bounds a caller, never the mailbox that
receives what they cause, which is why AV10 is a cooldown per account under
a limit per IP rather than a tighter limit.
Open for decision, not a defect: moderation.AUTO_BLOCK_THRESHOLD is 3.
Three distinct accounts blocking a hash adds it to the list every node
enforces, network-wide, automatically, with manual admin removal the only
undo. That is already far better than the anonymous version it replaced, and
it is still a censorship primitive an attacker buys for the price of three
email addresses. Raising it buys little; requiring the reporting accounts to
be more than a day old would cost a patient attacker a day and cost an honest
reporter nothing after their first. Left as it is because it is a moderation
policy rather than a bug, and the person who sets that policy is the operator.
admin.py was read under this lens and needed nothing. Its moderator/admin
line is drawn explicitly — a moderator may not change a role, may not revoke,
and may not touch an admin's account at all — self-modification is refused, and
every list it serves is bounded. It is the part of the hub where the question
had already been asked.
| Label | The rule it names |
|---|---|
| AV1 | An empty claim is a claim on nothing. A node's group set is authorized ∩ claimed, and an absent or empty group_ids registers it for no group rather than all of its owner's — on registration and on update_groups alike (§7.2) |
| AV2 | A client treats the hub's node list as candidates, not a ranking, and tries the next one on a not_hosted refusal (MESHBAY_NODE_PROTOCOL.md §6.3) |
| AV3 | A node speaks only for the groups it is registered for. chat_notify names a group and is checked against that node's set before a notification is written for anyone, and it is rate-limited per node — the fan-out is one write per member |
| AV4 | Nobody names a third party's address. A swarm source publishes a transport and a port, never a host; where a peer is comes from its node record, stamped with the address its announce arrived from. The number of hashes one account may claim is bounded |
| AV5 | An answer is accepted only from the node the offer was sent to. A peer_id is bound to its node, so no connected node can resolve another's pending offer |
| AV6 | A relay proves possession of its approved key. A public key is not a password, and the register call is unauthenticated by design — it is not a user — so the proof is the only thing standing between a stranger and where nodes send relayed traffic |
| AV7 | A node bounds how many peers it holds and how long an unproven one lasts. The hub's cap is per calling account, which is a limit on each member and not on the machine, so without this an operator's exposure grew with the size of their groups |
| AV8 | One account cannot make the hub mail another at will. The invitation email's subject comes from the group row, never from the request, and the endpoint is metered |
| AV9 | No mail is sent from the event loop. smtplib is synchronous and waits up to ten seconds; called from an async handler that wait is the whole instance's, not one request's. Every send goes through mail.send_off_loop. Argon2 is held to the same rule: every derivation runs on one dedicated worker thread (auth.*_off_loop), never on the loop and never two at a time, because two concurrent lanes=4 derivations deadlock in OpenSSL. So is the node's disk: every filesystem call on a group's content — the stat as much as the read, since a stat is what wakes a sleeping disk — goes through roots.off_disk, onto one worker thread per root set. A spun-down or network-mounted root answers its first syscall in seconds, and on the loop that is every group, every stream and the hub socket waiting for a platter. ffmpeg's own output too, through asyncio.to_thread rather than that per-root thread: a temp file is not a group root and has no platter to serialise against, but a whole transcode read inline is still tens of megabytes of blocking read |
| AV10 | Every path that makes the hub send mail is metered, per account. A rate limit that counts by IP bounds a caller, not an inbox. Changing one's address mails an arbitrary stranger, so it carries a cooldown and a daily ceiling; a reset request and a registration resend carry cooldowns |
| AV13 | The mail server is not a relay, and mail.py is where that is decided. Every message passes one function; purpose is keyword-required and checked against a closed list, so a helper that names anything else does not send and one that names nothing is a TypeError. Under it sit a bound per recipient — the thing a person being mail-bombed actually experiences, unmoved by which account, address or endpoint asks — and an instance-wide hourly ceiling, because registration is open and "per account" is a bound an attacker buys more of. The recipient's daily cap counts every purpose; its cooldown is per family, invitations on one side and the account's own steps on the other, because one cooldown for both refused — silently — the sign-up code of everyone who registered within two minutes of receiving an invitation link |
| AV11 | A namespace a client writes into is closed, and its rows are capped. The preference key space is an allow-list plus default_tab:<group_id> checked as a group id, the value is length-bounded, and the row count per account is bounded |
| AV12 | Every list has an upper bound on limit and a floor under offset. Including the ones that take no authentication at all — the public group directory and the content blocklist |
| AV16 | A bound the operator can see and change, and that a restart does not forget. The mail allowance lives in mail_quota, not in a module dict — a deploy used to hand out a fresh budget, and the hub is deployed often. The values are settings with defaults in hub.toml and a block in the admin panel, because the hour a budget runs out is not when anyone wants to edit a file and restart; /v1/admin/mail says how much of the hour is left, which was previously visible only as an absence of mail |
| AV17 | A global ceiling being reached is an event, not an absence. When the hourly budget runs out the administrators are notified — once per hour, because a flood is what spends it and one alert per refusal buries the message under its own cause — and the panel says which of the two ceilings fell: newcomers turned away, or somebody locked out of their account unable to get back in |
| AV19 | Nothing carries the path to the migrations. meshbay-hub migrate derives it from the installed package, so the RPM, the DEB, a venv and a checkout all agree. A unit naming alembic.ini names a file whose %(here)s stops being true the moment packaging moves it |
| AV18 | The hub runs on exactly one worker, and says so at startup. _connected_nodes, _node_groups, _webrtc_answers and the relay registry are per-process: a second worker makes a node intermittently unreachable for half its members, which is a symptom that describes something else entirely |
| AV14 | MHP binds its audience, and the hub reads its own identity at call time. A token is minted for one peer and accepted by that peer only. federation.py bound _hub_id and _hub_sk_pem at import, which is before load_hub_keypair runs, so it signed with None and called itself meshbay.org whatever the instance was named — and the verifier named no audience for the aud the issuer sets, which PyJWT refuses outright. MHP could not complete one authenticated request between two hubs |
| AV15 | A hash is checked for shape before it is a key lookup, on the unauthenticated blocklist endpoints a node consults |
| AV20 | Chat is bounded in size and in rate, like every other member-supplied write (§6.6). A message is a row on the operator's disk that nothing expires, a relayed copy for every connected member and a notification for every member of the group; the only ceiling was the frame size. Uploads had carried four protections and a cap since C5a because somebody asked what one member costs the others on that path, and nobody had asked it on this one |
| AV21 | A lease is what the node granted, not what the client called it (§5.5). tr was read as a boolean, so any non-empty string skipped the leaseless ceiling and every cap behind it, and a queued transfer was held back only by the honesty of the client waiting in the queue |
| AV22 | The node's own controls take no authority from a hub token (§6.7). node_status, node_settings_set, roster_read, denylist_read, denylist_clear and node_reload were gated on the account id in the JWT, which is the hub's to choose — NS4 and M3 with the check written the other way round. The gate is a proved operator device, which a hub holding no user keys cannot produce |
| AV23 | An upload's owner is recorded when the upload ends and applied when the entry is created, which are different moments (§5.4). Written against the index at the end of the upload it matched nothing, every time, and left every uploaded file owned by nobody — so no member could delete what they had sent |
| AV24 | A node registered for no group is refused signaling, not exempted from it (§7.2). The membership check was written as "if the node claims any group", so it skipped itself — membership, group status and the public-group gate together — for the node AV1 made commonplace: the unconfigured one, which is also the one least able to absorb the work |
| AV25 | Which nodes host a group is answered to its members (§7.3). Only the public case checked, so a private group told any authenticated account that knew its id which machines hosted it — and an ex-member knows that id for ever |
| AV26 | A sign-in lockout refuses passphrase sign-in and nothing else (§7.7). It is keyed by username, usernames are public, and so anyone can spend somebody else's attempts. Open sessions, renewal and device sign-in are untouched and a reset code ends it, which bounds what a stranger buys to one forced sign-in. The lockout is a DoS primitive by construction; this is the ceiling on it |
| AV27 | A free-text third-party search is bounded per member and per node (§6.5). tmdb_search_req spends the operator's credential, which TMDB rates and the whole group's automatic matching depends on, so one member holding a search box degrades the library for everyone. Per member and not per connection — three tabs is one person — and kept in the group context so a reconnect does not reset it. The refusal is an error, because an empty result list is what "no such film" looks like |
| AV29 | An invitation link is bounded on both halves and its mail on the sender (§3.4, §7.3). Twenty outstanding per group on the node (bearer codes) and on the hub (tickets); and because a link mail reaches an address the hub has no relationship with, at the request of anyone who owns a group, it is counted per sending account per day (mail.invite_link_daily_cap, 10), under the recipient and instance bounds and outside the recovery reserve (invite_link is not a recovery purpose) |
| AV28 | How many node keys one account may announce is bounded (§7.2). Each is a row plus an IP-log row under a one-year retention, so an account in a loop writes a year of storage on the operator's disk having paid only for signatures. Proof of possession (M8) settles whose key it is and not how many. Counted only where a row is added: re-announcing a key already held keeps working at the ceiling, or a node that reached it could never refresh its address again |
| AV30 | What one member's offers cost a node is bounded per account and per node, and the bound admits the heaviest ordinary account (§7.2). Each offer makes the node allocate a peer connection. A budget of 120 per node refilled at two a second bounds a member there without touching their other nodes, and it is counted by account because a mobile carrier shares one IPv4 address among many subscribers. Pending offers are capped at 32 per account. Both refusals carry Retry-After and the client retries them, because a refused offer otherwise reads as a node that is down |
13.6 Chat design findings
| Label | The rule it names |
|---|---|
| F1 | A sender's signing key is bound to the roster, never generated fresh inside a distribution any member can produce. Otherwise every member can replace another's chain and forge them silently |
| F2 | A second device cannot destroy the first's ability to send or be read — there is no per-device chain to drop (§4.5) |
| F3 | There is no skipped-key cache to grow without bound |
| F4 | Rotating the group key is a re-wrap, not the destruction of the archive. Epoch keys are wrapped at delivery, never at rest (§4.5) |
| F5 | History is served to devices that were not present, which is what "load older" means and is incompatible with a ratchet |
| F6 | Ciphertext is not corrupted on the history path |
| F7 | One account can hold two connected devices: the peer registry is keyed per connection, and a broadcast excludes the sending session, not the sending account |
| R1–R22 | The regression register for that work — each row a concrete way the feature could break something that already worked, with the guard that stops it. R1 (rotation keeps history readable), R5 (an older client is refused with a stated reason, not left showing gibberish), R11 (own-ness from the account id), R15 (no epoch key in a plaintext store), R16 (retention deletes messages, never epoch keys) are the ones cited elsewhere |
13.7 Windows port
| Label | The rule it names |
|---|---|
| W1 | Platform directories: no hardcoded XDG paths |
| W2 | Signal handling is platform-guarded |
| W3 | Daemon lifecycle: a per-user startup launcher by default, a scheduled-task service mode offered, switchable after install (§11.2) |
| W4 | Packaging: one per-user installer carrying client and node, with media tools bundled |
| W5 | File permission calls are skipped where they have no meaning |
| W6 | Media-tool discovery fails at startup with a stated reason rather than at first use |
| W7 | CLI messages name the right paths and commands for the platform |
| W8 | The test suite is green on the platform, encoding included |
| W9 | ICE interface matching works where an adapter's name is a GUID rather than a kernel name, and fails open |
13.8 Desktop client decisions
| Label | The decision |
|---|---|
| E1 | Electron, plus an optional Python sidecar for hub-less group:// over QUIC |
| E2 | Device linking: an already-pinned key countersigns, bound by a one-time code the new device generates (§3.3) |
| E3 | Hybrid account creation: register with a passphrase-derived auth_key — the recovery path — then authenticate day to day with a device key |
| E4 | Signed admin ops over MNP for everything group-scoped; first run stays local; the loopback API is never exposed to the network |
| E5 | LAN enrolment of a headless node is out of scope for v1, kept implementable |
| E6 | The browser SPA stays. A native client must not prevent web use |
| E7 | Several named roots per group, forming one virtual root (§6.2) |
| E8 | exFAT/NTFS and Windows are the common case. Linux ships first; that is build order, not population (§10) |
| E9 | Group-related server state lives on the node. Always (§1.3) |
| O1 | Initial key setup in the pre-proof window — deferred; that window is where C4 and C5b came from |
| O2 | A LAN enrolment door — one endpoint, bounded window, one-time code, closing permanently on success |
| O3 | device_policy {allow_bundle: false}, signed by a pinned key — the mechanism that actually closes C4 (§3.7) |
| O4 | Isolating the node-admin panel from the process holding user keys |
| O5 | An unlock key in the environment, for the node |
| O6 | The engine version floor, verified rather than assumed |
| O8 | A minimum client version in the hub version endpoint — done (§5.6) |
| O10 | Canonical file identity across filesystems, defined once and shared (§10) |
| O11 | A root alias where the basename cannot be used (§6.2) |
| O12 | Derived data: revised — the node caches durably in its own data_dir (§6.5) |
| O13 | Hub identity pinning. The client points at a hub by URL and nothing pins that hub's identity. Bounded, because a substituted hub can neither read content nor ship the code to a native client — worth doing all the same |
| V1–V13, P1–P5 | Per-application open items: wording of a disabled-service state, whether artwork reuses the chunk path, cache TTL, multi-track surfacing, HEIC/RAW support, a fuller EXIF panel, lightbox preloading, album-boundary behaviour, cover selection |
14. Decisions that are not revisited
14.1 Structural
- Multi-group on a single port.
- Signaling punch and connect through the hub socket.
- Chat is a core feature, not a module.
- Chat encryption is a sealed archive with per-device keys and signed messages — not sender keys, not a ratchet (§4.5).
- Tokens carry group claims and the node checks them.
- Admin roles are hub configuration.
- Refresh-token rotation is family-based.
- Email is encrypted at rest.
- Argon2id parameters are versioned and migrate on login.
- Browser transport is WebRTC DataChannel with ICE/STUN.
- The hub is a registrar and signaling relay, never in the data path.
- Chat is stored on nodes, never on the hub.
- The web UI is a small-framework SPA: dark/light, responsive, i18n in ten languages.
- Site-specific pages are an overlay, separate from the generic hub.
- Video streaming is fragmented-MP4 remux on the node, source buffer in the client.
- ICE is primary for browser and native alike. QUIC is kept at parity for LAN, port-forwarded and hub-less access. TCP+TLS and the node HTTP API do not exist.
punch_nat()is a direct-connection helper, not a traversal stack (§5.1).- The desktop shell is Electron. What is unchanged and non-negotiable: UI assets ship inside the package and load from disk (§8.2).
- A second device is admitted by device linking, not by an operator code (§3.3).
- Private keys never leave the device on native clients. Qualified: a browser has no durable storage of its own and still needs a bundle on each node, so C4 closes for an account only when it opts out of browser use.
- Hub minimisation is enforced by an acceptance test, not by policy. The hub must be unable to see keys, content or file listings.
- No new code exchanges between people. Safety numbers are refused for identity verification, permanently. The device-linking code is between a person's own devices and is unaffected. The total user-visible cost of the whole authorship story is one notice: "this account's key changed".
14.2 Client architecture
| # | Decision |
|---|---|
| D1 | The hub keeps serving the web UI. It is the zero-install path and it stays. What must then be true: a strict CSP, a signed digest of the served bundle that any third party can verify, an explicit reduced-trust notice, and docs that never claim end-to-end integrity for that path |
| D2 | A native client, offered alongside the SPA — not as a replacement. The browser-extension options (an extension that verifies the served bundle; an extension that ships the UI) were analysed and are not taken up |
| D3 | Transport: ICE primary, QUIC at parity, TCP and HTTP removed |
| D4 | Hub minimisation deferred, and may be dropped. The hub stays in the trusted path by choice |
The reason a native client is justified, and the reason it is not justified. It is justified on product grounds: durable keys, no browser tab, background connectivity, better video, hub-less access. It is not the fix for T3 unless reproducible builds ship with it — a binary from the same operator relocates trust rather than removing it. What genuinely changes is detectability: a browser attack is one HTTP response aimed at one user, leaving no artifact; a native attack requires shipping a build, which is hashable, archivable and comparable.
Its real costs were under-weighted once and are recorded: patch velocity is owned rather than inherited from a browser vendor, and the renderer parses attacker-controlled content from nodes. Accepted deliberately.
For node operators specifically, the CLI beats every client. The operator holds the group key and is the content authority; a CLI removes their dependency on hub-served code at a fraction of any client's cost. If only one thing were built against T3, it should be that — and it was.
15. State of the build
15.1 Built and running
The hub, the node daemon, both transports, the unified handshake, admission and pairing, invitation links, device linking with member-visible evidence, per-node identity, named roots with RO/RW and eject/plug, the indexer with partial hashing, uploads, the sealed index and sealed upload path, encrypted chat with epochs, video streaming with seeking, audio-language and subtitle selection, transfer leases with queueing, pause and resume, the group-application framework with Chat, Files, Videos, Music and Photos, cross-group search with source merging, per-account playlists, casting to a Chromecast with subtitles rebased onto the relay's clock, the operator CLI and loopback control API, the desktop client through its identity and download stages, account recovery, and the Windows port through packaging.
The packages install: a machine has been taken from the built artefacts to a
running hub and node on Ubuntu 26.04 (.deb), Fedora 44 (.rpm) and
Windows 11 (.exe). What is not done is signing them (Stage D11, D12), which
is a different question from whether they install.
Managing a node and setting one up are built, and not where the desktop stages
put them. §8.3 is why: one UI source means a panel written for the desktop
client is a panel the browser does not have, so both live in static/ and the
client ships them. node-page.js is the management surface over the operator
ops — status, groups, roots with eject, plug and add, roster, peers, pairing,
audit export. create-group-page.js is the first-run path: it detects the node,
provisions it, takes the group's name and roots, initialises the key and pairs.
The one part that is genuinely native is the service unit, and the desktop main
process runs it — systemctl --user on Linux, Task Scheduler on Windows.
15.2 Not built
| Stage D9 | Python sidecar — group:// over QUIC |
| Stage D11 | Windows code signing — the installer runs on Windows 11; the binary is unsigned |
| Stage D12 | Release key, signed repositories, updates through the OS |
| — | DLNA/UPnP casting (§11.4) |
| — | Bitmap subtitles (PGS, VOBSUB — about a fifth of the embedded streams). No WebVTT without OCR; they are not listed rather than listed and blank. Burn-in covers them and costs -c:v copy, which is what the eight-slot sizing assumes never happens |
| — | Delegation (§3.4) |
| — | Tier 3 roster attestation (§3.3) |
| — | Android client |
| — | Federation between two hubs. The protocol is written and switched off in the code (§7.6); what is not built is one run between two machines |
15.3 Open, and why each is where it is
| Item | Status |
|---|---|
| C4 for browser-using accounts | Open until the signed bundle opt-out ships (O3) |
| T3 for browser users | Accepted permanently. Removed for native clients, and that removal's value depends on reproducible builds |
| Hub identity pinning (O13) | Nothing pins the hub's key. Bounded, because a substituted hub can neither read content nor ship code to a native client |
| Aggregate upload quota | Per-file caps exist and the operator sets theirs (§6.4); a per-user or per-group total does not |
| A signed upload transcript | Ownership is recorded by the node and verifiable by nobody else (§5.4). Making it provable is a transcript the uploader signs, stored with the entry — designed in outline, not built |
| Forward secrecy in group chat | Given up deliberately and on the record (§4.5). If it becomes a requirement it belongs in 1:1 DM |
| Metadata at the hub | Membership, and who posted in which group and when. A known leak, not a solved problem (§7.1) |
| The exact-hash content check | Structural, not functional (§7.5) |
| QUIC | Off by default, and not at parity: it serves the index and file chunks with no transfer lease, no leaseless ceiling and no root-availability check, does its file I/O on the event loop, and returns exception text to the peer (L3). No client speaks it. Either it comes to parity or it goes; until then §5.1's "chat is the only gap" is the one sentence here that overstates the code |
| The relay registry | Closed in the code: relay.RELAYS_ENABLED is False and every /v1/relays route answers 503, as federation does. Nothing in the tree calls them, node or client, and §11.1 measured two ISPs with no TURN relay needed. Kept code that nothing calls is what L7 says not to keep; it stays only as the proof-of-possession design (AV6) until a node needs a relay or it is deleted |
| A very high bitrate wedges the player against a small buffer ceiling | Where even the floor read-ahead does not fit — ninety seconds plus the minute kept behind, at the file's bitrate, above what the engine will hold — the film stalls: measured on the harness at 9.3 Mbit/s against a 100 MB ceiling, 100.8 s of film played in 900 s of wall clock. Predates the byte budget and is unchanged by it, to the tenth of a second; what the budget did change there is the refusal count, 1560 → 2. The fix is not a bound at all, it is a second stage of buffer outside the SourceBuffer, which means gating the append path — the riskiest change in this area and not one to make alongside another |
The reconnect backoff only wakes on visibilitychange |
So a tab that stays visible through an outage — which is what a screen wake lock guarantees while a film is playing — waits out the full backoff, up to 30 s, after the network is already back. Nothing listens for online |
| Per-device revocation has no CLI | A device is revoked over MNP (roster.revoke_device), from a device the node has already pinned. On a headless node the operator's only lever is member unpin, which removes every device of that account — so the per-device control the roster is built around is reachable from an interface and from nowhere else. §6.7 listed a meshbay-node member device list\|revoke verb that was never written, and that listing is how this was found: USERGUIDE.md was the first document written by reading the CLI rather than this specification, and the verb it copied out did not run |
| Migrations run on SQLite only | The chain reaches head and agrees with the models there (§12), which is not where it ships. The exposure is one revision deep, not the whole chain: every revision behind the first packaged release was development that no installation ever ran, so nothing replays them on PostgreSQL. What is unguarded is the next migration — a default, an index type or a constraint PostgreSQL refuses reaches a deploy without the suite saying so |
| The loopback path removes access without writing an audit entry | ops.revoke_member and ops.unpin_member log to the daemon's log and nothing to audit.db; the MNP admin handlers doing the same work audit member_revoke and member_unpin. So a removal made from the node page or the CLI — the two doors an operator sitting at their own machine actually uses — leaves the journal showing an admission and then, whenever that person next connects, a join_refused with nothing in between to explain it. §5.4's signed transcript is not what is missing: a loopback caller is authorized by being on localhost with the run token and signs nothing, so the gap is the record, not the authority. Found by reading a node's audit log for a refusal whose cause was six hours earlier and unrecorded |
| The Create group wizard calls two hooks after an early return | CreateGroupWizard (create-group-page.js) returns during node detection, before its useRef/useEffect for provisioning, so the hook count changes between renders. Preact tolerates a list that grows, and nothing is known to break; test_hook_ordering.py checks declaration order, not this. Found while tracing the frozen-fields report, which had another cause (ask.js) |
| A node key is read from the terminal or the desktop client, never a browser | Accepted. meshbay-node status on the node's own machine and Node → Overview in the desktop client are the two places the key can be read; the Node page is Electron-only, because platform.node resolves to "not available" without the bridge, and no hub route exposes the key. The create-group wizard links it automatically over that same bridge, so the manual paste in Profile → Link Node exists for the operator who runs the node from a terminal and the hub from a browser — who has a terminal by definition. Anyone linking a node is already at a shell prompt, so a browser-reachable copy would buy nothing and widen what the hub knows about the node |
| Listing a group's folders walks every root on the event loop | index_sync_message (transport/wire.py) builds its dirs field with list_dirs, an rglob("*") over every root, and nothing sends it off the loop: the WebRTC index_sync handler, the daemon's index push and QUIC all call it inline. So each index request from any member is a directory walk of the whole library that every other peer on the node waits behind. test_disk_io_off_loop.py never saw it, because it reads the transport's own modules and the walk is one call away in wire.py. Found by widening what that test reads, not by a symptom |
| The transcoded-seek test passes without transcoding | test_a_transcoded_video_keeps_accurate_seeking (test_stream_seek_audio_alignment.py) forces the re-encode branch by swapping the module's BROWSER_INCOMPATIBLE_VIDEO_CODECS, then checks only that the result has no audio gap. The copy path also leaves no gap on that clip, so pointing the swap at a module the streaming code does not read still passes: the test cannot tell that the branch it is named after never ran. It should assert the re-encode happened (the re-encoding log line, or the encoder in the ffmpeg argv). Found by breaking the swap on purpose while moving the streaming code |
16. Concordance
Older documents, notes outside this repository and every earlier revision in
git cite sections of the documents this one replaces. Those documents are no
longer in the tree — they were removed on 2026-09-11, once their content was here and
this table could resolve every reference they left behind. git log -- docs/
recovers any of them.
The code does not need this table: since 2026-09-19 every comment and test cites
MESHBAY_DESIGN.md and a section directly, and a new one must. What this
resolves is what cannot be rewritten — a commit message, a git blame, a
superseded document kept under docs/, a note somebody holds elsewhere.
| Cited as | Read |
|---|---|
draft-v5 §2, draft-v6 §4 — security claims |
§2.2 |
draft-v5 §3 — transport, NAT traversal |
§5.1 |
draft-v5 §4, §4.1–4.4 — handshake, transcript, channel binding, mutual auth |
§5.2 |
draft-v5 §5.1, draft-v6 §2.3, §2.4b — privileged operations, key activation, authorship |
§5.4 |
draft-v5 §5.2, draft-v6 §2.1, §2.1b — uploads |
§6.4 |
draft-v5 §5.2 — nothing derived beside the originals |
§6.5 |
draft-v5 §5.2b — removing a directory |
§6.4 |
draft-v5 §5.3, §5.4 — operator interface, local admin UI |
§6.7 |
draft-v5 §5.5, invite-pairing-v1.md §3, §5, §7 — admission and key delivery |
§3.4 |
draft-v5 §6.1, draft-v6 §2.5, §2.9 — hub role, group names |
§7.1, §7.3 |
draft-v5 §6.2–§6.4 — node registration, signaling, client addresses |
§7.2 |
draft-v5 §7 — cryptography |
§4.1–§4.4, §4.6 |
draft-v5 §7.1 — the keypair bundle and C4 |
§3.7 |
draft-v5 §8.1, §8.2, desktop-client-v1.md §2 — the two clients, the shell |
§8.1, §8.2 |
draft-v5 §9, draft-v6 §5 — open items |
§15.3 |
draft-v5 §10 — testing posture |
§12 |
draft-v6 §2.2, desktop-client-v1.md §4, §4.5, §4.6 — device linking |
§3.3 |
desktop-client-v1.md §4.8 — authorship, the tiers |
§3.3, §5.4 |
desktop-client-v1.md §5, §5.1, auth-confirm.md — accounts, change and recovery |
§3.1, §3.6 |
desktop-client-v1.md §6.7, refactor-groups.md §1.1, §1.5b — roots, RO/RW, eject |
§6.2 |
desktop-client-v1.md §6.8, draft-v6 §3, WINDOWS-PORT.md — portability |
§10 |
desktop-client-v1.md §6.9 — a root that goes away |
§6.2 |
desktop-client-v1.md §6.10, mediacenter.md §2, §5.2, §5.3 — views not a catalogue, derived data |
§6.5, §9.1 |
desktop-client-v1.md §8.1 — URL space |
§8.4 |
desktop-client-v1.md §10b, refactor-node-ui.md — group settings, the node's admin surface |
§9.3, §6.7 |
draft-v6 §2.7, apps.md §1–§3 — the application framework |
§9.1–§9.3 |
apps.md §2b, refactoring-search.md — cross-group search, source merging |
§9.11 |
playlists.md — per-account cross-group state |
§9.10 |
apps.md §4, refactor-groups.md §3, §4.1 — adding an application |
§9.4 |
draft-v6 §2.8 — instance policy, suspend vs revoke |
§7.4, §7.5 |
draft-v6 §2.10, refactor-groups.md §1.7 — link previews |
§6.5, §9.6 |
draft-v6 §2.11 — node settings |
§6.8 |
draft-v6 §2.12 — STUN and ICE filtering |
§5.1 |
mediacenter.md §3, §4, §5.4–§5.7 — Videos |
§9.7 |
musicbay.md §2, §3, §4, §5, §6 — Music |
§9.8 |
photos.md §2, §3, §4, §5 — Photos |
§9.9 |
refactor-groups.md §1.6 — generic app-directory ops |
§9.3 |
chat-sender-keys.md §5, §6, §8, §13 — chat encryption, sender authentication |
§4.5, §3.3 |
indexing-v2.md — partial-read hashing |
§6.3 |
per-node-identity-v1.md — identity keys per node |
§3.2 |
captcha.md — registration gate |
§7.7 |
tmp-decisions.md D1–D4 |
§14.2 |
first-review.md, second-review.md, third-review.md — any finding |
§13, which defines every label. §13.1 is the first review, §13.3 the second, §13.4 the third |
USERGUIDE.md — anything |
it was wrong and is gone. §2 for the trust model, §3 for identity, §6.7 for the operator surface |
QUICKSTART.md — anything |
it was stale and is gone. PACKAGING-GUIDE.md installs; §6.7 lists the operator commands |
devel-phases-next.md structural decisions |
§14.1 |
cast-smart-tv.md |
§11.4 |
Older references to draft-v3 and draft-v4 sections point into old-draft.md,
which is a historical archive and is not authoritative.