diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/MESHBAY_DESIGN.md | 57 | ||||
| -rw-r--r-- | docs/playlists.md | 1064 |
2 files changed, 831 insertions, 290 deletions
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md index 2b6d1ac..aaa232c 100644 --- a/docs/MESHBAY_DESIGN.md +++ b/docs/MESHBAY_DESIGN.md @@ -32,7 +32,7 @@ | **this document** | the architecture, the trust model, and the reason each decision is what it is | | `MESHBAY_NODE_PROTOCOL.md` | the MNP wire format, message by message | | `transfers-v1.md` | the transfer system's failure-mode analysis, kept because a synthesis cannot carry "every way a slot can be lost" | -| `playlists.md` | the playlist design in full — decided, not built (§9.10) | +| `playlists.md` | the playlist design and its interface in full — decided, not built (§9.10) | | `cast-smart-tv.md` | the DLNA/UPnP device backend — designed, not built (§11.4) | | `WINDOWS-PORT.md` | the Windows port's audit and packaging detail (§11.2) | | `PACKAGING-GUIDE.md`, `HTTPS.md`, `MAIL-SERVER.md`, `windows-build.md` | installation and server operations | @@ -2327,10 +2327,23 @@ removed from the hub, and it is the same rule that keeps resume positions local: *nothing new learns what you watch.* An encrypted blob on the hub is technically trivial and is still refused. -It lives on the node instead, as **an opaque per-account blob in `bundles.db`** — -the same shape as the keypair bundle, which the node already stores and cannot -read. **No new trust boundary**: the node is not asked to hold a kind of thing it -does not already hold for that same account. +It lives on the node instead, as **opaque per-account blobs in `bundles.db`** — the +same shape as the keypair bundle, which the node already stores and cannot read. +**No new trust boundary**: the node is not asked to hold a kind of thing it does +not already hold for that same account. + +**One blob per playlist, plus a small manifest** — not one blob for the collection, +and the reason is write amplification rather than size. Under a single blob, +starring one track rewrites and re-uploads the whole collection to every node +reached; split, it rewrites that one playlist. The manifest — names, revisions, +tombstones, counts, a few KB — is also the only thing every menu needs, so "add to +playlist" draws instantly with every node offline, and a body is fetched only when +its playlist is opened or played. Blobs are **compressed before sealing and padded +after**: the payload is repetitive enough to be worth a factor of three, and the +padding is what stops a ciphertext length from counting somebody's tracks. The node +caps each blob and the account's total, and **refuses rather than truncates** — a +truncating cap silently loses tracks, which is the failure the whole design exists +to prevent. **The key is the one thing that must not be got wrong.** Identity keys are per node (§3.2), so a blob encrypted under one is unreadable from every other node — the @@ -2340,14 +2353,36 @@ is the bundle key, so `playlist_key = HKDF(bundle_key, info = Argon2 run, and a purpose-separated subkey rather than the bundle key reused with a different AAD (§4.4's rule). The nonce is 96 random bits and never a counter, for exactly the reason chat's is (§4.5): two devices of one account derive the *same* -key, which is the point. +key, which is the point. The AAD names the blob's *kind*, so one playlist's body +cannot be served in place of another's. **Merge is the hard third, and the granularity is what makes it tractable.** The -unit is **one playlist, not the collection**; revision counters order writes, never -the wall clock; and **a deletion is a tombstone, never an absence** — an absence is -indistinguishable from a device that has not seen the addition yet. A node that is -offline for a month therefore cannot corrupt anything: it holds an older revision of -some playlists and is overwritten per playlist, not wholesale. +unit is **one playlist, not the collection** — which the per-playlist blob now +makes true of the storage as well, so two devices editing two playlists do not even +write the same row. Revision counters order writes, never the wall clock; and **a +deletion is a tombstone, never an absence** — an absence is indistinguishable from a +device that has not seen the addition yet. A node that is offline for a month +therefore cannot corrupt anything: it holds an older revision of some playlists and +is overwritten per playlist, not wholesale. + +**The interface is a menu inside Music, not a page.** Music's player is already +persistent at shell level and already holds a queue that crosses groups; *loading* a +playlist replaces that queue, and below that call a playlist and an album are +indistinguishable — so auto-advance, shuffle and prefetch are unchanged by +construction. The one genuine code change is that the queue can today only be +*replaced*: "play next" and "add to queue" require it to become appendable, which +makes it a small reducer rather than three pieces of component state. Everything +else is a context menu on a cover or a track row, and one button in Music's sticky +toolbar. Because Music is mounted by both the group page and the Search page +(§9.11), a playlist built inside a group is managed from the consolidated view with +no second surface and no application-registry entry. + +**Unavailability is answered at play time, not at add time.** Whether a group is +reachable is only knowable by dialing, and refusing to add a track because its node +is off tonight loses the user's intent permanently to a condition that lasts an +evening. So adding never dials; playback skips, distinguishing a file that will not +decode (a property of that file) from a group that does not answer (a property of +that group, whose tracks are then skipped together). **It adds no dialing and no new streaming path.** Sync rides connections the client already makes, and playback is unchanged (§9.8). diff --git a/docs/playlists.md b/docs/playlists.md index 485011a..ecbbac4 100644 --- a/docs/playlists.md +++ b/docs/playlists.md @@ -1,23 +1,22 @@ # MeshBay — Playlists (design) > Status: **proposal**, not implemented. This was deferred out of the Music -> application's design, -> which deferred it for the right reason: *"a genuinely new category of -> per-account node state, not covered by anything E9 already enumerates — -> needs its own design pass (ownership, sync across devices, whether it's -> node-local or something else)"*. This document is that pass. +> application's design, which deferred it for the right reason: *"a genuinely +> new category of per-account node state, not covered by anything E9 already +> enumerates — needs its own design pass (ownership, sync across devices, +> whether it's node-local or something else)"*. This document is that pass. > > Read `MESHBAY_DESIGN.md` §9.8 first — Music is built, and this adds nothing -> to its playback path. Read §9.11 second: the -> cross-group consolidation this feature needs already exists there, and -> most of the work is recognising that. +> to its playback *path*. Read §9.11 second: the cross-group consolidation +> this feature needs already exists there, and most of the work is +> recognising that. > > **Scope, settled before writing this:** a playlist belongs to **one > account** and is never shared with other group members. That answer is what -> keeps §5 small; see §10 for what changes if it is ever reversed. +> keeps §6 small; see §13 for what changes if it is ever reversed. > -> Follows the project convention: every claim names the adversary it holds -> against (§9). +> Follows the project convention: every claim names the adversary or the +> failure it holds against (§12). --- @@ -28,36 +27,64 @@ turn up on that person's other devices — and so that one playlist may hold albums from **several different groups on several different nodes**, the way the Search page already searches a consolidated view. With the constraint, stated up front, that nodes go offline for an evening or for a month and that -this must not corrupt anything. +this must not corrupt anything. And — added after the first draft, and now +half of this document — **a user interface**: the queue verbs (play, play +next, enqueue) that a music player is expected to have, a way to build a +playlist from what is on screen, a way to manage playlists from one button, +and none of it breaking the album auto-advance and prefetch that already +work. + +## 0.1 What changed from the first draft, and why + +The first draft answered the storage question and stopped. Three things in it +were wrong or missing, and they are corrected here rather than patched: + +| | First draft | Now | § | +|---|---|---|---| +| Size | one blob for everything, capped at 256 KB | **manifest + one blob per playlist**, compressed, stored as a BLOB; caps per unit | §4, §5 | +| The stored track reference | "takes it unchanged" into `onPlayQueue` | **it does not** — `size` and `name` are missing and both are required to fetch a track at all | §5.3 | +| UI | one route `/playlists` and a sidebar entry | **dropped.** A menu in Music's sticky toolbar, which Search → Music inherits for free | §9, §13 O7 | + +The 256 KB figure contradicted its own worked example: the draft estimated +1 000 tracks at ~200 bytes as "~200 KB", against a cap applied to `blob_enc`, +which was a **base64 TEXT** column — so its own example was already ~267 KB on +the wire, over a cap it declared generous. §4 redoes the arithmetic with the +fields that actually have to be there. --- ## 1. What this design does not reopen -- **Views over the index, never a catalogue** (`MESHBAY_DESIGN.md` §6.5, §9.1, - draft-v6 §2.7). A playlist is a list of *references*; it creates no second - identity for a file and no server-side database of content. -- **Nothing about content reaches the hub** (H7, draft-v6 §2.5). §3.1. -- **No new streaming path.** `MESHBAY_DESIGN.md` §9.8 stands untouched: a track is - fetched through `pipelinedDownload` and handed to `<audio>`. Playlists add - no node-side playback code whatsoever. -- **Node-side derived data lives in the node's own `data_dir`**, never beside - a shared file (`MESHBAY_DESIGN.md` §6.5). The blob in §3.3 goes in `bundles.db`. +- **Views over the index, never a catalogue** (`MESHBAY_DESIGN.md` §6.5, §9.1). + A playlist is a list of *references*; it creates no second identity for a + file and no server-side database of content. +- **Nothing about content reaches the hub** (H7). §3.1. +- **No new streaming path.** `MESHBAY_DESIGN.md` §9.8 stands untouched: a track + is fetched through `pipelinedDownload` and handed to `<audio>`. Playlists add + no node-side playback code whatsoever, and no change to transcode, chunking, + or the GEK. +- **Node-side derived data lives in the node's own `data_dir`**, never beside a + shared file (`MESHBAY_DESIGN.md` §6.5). The blobs in §3.3 go in `bundles.db`. - **Identity keys are per node** (`MESHBAY_DESIGN.md` §3.2). §3.4 exists entirely because of this, and getting it wrong is the fastest way to ship a playlist that cannot be read from the second device it was invented for. +- **The player bar stays where it is.** It is already at shell level in + `app.js`, already resolves a connection per track from `entry.groupId`, and + already crosses groups within one queue. The cross-group requirement reads + like the hard part and is in fact already built. --- -## 2. Three problems, and only one of them is hard +## 2. Four problems, and only two of them are hard | | Problem | Where it is solved | |---|---|---| -| A | Somewhere durable to put a few KB per account | §3 — several acceptable answers, one clearly best | -| B | How to **name** a track that lives in one group on one node, inside a list that spans several | §4.2 — where the silent failures live | -| C | Reconciling N copies when nodes are ON and OFF | §5 — the stated fear, and it is well founded *only* for the wrong granularity | +| A | Somewhere durable to put a few hundred KB per account | §3 — several acceptable answers, one clearly best | +| B | How to **name** a track that lives in one group on one node, inside a list that spans several | §5.3 — where the silent failures live | +| C | Reconciling N copies when nodes are ON and OFF | §6 — the stated fear, well founded *only* for the wrong granularity | +| D | A queue that can be **appended to**, not only replaced | §8 — the player's queue is three `useState`s reset wholesale; that is the one real code change | -The request reads as a storage problem. Storage is the easy third of it. +The request reads as a storage problem. Storage is the easy quarter of it. --- @@ -66,14 +93,14 @@ The request reads as a storage problem. Storage is the easy third of it. ### 3.1 Not the hub, and the rule is narrower than "nothing on the hub" The hub *does* hold small per-account state: `user_preferences` -(`db/models.py:303`), behind an allowlist of keys that already includes +(`db/models.py:302`), behind an allowlist of keys that already includes `music_keep_screen_on` and `default_tab:<group_id>`. So the operative rule is not "the hub stores nothing about a user". It is: > **No content metadata on the hub.** A playlist is, literally, a list of blake3 hashes of private-group files — -plus, if it is to render while nodes are offline (§4.2), their titles and +plus, if it is to render while nodes are offline (§5.3), their titles and artists. That is the exact object H7 removed from the hub, and it is what `CLAUDE.md` already pins one size smaller for the video resume position: *"nothing new learns what you watch"*. @@ -91,20 +118,20 @@ argument. If a future need makes hub involvement genuinely attractive, the only acceptable shape is a **version vector** — `playlists_rev: 7` in `user_preferences`, telling a client that some node is stale — never the -payload. Even that is probably not worth its own migration; §6 gets the same +payload. Even that is probably not worth its own migration; §7 gets the same result with no hub change at all. ### 3.2 Not node-to-node Refused, and not on cost grounds. Nodes do not know each other, share no -authenticated channel, and `replication.py` is legacy public-content code -that has nothing to do with this. Beyond the protocol that would have to be -invented, it leaks the thing this architecture is most careful about: -node A would learn that this account also uses node B — that two unrelated -operators host the same person. Per-node identity exists precisely so that -this correlation is unavailable (`MESHBAY_DESIGN.md` §3.2). +authenticated channel, and `replication.py` is legacy public-content code that +has nothing to do with this. Beyond the protocol that would have to be +invented, it leaks the thing this architecture is most careful about: node A +would learn that this account also uses node B — that two unrelated operators +host the same person. Per-node identity exists precisely so that this +correlation is unavailable (`MESHBAY_DESIGN.md` §3.2). -### 3.3 The node, as an opaque per-account blob — a shape already built +### 3.3 The node, as opaque per-account blobs — a shape already built The node already stores an encrypted per-account object it cannot read: the keypair bundle (`bundle_store.py`, table `keypair_bundles`, MNP ops @@ -118,22 +145,36 @@ A new table in the same `bundles.db`: ```sql CREATE TABLE IF NOT EXISTS user_blobs ( user_id TEXT NOT NULL, - kind TEXT NOT NULL, -- "playlists" today; the column is the - -- reason a second one needs no migration + kind TEXT NOT NULL, -- "playlists" (the manifest), or + -- "playlist:<id>" (one playlist's tracks) rev INTEGER NOT NULL, - blob_enc TEXT NOT NULL, - blob_enc_recovery TEXT, -- reserved, §10 O2 — see below + blob_enc BLOB NOT NULL, -- NOT TEXT — see below + blob_enc_recovery BLOB, -- reserved, §13 O2 stored_at TEXT NOT NULL DEFAULT (datetime('now')), PRIMARY KEY (user_id, kind) ); ``` +Two deliberate differences from `keypair_bundles`, each of which is a mistake +avoided rather than a preference: + +- **`BLOB`, not `TEXT`.** `keypair_bundles.bundle_enc` is base64 TEXT because + it holds a few hundred bytes and nobody will ever notice the 33 % tax. This + table holds hundreds of KB, where a third is not a rounding error. The + transport already carries msgpack `bin` (`transport.js:3571` handles it + explicitly), and `aiosqlite` binds `bytes` natively, so base64 buys nothing + here and costs a third of every write. +- **`kind` is a namespace, not an enum.** `"playlists"` is the manifest; + `"playlist:<uuid>"` is one playlist's tracks. The allowlist in §8.2 is a + *pattern*, not a literal set, and that is the whole reason the split in §4 + costs no second table. + `blob_enc_recovery` is declared now and left NULL. `bundle_store.py` already -carries `_migrate_keypair_recovery` — a `PRAGMA table_info` dance — for -exactly this column added to exactly this kind of table one release late. -`CREATE TABLE IF NOT EXISTS` never adds a column, which is the node-local -twin of the `create_all()` lesson in `CLAUDE.md`. Declaring the slot costs -nothing today and removes the migration entirely. +carries `_migrate_keypair_recovery` — a `PRAGMA table_info` dance — for exactly +this column added to exactly this kind of table one release late. `CREATE TABLE +IF NOT EXISTS` never adds a column, which is the node-local twin of the +`create_all()` lesson in `CLAUDE.md`. Declaring the slot costs nothing today +and removes the migration entirely. ### 3.4 The key — the one thing that must not be got wrong @@ -152,38 +193,127 @@ playlist_key = HKDF-SHA256(bundle_key_v2, info = "meshbay:playlists:v1") Three consequences, each of which is a line of code somewhere: - **`deriveEncryptionKey` must return an HKDF handle as well as the AES-GCM - one.** Today it imports the 32 Argon2 bytes non-extractably as `AES-GCM` - with `['encrypt','decrypt']`, from which nothing can be derived. Import the - *same* `out.hash` a second time as `HKDF` with `['deriveKey']`. One Argon2 - run, two handles, no extra cost on the 650 ms sign-in path. Do **not** add a - second Argon2 call, and do **not** encrypt playlists directly under the - bundle key with a different AAD: `groupbox.py` already sets the convention - ("*Never reuse `chunk_key_aes` with a pseudo-file for this*") and - purpose-separated subkeys are what it looks like here. -- **v2 only.** `bundleKey` is `{v2, v1}` so that a bundle written before the - KDF changed can still be opened. Playlists are new; there is no legacy blob - and therefore no v1 branch. One less thing that can silently take the wrong - fork. + one.** Today (`keyderive.js:154`) it imports the 32 Argon2 bytes + non-extractably as `AES-GCM` with `['encrypt','decrypt']`, from which nothing + can be derived. Import the *same* `out.hash` a second time as `HKDF` with + `['deriveKey']`. One Argon2 run, two handles, no extra cost on the 650 ms + sign-in path. Do **not** add a second Argon2 call, and do **not** encrypt + playlists directly under the bundle key with a different AAD: `groupbox.py` + already sets the convention ("*Never reuse `chunk_key_aes` with a pseudo-file + for this*") and purpose-separated subkeys are what it looks like here. +- **v2 only.** `bundleKey` is `{v2, v1}` so that a bundle written before the KDF + changed can still be opened. Playlists are new; there is no legacy blob and + therefore no v1 branch. One less thing that can silently take the wrong fork. - **The nonce is 96 random bits, never a counter.** Two devices of one account - derive the *same* playlist key — that is the whole point — so a counter - would repeat. This is the same reasoning already recorded for chat subkeys - in `CLAUDE.md`'s module table, and it is safe for the same reason. + derive the *same* playlist key — that is the whole point — so a counter would + repeat. This is the same reasoning already recorded for chat subkeys in + `CLAUDE.md`'s module table, and it is safe for the same reason. + +A passphrase change re-derives the key and every blob must be re-encrypted on +the next write, exactly as the keypair bundle already is ("*re-encrypted on the +next backup*"). With the split of §4 that is now N+1 writes rather than one — +do them lazily, on each playlist's next write, not in a loop at sign-in. + +**AAD** = `"user_blob|<kind>|<user_id>"`, mirroring `groupbox.associated_data`. +`<kind>` is the full namespaced kind, so a body sealed for playlist A cannot be +served in place of playlist B's — which a bare `"playlists"` AAD would have +allowed, and which is a real substitution now that there is more than one row. +It does not, and cannot, prevent rollback — §6.4. + +--- + +## 4. Size — the first draft's arithmetic, redone + +### 4.1 What an entry actually costs + +§5.3 settles which fields have to be stored. With short msgpack keys: + +| field | bytes | +|---|---| +| `id` (blake3 hex) | 65 | +| `g` (group UUID, 36 chars) | 37 | +| `n` (filename) | ~31 | +| `p` (path) | ~56 | +| `t` (display title) | ~26 | +| `a` (artist) | ~16 | +| `b` (album) | ~23 | +| `s` (size), `d` (duration), `tn`, `hv` | ~10 | +| 11 keys + map header | ~34 | +| **total** | **~300 B** | + +The first draft said 200 B, from a field list that could not play a track. + +### 4.2 Three realistic accounts + +| | entries | plaintext | base64 (draft) | deflate + BLOB (this draft) | +|---|---|---|---|---| +| draft's own example | 1 000 | 300 KB | **400 KB** — over its own 256 KB cap | ~100 KB | +| ordinary user: Favourites 2 000 + 15 lists | ~4 250 | 1.3 MB | 1.7 MB | ~430 KB | +| heavy user | 20 000 | 6 MB | 8 MB | ~2 MB | + +So: **the doubt was correct, and by roughly an order of magnitude.** 256 KB is +not enough for one realistic Favourites list, let alone a collection. + +### 4.3 Four fixes, in order of how much they matter + +**1. Split the storage per playlist (§3.3's `kind` namespace).** This is the +one that matters, and not mainly for the cap. Under one blob, starring a single +track rewrites and re-uploads *the whole collection* — 1.7 MB, to every node +reached, for one star. Twenty stars in an evening is 34 MB over WebRTC per node. +Split, the same star rewrites Favourites (~200 KB) plus a ~3 KB manifest. The +merge unit was already one playlist (§6.1); this makes the *storage* unit agree +with it, which is also why the merge gets simpler rather than harder. + +It has a second payoff, bigger than it looks: the **manifest alone is enough to +draw every menu in §9**. "Add to playlist ▸" lists names, and names are in the +manifest — a few KB, always in IndexedDB, rendered instantly with every node +offline and no body fetched. Under one blob that menu waits on megabytes. + +**2. Compress before sealing.** `CompressionStream('deflate-raw')` — native in +Chromium, Firefox 113+, Safari 16.4+, and therefore in the Electron build; no +dependency, and the project has no compression library today (`zipstream.js` +deliberately stores rather than deflates, for a different reason). The shape is +about as compressible as data gets: the 11 keys repeat every entry, `group_id` +repeats across the whole list, and path prefixes repeat per album. 3× is +conservative for this; 4× is typical. + + **Order is compress → pad → seal, never seal → compress.** Ciphertext does not + compress, and the padding must be inside the AEAD. + +**3. `BLOB` not `TEXT`** (§3.3) — a third of every write, free. + +**4. Caps that refuse, never truncate.** A truncating cap silently loses tracks, +which is the failure this whole document exists to prevent. + +| limit | value | enforced by | +|---|---|---| +| manifest (`kind = "playlists"`) | 64 KB sealed | node | +| one body (`kind = "playlist:<id>"`) | 1 MB sealed ≈ 10 000 tracks | node | +| all rows for one account on one node | 8 MB | node | +| tracks per playlist | 10 000 | client, before sealing | +| playlists per account | 200 | client, before sealing | -A passphrase change re-derives the key and the blob must be re-encrypted on -the next write, exactly as the keypair bundle already is ("*re-encrypted on -the next backup*"). +The client limits exist so the ordinary case produces a sentence in the UI +rather than an MNP error; the node limits exist because a client is not +trusted to hold to them, and this is an unbounded write primitive pointed at +someone else's disk. Noted in passing, and out of scope here: +`_do_keypair_bundle_store` (`webrtc_server.py:1113`) has **no cap at all** +today. Worth its own line somewhere. -**AAD** = `"user_blob|playlists|<user_id>"`, mirroring -`groupbox.associated_data`. It binds the ciphertext to its owner and its -kind. It does not, and cannot, prevent rollback — §5.4. +**Padding.** Pad the compressed plaintext up to the next 4 KB before sealing. +The length otherwise tells the operator roughly how many tracks this account +has collected, and — once compression is in — roughly how repetitive its paths +are. Cheap, and it is the only metadata this design leaks to a node that the +node cannot already see. See §12 for the chosen-plaintext angle and why it is +named and then dismissed. --- -## 4. The data model +## 5. The data model -### 4.1 The blob +### 5.1 The manifest — `kind = "playlists"` -Plaintext-before-sealing, msgpack (same encoding as everything else on MNP): +Small, always resident, and the only thing every menu needs: ``` { @@ -191,11 +321,12 @@ Plaintext-before-sealing, msgpack (same encoding as everything else on MNP): "playlists": { "<playlist_id>": { "name": "Evening", - "rev": 7, + "rev": 7, -- of the manifest entry + "body_rev": 41, -- of the body blob, §5.2 "device": "<device_pk_ed25519, first 16 hex>", - "updated_at": 1757000000, -- display only, never a merge input + "updated_at": 1757000000, -- display only, never a merge input "deleted": false, - "tracks": [ <entry>, ... ] + "count": 312 -- display only; the body is authoritative }, ... } @@ -205,163 +336,208 @@ Plaintext-before-sealing, msgpack (same encoding as everything else on MNP): `playlist_id` is a client-generated UUID, with one reserved value: `"favorites"`. **Favourites is a playlist**, not a second mechanism — deciding that now is what stops a parallel store being built next to this one in three -months. +months. It is *implicit*: never created by the user, materialised on first use, +always first in every picker (as asked), and **it cannot be deleted** (§9.3). +Emptying it is fine. Refusing the delete outright is what keeps §6.3's +tombstone rule free of an exception, and a tombstone rule with an exception is +how §6.3's defect ships. -### 4.2 A track reference, and why each field is there +### 5.2 A body — `kind = "playlist:<id>"` ``` -{ "group_id": ..., "file_id": <blake3 hex>, "hash_version": 1|2, - "path": "Some Artist/An Album/03 - A Track.flac", - "title": ..., "artist": ..., "album": ..., "duration": 214 } +{ "v": 1, "id": "<playlist_id>", "rev": 41, "device": "...", + "tracks": [ <entry>, ... ] } ``` -A bare hash is not enough, and each field prevents a specific failure: +Array order *is* playlist order. `rev` here is the body's own, and the +manifest's `body_rev` is a copy of it — a client that sees a manifest naming +`body_rev: 41` while holding body `rev: 39` knows to fetch without fetching to +find out. A body whose manifest entry is `deleted` is dropped locally and +`user_blob_delete`d on each node as it is reached; the tombstone in the +manifest is what survives, not the body. -- **`group_id`** — the player resolves its connection per track from - `entry.groupId` (`music-player.js:288`). Without it there is nothing to - dial, and a `file_id` alone has no meaning outside a group. -- **`title` / `artist` / `album` / `duration`, denormalised** — this is not - redundancy, it is the core of the design. With them a playlist renders - **completely** with every node offline, unplayable entries greyed out, in - the same spirit as the Search page reporting its `unreachable` list rather - than failing. Without them, an offline playlist is a column of hex strings — - and *that* is the incoherence the request is worried about. It is a - display problem, and it is solved by copying four small strings. -- **`hash_version`** — the index already has two hashing schemes - (`protocol.py:292`: 1 = whole file, 2 = 45 MB sample). A re-hash would - orphan every entry in every playlist, silently and all at once. -- **`path`** — content addressing survives a move; a path survives a - re-encode. Keeping both means either can repair the other: on a sight of - the live index, an entry whose `file_id` is absent but whose `path` matches - has its id rewritten in place (and vice versa), once, on the client. +### 5.3 A track reference — and the field the first draft omitted -Rehydration is by design cheap: a playlist entry is a subset of the -`IndexEntry`-plus-`groupId` shape the player already consumes, so -`onPlayQueue(tracks, startIndex)` takes it unchanged. +``` +{ "id": <blake3 hex>, "g": <group_id>, "hv": 1|2, + "n": "03 - A Track.flac", -- filename + "s": 41238711, -- bytes + "p": "Some Artist/An Album", -- directory path, group-relative + "t": "A Track", "a": "Some Artist", "b": "An Album", + "d": 214, "tn": 3 } +``` -**Availability bonus, close to free.** `source-merge.js` exists because the -same content appears in more than one group. At play time, if the entry's own -`group_id` has no reachable node but the same `file_id` appears in another -cached index whose node is up, play it from there. That is a global playlist -that heals itself when one operator's machine is off, reusing the -fold-on-content-hash logic already written for Search. Default on; see §9 for -the one thing it changes. +Each field prevents a specific failure: -### 4.3 Size +- **`g`** — the player resolves its connection per track from `entry.groupId` + (`music-player.js:292`). Without it there is nothing to dial, and a hash alone + has no meaning outside a group. +- **`s` and `n` — the correction.** The first draft's entry had neither, and + claimed the result was droppable into `onPlayQueue` unchanged. It is not. + `fetchTrackBlob` (`music-player.js:289`) computes + `Math.ceil(entry.size / CHUNK_SIZE)` to know how many chunks to ask for, calls + `guessMime(entry.name)` for the blob type, and tests `entry.name` against + `NEEDS_TRANSCODE_RE` to decide whether a `.wma`/`.mpc` needs the node-side + conversion first. **Without `s` a playlist entry cannot be downloaded at + all**, and without `n` it is downloaded and then handed to `<audio>` with the + wrong MIME, or a WMA is handed over raw and fails to decode in every browser. + The symptom is "playlists don't play", found at integration time and nowhere + earlier. +- **`t` / `a` / `b` / `d` / `tn`, denormalised** — this is not redundancy, it is + the core of the design. With them a playlist renders **completely** with every + node offline, unplayable entries greyed, in the same spirit as the Search page + reporting its `unreachable` list rather than failing. Without them, an offline + playlist is a column of hex strings. A display problem, solved by copying four + small strings. +- **`hv`** — the index already has two hashing schemes (`protocol.py:297`: + 1 = whole file, 2 = 45 MB sample). A re-hash would orphan every entry in every + playlist, silently and all at once. +- **`p`** — content addressing survives a move; a path survives a re-encode. + Keeping both means either can repair the other: on a sight of the live index, + an entry whose `id` is absent but whose `p`+`n` match has its id rewritten in + place (and vice versa), once, on the client. -A thousand tracks at ~200 bytes each is ~200 KB before compression. That is -small, and it is also an unbounded write primitive pointed at someone else's -disk, so: +**Rehydration, and the whitelist that is not optional.** A stored entry is +mapped to the player's entry shape (`id, name, path, size, duration, +display_title, artist, album, track_no, hash_version, groupId`) on load, and +the reverse map on save is a **strict field whitelist**. The entries the UI +hands over carry live junk: `_tRef` and `_gRef` (a transport and a CryptoKey, +attached by the Search page's merge), `_origPath`, and whatever else a future +view hangs on them. msgpack-encoding a transport object throws at best; storing +a stale one and reading it back a week later is worse, because it looks like a +connection and is not. Whitelist on the way in, in `playlists.js`, once. -- The node **caps and refuses**, never truncates. 256 KB of `blob_enc` is - generous for the shape above. (Noted in passing: `_do_keypair_bundle_store` - (`webrtc_server.py:1056`) has no cap at all today. Out of scope here, worth - its own line somewhere.) -- The client **pads the plaintext up to the next 4 KB** before sealing. The - ciphertext length otherwise tells the operator roughly how many tracks this - account has collected. Cheap, and it is the only metadata this design leaks - to a node that the node cannot already see. +**Availability bonus, close to free.** `source-merge.js` exists because the same +content appears in more than one group. At play time, if the entry's own `g` has +no reachable node but the same `id` appears in another cached index whose node +is up, play it from there. That is a global playlist that heals itself when one +operator's machine is off, reusing the fold-on-content-hash logic already +written for Search. Default on; see §12 for the one thing it changes. --- -## 5. Merge — the part that has to be right +## 6. Merge — the part that has to be right -The stated fear is correct **for a single blob under last-writer-wins**: node -A is off while an edit is made, node B is off while the next one is, and one -edit disappears with nothing to show for it. Four rules remove it. +The stated fear is correct **for a single blob under last-writer-wins**: node A +is off while an edit is made, node B is off while the next one is, and one edit +disappears with nothing to show for it. Four rules remove it. -### 5.1 The unit is a playlist, not the collection +### 6.1 The unit is a playlist, not the collection -The blob is a map keyed by `playlist_id`, and merging is per key. Two -playlists edited on two devices never collide, which is the overwhelmingly -common case for one person with two or three devices. +The manifest is a map keyed by `playlist_id` and merges per key; bodies are +separate rows and merge one against one. Two playlists edited on two devices +never collide, which is the overwhelmingly common case for one person with two +or three devices. §4.3's split means this is now true of the *storage* as well +as the merge, so two devices editing two playlists do not even write the same +row. -### 5.2 `rev`, never the wall clock +### 6.2 `rev`, never the wall clock -Each playlist carries a monotonic `rev` and the `device` that last wrote it. -Merge takes the higher `rev`; a tie is broken by the lexicographically -smaller `device`, so every device reaches the same answer without talking to -any other. `updated_at` is carried for display and is **never** read by the -merge — clocks across devices are not trustworthy, and a clock-based merge -fails roughly one time in twenty, which is the frequency at which this -codebase's history says a defect ships. +Each manifest entry and each body carries a monotonic `rev` and the `device` +that last wrote it. Merge takes the higher `rev`; a tie is broken by the +lexicographically smaller `device`, so every device reaches the same answer +without talking to any other. `updated_at` is carried for display and is +**never** read by the merge — clocks across devices are not trustworthy, and a +clock-based merge fails roughly one time in twenty, which is the frequency at +which this codebase's history says a defect ships. -### 5.3 A deletion is a tombstone, never an absence +### 6.3 A deletion is a tombstone, never an absence -`deleted: true`, kept. Absence must mean "this copy is older than the one -that created it". Otherwise a node rehomed after three weeks **resurrects -every deleted playlist** — this is the single most likely defect in the whole -design, it looks like a sync working correctly right up until it doesn't, and -it deserves its own named test. Tombstones are collected only when every -known node reports a `rev` at or above the deleting one, which for a +`deleted: true` in the manifest, kept. Absence must mean "this copy is older +than the one that created it". Otherwise a node rehomed after three weeks +**resurrects every deleted playlist** — this is the single most likely defect in +the whole design, it looks like a sync working correctly right up until it +doesn't, and it deserves its own named test. Tombstones are collected only when +every known node reports a `rev` at or above the deleting one, which for a single-node account is immediate and for a multi-node one is eventual; a tombstone is ~40 bytes, so there is no hurry. -### 5.4 Rollback, and why local-first answers it +A body arriving for a playlist the manifest tombstones is **dropped, not +resurrected** — the manifest is the authority on existence. That ordering is +the whole reason `count` lives in the manifest rather than being derived from +the body. -AEAD authenticates a blob; it does not stop a node handing back an older one -it still has (or a fresh one it never received). The defence is that **the -client is the authority**: the merged state lives in the client's own -IndexedDB, and merge takes the maximum `rev` across *local plus every node -answering*. A stale or lying node can only lose the tie. It can never lower -the merged state, because the local copy is one of the inputs. +### 6.4 Rollback, and why local-first answers it -This is what makes an offline node a non-event rather than a hazard. Nodes -are backups and a transport. They are not the source of truth, and no node -being reachable at all still leaves every playlist correct and, thanks to -§4.2, fully legible. +AEAD authenticates a blob; it does not stop a node handing back an older one it +still has (or a fresh one it never received). The defence is that **the client +is the authority**: the merged state lives in the client's own IndexedDB, and +merge takes the maximum `rev` across *local plus every node answering*. A stale +or lying node can only lose the tie. It can never lower the merged state, +because the local copy is one of the inputs. -### 5.5 What is genuinely lost, stated plainly +This is what makes an offline node a non-event rather than a hazard. Nodes are +backups and a transport. They are not the source of truth, and no node being +reachable at all still leaves every playlist correct and, thanks to §5.3, fully +legible. -Device 1 makes an edit, reaches no node, and is then lost or cleared: that -edit is gone. This is the exposure of any offline-first application, it is -not fixable without a durable always-reachable writer (which is §3.1, and -refused), and it is still enormously better than today, where the same edit -is lost on a cache clear regardless of what was online. +### 6.5 What is genuinely lost, stated plainly + +Device 1 makes an edit, reaches no node, and is then lost or cleared: that edit +is gone. This is the exposure of any offline-first application, it is not +fixable without a durable always-reachable writer (which is §3.1, and refused), +and it is still enormously better than today, where the same edit is lost on a +cache clear regardless of what was online. --- -## 6. Sync — and the point is that it adds no dialing +## 7. Sync — and the point is that it adds no dialing A sweep of every group's node costs 10 s per unreachable one; the Search page does it deliberately, batched three at a time, because the user asked it to. Playlists must not do that at sign-in. -**Sync rides on connections that were happening anyway.** Whenever a -transport to any node is open for any other reason — opening a group, the -Search page's sweep, the music pool dialing to play a track — the client -piggybacks a `user_blob_fetch`, merges, and sends `user_blob_store` back if -that node's copy is behind. `ConnectionPool` (`search-page.js:36`) already -holds up to `MAX_POOL_SIZE` (12) live connections and hands them out by -group, so this is a hook, not a new mechanism. +**Sync rides on connections that were happening anyway.** Whenever a transport +to any node is open for any other reason — opening a group, the Search page's +sweep, the music pool dialing to play a track — the client piggybacks a +`user_blob_fetch` for the **manifest**, merges, and sends `user_blob_store` back +if that node's copy is behind. `ConnectionPool` (`search-page.js:82`) already +holds up to `MAX_POOL_SIZE` (12) live connections and hands them out by group, +so this is a hook, not a new mechanism. + +**Bodies are fetched lazily**, and this is what the split buys at sync time as +well as at menu time: the manifest says which bodies are stale (`body_rev`), and +a body is pulled when it is opened, loaded, or edited — not on every connection. +A user with 40 playlists who only ever plays two never transfers the other 38. Two additions on top: -- **An explicit "Sync now"** on the Playlists page, which does the Search-style - sweep and reports which nodes it could not reach — the same honest - reporting `fetchAllIndexes` already does. -- **On sign-in, nothing.** The local copy is authoritative and complete - (§5.4); the first group opened will reconcile. +- **An explicit "Sync now"** in the playlist menu, which does the Search-style + sweep and reports which nodes it could not reach — the same honest reporting + `fetchAllIndexes` already does. +- **On sign-in, nothing.** The local copy is authoritative and complete (§6.4); + the first group opened will reconcile. -A device that only ever opens one group therefore only ever converges with -one node. That is correct and not a defect: convergence is eventual, and the -copy the user is looking at is right the whole time. +A device that only ever opens one group therefore only ever converges with one +node. That is correct and not a defect: convergence is eventual, and the copy +the user is looking at is right the whole time. It is also the honest answer to +"I made this playlist inside group A" (§11): the write lands on whatever node +was reachable, which is usually A's, and spreads from there. --- -## 7. Protocol and node-side implementation +## 8. Protocol and node-side implementation -### 7.1 MNP — additive, MINOR bump (2.1) +### 8.1 MNP — additive, MINOR bump (2.1) ``` -user_blob_store { kind, rev, blob_enc } client → node -user_blob_fetch { kind } client → node -user_blob_resp { kind, rev, blob_enc|null } node → client +user_blob_store { kind, rev, blob_enc } client → node +user_blob_fetch { kind } client → node +user_blob_list { } client → node +user_blob_delete { kind } client → node +user_blob_resp { kind, rev, blob_enc|null } node → client +user_blob_list_resp { blobs: [{kind, rev}] } node → client ``` -Modelled on `KEYPAIR_BUNDLE_*` in every respect, including that the node -stores and returns an opaque string. `kind` is validated against a small -allowlist (`{"playlists"}`) so the table does not become an arbitrary +`user_blob_list` is new relative to the first draft and is what the split +requires: a client that has lost its local state (a cache clear, a new device) +must be able to discover which bodies exist without guessing UUIDs. It returns +`kind` and `rev` only — never a payload — so it costs one small message and +tells the node nothing it does not already know from its own table. + +Modelled on `KEYPAIR_BUNDLE_*` in every respect, including that the node stores +and returns an opaque byte string. `kind` is validated against a **pattern** — +`playlists` or `playlist:<uuid>` — so the table does not become an arbitrary key/value store for whatever a client feels like writing. Every reply carries `req_id` through the ordinary `_send` path. This is not @@ -369,150 +545,480 @@ optional and does not need re-arguing: `CLAUDE.md` records at length what arrival-order matching costs, and the victim is never the request that was answered wrongly. -### 7.2 Node side +### 8.2 Node side | Piece | Where | What | |---|---|---| -| Storage | `meshbay_node/bundle_store.py` | `user_blobs` table (§3.3), `store_user_blob` / `fetch_user_blob`, same shape as `store_keypair` / `fetch_keypair` | -| Handlers | `transport/webrtc_server.py` | `_do_user_blob_store` / `_do_user_blob_fetch`, `self._user_id` from the authenticated session (NS6), never from the message | -| Cap | same | Refuse over 256 KB with a stated reason; refuse an unknown `kind` | -| Audit | same | `user_blob_store` / `user_blob_fetch` events, same as `keypair_bundle_store` already logs | +| Storage | `meshbay_node/bundle_store.py` | `user_blobs` table (§3.3); `store_user_blob` / `fetch_user_blob` / `list_user_blobs` / `delete_user_blob`, same shape as `store_keypair` / `fetch_keypair` | +| Handlers | `transport/webrtc_server.py` | `_do_user_blob_store` / `_fetch` / `_list` / `_delete`; `self._user_id` from the authenticated session (NS6), **never** from the message | +| Caps | same | §4.3's four limits. Refuse with a stated reason; refuse a `kind` that does not match the pattern | +| Audit | same | `user_blob_store` / `_fetch` / `_delete` events, as `keypair_bundle_store` already logs | `user_id` comes from the session, exactly as `_do_keypair_bundle_store` takes -it — a `user_id` in the message body would let any member read or overwrite -any other member's blob, which is the C5 shape one size down. +it — a `user_id` in the message body would let any member read or overwrite any +other member's blob, which is the C5 shape one size down. **No hub change. No change to indexing, streaming, transcoding, or the GEK.** -### 7.3 What the node can and cannot do with it +### 8.3 What the node can and cannot do with it -It can delete it, lose it with its disk, or serve a stale copy (§5.4 covers -the last). Durability is "the local copy, plus N node copies", with no -guarantee from any single one — which for an account on one node means the -local copy matters. Worth one line in the UI, not a warning dialog. +It can delete it, lose it with its disk, or serve a stale copy (§6.4 covers the +last). Durability is "the local copy, plus N node copies", with no guarantee +from any single one — which for an account on one node means the local copy +matters. Worth one line in the UI, not a warning dialog. --- -## 8. Client side +## 9. The queue — the change the UI actually needs + +This section answers the question asked directly, because the answer is yes and +the reason is worth stating precisely. + +### 9.1 Yes: "load a playlist" replaces the current queue -### 8.1 A pure module, deliberately separate from the UI +There is already a current list, it is what shuffle operates over, and loading a +playlist into it is the right shape. Concretely, in `music-player.js`: -`static/playlists.js` holds the data layer and **no UI**: load/merge/save, -the tombstone rules, the id/path repair, seal and open. This is not tidiness. -Source-reading tests are weak evidence and are most of what this repo can do -for the SPA — but a pure function over two objects can be *executed*, the way -`tests/harness/mse_harness.mjs` lifts the real player functions and runs -them. The merge is the one part of this feature that can be properly tested, -so it must not be entangled with a component. +```js +const [tracks, setTracks] = useState([]); // the queue, in insertion order +const [order, setOrder] = useState([]); // indices into tracks — the play order +const [pos, setPos] = useState(0); // index into order +``` -`tests/harness/playlist_merge.mjs` runs the real `merge()` over scripted -divergences: two devices, one node offline for each in turn, a delete on one -side and an edit on the other, a resurrect attempt, a `rev` tie. Model the -environment, never the code under test. +`order` is the indirection that makes shuffle work without disturbing `tracks` +(`shuffledOrder`, `toggleShuffle` at `music-player.js:444`), `pos` walks it, and +`prefetchNext(fromPos)` reads `tracks[order[fromPos + i]]`. An album handed over +by `onPlayQueue(album.tracks, i)` and a playlist handed over the same way are +**indistinguishable** below that call. So: -### 8.2 Where it appears +> **Load = `onPlayQueue(playlistTracks, 0, null, 'replace')`.** Confirmed. The +> existing queue is discarded, shuffle applies to the new one on the next +> render, prefetch warms from the new index 0, and auto-advance runs exactly as +> it does for an album — because it *is* the same code path, untouched. -Playlists cross groups, so they do not belong to a group's Music tab: +### 9.2 The real work: the queue can only be replaced today -- **New route `/playlists`** and `static/playlists-page.js`, a sibling of - `/search`, plus a sidebar entry. It renders from the **cached** indexes - (`getAllCachedIndexes`), so it opens instantly and works with everything - offline; it dials only to play, or on "Sync now". -- **"Add to playlist"** in `music-app.js` (album and track level) and in - `search-page.js`'s music results — both already have the entry in hand, - with `groupId` attached. -- **The player bar needs nothing.** It is already at shell level in `app.js`, - already resolves a connection per track from `entry.groupId`, and already - crosses groups within one queue. This is worth stating loudly, because the - cross-group requirement reads like the hard part and is in fact already - built (the cross-group merge already covers it — `MESHBAY_DESIGN.md` §9.11). +`onPlayQueue` flows to `app.js:738`: -### 8.3 The IndexedDB detail that will otherwise be missed +```js +setMusicQueue({ tracks, startIndex, nonce: Date.now() }); +``` -`hub-client.js` opens `meshbay` at `IDB_VERSION = 1` with a single store, -`group_indexes`. Adding a `playlists` store means **bumping to 2** and -handling it in the existing `onupgradeneeded`, which currently creates one -store and would otherwise never run again. A store that is never created -throws on first access, at a point far from the version constant. +and the player's effect keyed on `[queue]` (`music-player.js:344`) resets +`tracks`, `order`, `pos`, the error and the failure counter — wholesale, every +time. **There is no way to add to the queue without restarting it**, and "play +next" and "enqueue" are exactly that. This is problem D, and it is the only part +of the UI request that is not a new component over existing plumbing. -### 8.4 Checklist, per `MESHBAY_DESIGN.md` §9.4 +**The shape:** one more field, defaulted, so no existing call site changes. -1. `playlists.js` (data) and `playlists-page.js` (UI). -2. Route + sidebar entry in `app.js`. -3. Nothing for caching: every file under `static/` feeds the `/a/<hash>/` - fingerprint. -4. `test_hook_ordering.py` (`STATIC_FILES`) and - `test_transport_contracts.py` (`SPLIT_FILES`) — both new files. -5. i18n keys in all ten `static/locales/*.js`; `test_locales.py` holds them - to `en.js`'s key set. -6. `npm run sync-ui` in `meshbay-client`, confirmed reported. +```js +onPlayQueue(tracks, startIndex, source, op) // op: 'replace' | 'next' | 'append' +``` + +- `'replace'` (the default, and what every call site today means) — today's path, + behaviour for behaviour. +- `'next'` — splice the new indices into `order` at `pos + 1`. `pos` does not + move, the `<audio>` element is not touched, nothing reloads. +- `'append'` — push them at the end of `order`. + +### 9.3 Collapse `tracks` / `order` / `pos` into one reducer + +Three `useState`s updated from one event is where this ships a defect. `setOrder` +needs the length `setTracks` is about to produce, and cannot see it; read from +the stale closure instead and two enqueue clicks in the same tick produce indices +that both point past the end — a queue with holes, from a double click. + +So `queue-ops.js`: a pure reducer over `{ tracks, order, pos }` with actions +`replace / insertNext / append / removeAt / skipTo / reshuffle`. Two reasons, +and the second is the one that counts: + +1. It is the only way the append is correct under batching. +2. **It can be executed by a test.** Source-reading tests are weak evidence and + are most of what this repo can do for the SPA — but a pure function over a + state object runs, the way `tests/harness/mse_harness.mjs` lifts the real + player functions and runs them. `tests/harness/queue_ops.mjs` drives: append + to an empty queue, insert-next at the last position, insert-next while + shuffled, two appends in one tick, remove the currently playing track, + toggle shuffle after an append. Model the environment, never the code under + test. + +The regression guard is that `replace` must produce exactly today's +`{ tracks, order, pos }` for the same inputs — it is a mechanical lift of the +existing effect body, and the harness pins it. + +### 9.4 Three details that are cheap now and expensive later + +- **Shuffle and append.** Appended tracks go at the end of `order`, in the order + given, *not* reshuffled into the middle. That is what every player does and + what "add to queue" means. `toggleShuffle` rebuilds `order` from + `tracks.length` and so stays correct after any number of appends, for free. +- **`prefetchNext(pos)` must be called after an `insertNext`**, or the track the + user just asked to play next is the one track in the queue that is cold. One + line, and the only visible regression an enqueue can introduce. Nothing needs + evicting: `blobCacheRef` is keyed by `entry.id`, so a blob already fetched for + a track that has moved index is still keyed correctly. +- **`groupTransportRef` must only be set on `'replace'`** (`app.js:728`). It is + a single-slot fast path for the group currently open; repointing it because + the user enqueued one track from another group silently drops the playing + group back to the pool. Not a bug — the pool works — but a pointless one. + +### 9.5 Prefetch across groups — the one new consideration + +`prefetchDepth()` returns 5 on Wi-Fi, and `MAX_CACHED_BLOBS` is 6 to match. +Unchanged, and **must** stay unchanged: raising either is the obvious thing to +do for a long playlist and is wrong, because for an album those 5 tracks share +one connection, and for a cross-group playlist they may need **five different +node dials**, each up to 10 s, against a pool of 12. + +Rule: **prefetch only entries whose group already has a live connection in the +pool**; let the rest warm when the queue reaches them and the dial happens +anyway. An album is unaffected (one group, always live). A shuffled cross-group +playlist stops trying to open half the pool to warm tracks it may never reach. + +--- + +## 10. The user interface + +Nothing here is a new page. Requirement 6 — that all of this work inside a group +*and* on Search → Music — is satisfied by putting it in `music-app.js`, which +`group-page.js` and `search-page.js` (with `groupId="search"`, +`search-page.js:784`) already both mount. Written once, appears in both. + +### 10.1 A shared menu component, because there is none + +The codebase has no context menu and no dropdown outside the account menu in +`app.js:99`. `menu.js` provides `<Menu>` and `useMenu()`: -No `apps.js` registry entry and no `ALLOWED_APPS` change: this is not a group -application. It is a page, like Search. +- opens on **right-click** (`onContextMenu`, `preventDefault()`) and on a **dots + button** — `icon.js` already has `dots` (`icon.js:54`). Both affordances, both + platforms: the button is visible on hover on a fine pointer and always visible + under `@media (pointer: coarse)`. +- closes on Escape, on an outside click, and **on scroll**. The last one is not + optional here: Music's toolbar is a sticky band and the grid scrolls beneath + it, so a menu that survives a scroll is a menu now anchored to a different + album than the one it was opened on. +- flips its anchor when it would open past the viewport edge, which on a phone + is most of the time. +- submenus: a flyout on a fine pointer; **on a coarse pointer, a sheet**. A + two-level flyout (§10.3's "remove a track") has nowhere to go at 400 px, and + building it as a nested flyout and then discovering that on a phone is the + predictable version of this. + +Videos, Photos and Files all want this component eventually. It is deliberately +generic and lives outside `music-*.js`. + +### 10.2 The per-item menu — cover art and track rows + +On an album card (`AlbumCard`, `music-app.js:253`), on a detail-modal track row, +and on a flat-list row (`FlatTrackRow`, `music-app.js:370`): + +``` +Play → onPlayQueue(items, 0, source, 'replace') +Play next → 'next' +Add to queue → 'append' +──────────────────────── +Add to playlist ▸ Favourites + <other playlists, by name> + ──────────── + New playlist… +``` + +`items` is the album's tracks in track order, or the single track. The submenu +is drawn from the **manifest** (§4.3): instant, offline, no body fetched. +Favourites is always first, as asked, and is materialised on first use so it is +there on a fresh account with nothing created. "New playlist…" at the bottom +costs one item and covers the commonest way a playlist comes to exist — wanting +to put something in one. + +**One structural blocker, found in the existing markup.** `music-track-row` is a +`<button>` (`music-app.js:312`, `:373`, `music-player.js:167`) and `music-card` +is a `<div onClick>`. A dots `<button>` cannot nest inside a `<button>` — it is +invalid HTML and browsers reparent it, which breaks the row. The row must become +a `<div class="music-track-row">` holding a title `<button>` (the play action, +keeping today's keyboard behaviour) and a trailing dots `<button>`. Mechanical, +touches three call sites, and is the kind of thing that turns into "why does the +menu open behind the row" if it is discovered during implementation instead of +here. + +### 10.3 The single toolbar button + +One `tb-btn` in `.video-toolbar` in `music-app.js` — the sticky band +(`style.css` "Sticky chrome": `.sticky-chrome > .video-toolbar` pins under +`--nav-h + --chrome-h`), so it is reachable at any scroll position, as asked. +A new `playlist` glyph in `icon.js` next to `music`. One button, one icon, no +label — the toolbar already wraps to three rows on a phone +(`style.css:2557`) and has no room for a word. + +The menu, in the requested order: + +``` +Load a playlist ▸ <list> → replace the queue, play +Save the current queue as… <name prompt> ← §10.4 +New playlist… <name prompt> +Delete a playlist ▸ <list> → inline confirm +Remove a track from a playlist ▸ <list> ▸ <tracks> +──────────────────────────────── +Sync now ← §7 +``` + +Two additions to what was asked, both stated so they can be refused: **"Save the +current queue as…"** is placed at 2 because it is a creation verb and belongs +next to the other one; **"Sync now"** is §7's explicit sweep, which has to live +somewhere and this is the only playlist surface there is. + +Rules the list items carry: + +- **Load** replaces (§9.1) and starts playing. It fetches the body first if + `body_rev` says the local copy is stale, which is the one place a playlist + action can block — show the spinner the page already has. +- **New** takes a name only, as asked; tracks are attached afterwards. The name + must be unique *for this user*, checked against the manifest, case-folded with + the same `foldKey` the Music grouping already uses (`music-app.js`, exported) + rather than a second normaliser — a copy keeps agreeing until one of them + changes. +- **Delete** never lists Favourites (§5.1). It confirms inline; not + `window.confirm()`, which blocks the SPA's event loop and looks like a browser + alert in an Electron shell. +- **Remove a track** is the two-level one, and is a **sheet on mobile** + (§10.1): pick the playlist, then its tracklist with a trash button per row. + That sheet is also where reordering lands if O4 is ever taken up, which is + another reason not to build it as a flyout. + +### 10.4 Saving the current queue + +Reachable from two places, deliberately: the **queue panel** (`QueuePanel`, +`music-player.js:136`) gets a button in its top bar, because that panel is where +"the current queue" is a thing the user can see; and the toolbar menu mirrors it +for discoverability. + +It saves `order.map(i => tracks[i])` — **play order, not insertion order**. If +shuffle is on, that freezes the shuffle, which is what "save what I am listening +to" means. Entries pass through §5.3's whitelist on the way out. + +### 10.5 i18n + +Every string through `t()` and into all ten `static/locales/*.js`; +`test_locales.py` holds them to `en.js`'s key set. Favourites' display name is +`t('playlists.favorites')` — the reserved id `"favorites"` is what is stored, +never the localised name, or an account that switches language grows a second +favourites list. + +--- + +## 11. Unavailability — three different things, and the request conflates two + +| state | how it is known | already handled? | +|---|---|---| +| **Root unavailable** — a disk not mounted, group up and saying so | `nodeRoots[].available`, filtered into `availableEntries` (`group-page.js:716`) | yes — those entries never reach the UI | +| **Group unreachable** — no node answers | only by dialing | partly — see below | +| **File gone** — indexed, then deleted or moved | only at fetch | yes — as a fetch failure | + +### 11.1 Adding is never blocked on reachability + +The request asks that a track from an unreachable group not be added. Applied to +**playlists** that is the wrong rule and would make them lossy in exactly the +case they exist for: a playlist is durable, and a group that is off tonight is +back tomorrow. Refusing the add loses the user's intent permanently to recover +from a condition that lasts an evening. Worse, "is this group reachable" is only +answerable by dialing, and dialing to add is precisely the cost §7 refuses. + +So: **adding to a playlist never checks reachability, and never dials.** The +entry came from a rendered list, which came from an index this client holds. + +### 11.2 Adding to the queue does not dial either — playback skips + +Same reasoning, shorter: `play` / `play next` / `append` add what was on screen. +Loading a playlist loads all of it, greying entries whose group is in the known- +down set (the same set `search-page.js:535`'s `isDown` already keeps) without +removing them — a queue whose contents depend on network weather is worse than +one with a few dimmed rows. + +The skipping then happens where it can actually be known, at play time, and +**most of it is already built**: `advancePastFailure` (`music-player.js:215`) +was written for exactly this and is why a corrupt file between two good ones +does not stall an album. + +### 11.3 The one change: separate a bad file from a down group + +`MAX_CONSECUTIVE_FAILURES = 5` is sized for a corrupt file among good ones. A +playlist whose next six tracks all come from one offline group hits the bound +and stops, with an error, and the user reads that as "the playlist is broken". +That is a regression created by playlists, in code that is correct today. + +Split the two failures, because they are different properties of different +things: + +- **a decode / playback failure** is a property of *that file* — keep today's + bounded counter exactly as it is; +- **a connect / transport failure** is a property of *that group* — mark the + `groupId` down for the session, advance past **every** queued track from that + group in one step, and reset the counter. + +If that empties the queue, stop **once**, with a stated count — "3 groups +unreachable, 42 tracks skipped" — which is the same honest reporting +`fetchAllIndexes` does, and is what the 5-bound was protecting in the first +place. A group marked down is retried only on an explicit user action (pressing +play or next), never automatically, so a 300-track playlist does not re-dial a +dead node 300 times. `transport.waitForReconnect()` already covers the transient +within-group case and is untouched. --- -## 9. Security — per adversary +## 12. Security — per adversary | Claim | Passive hub | Active hub | Malicious node operator | Another member | |---|---|---|---|---| | Playlist contents (which tracks, which groups) | ✅ never transmitted to the hub | ✅ never transmitted to the hub | sealed under a key derived from the passphrase; the node holds ciphertext only | ✅ never served to anyone but the authenticated owner (`user_id` from the session) | -| Existence of playlists / how many | ✅ | ✅ | visible — one row, `stored_at`, and a padded length (§4.3) | ✅ | -| Editing timing | ✅ | ✅ | visible for writes reaching *that* node | ✅ | -| Integrity of the merged state | — | — | can serve stale or nothing; cannot lower the merged `rev` (§5.4) | — | +| How many playlists, and their relative sizes | ✅ | ✅ | **visible, and more so than in the first draft** — one row per playlist, each with its own `stored_at` and padded length | ✅ | +| Editing timing, per playlist | ✅ | ✅ | visible for writes reaching *that* node | ✅ | +| Integrity of the merged state | — | — | can serve stale or nothing; cannot lower the merged `rev` (§6.4) | — | | Deleting a playlist | — | — | can delete its own copy; other nodes and the local copy survive it | — | | Which tracks are actually played | — | — | already visible — the node serves the bytes | — | +**What the split costs, stated rather than buried.** One blob leaked one length +and one timestamp. N+1 blobs leak N+1 of each: an operator can see that this +account has nine playlists, that one of them is much larger than the rest, and +which single one was edited at 23:40. That is more than before. It is accepted +because the alternative is rewriting and re-uploading the entire collection on +every star (§4.3), which leaks the same timing anyway while costing an order of +magnitude more bandwidth — and because playlist *names* and *contents* remain +sealed throughout. + +**Compression before encryption, named and dismissed.** Compressing then sealing +makes the ciphertext length a function of the plaintext's redundancy, which is +the CRIME/BREACH shape. It does not apply here in the form that matters: there +is no adversary-chosen input mixed with a secret in one compression context per +request. The residual is a one-bit-per-user-action oracle — an operator who gets +the user to add a file *they named* to a playlist, and watches the length delta, +learns whether a string they chose already appeared. The 4 KB padding (§4.3) +means that oracle needs thousands of coordinated user actions to yield one +string. Written down, not defended further. + **The claim this design supports:** playlists add **no new authorization boundary and no new key hierarchy**. The node already stores an opaque -per-account object for this same account under this same key material; this -is a second payload of an existing kind, and the hub is not involved at all. +per-account object for this same account under this same key material; this is a +second payload of an existing kind, and the hub is not involved at all. -**The claim it must not make:** that a node cannot lose or withhold a -playlist. It can. The property is convergence with a local authority -(§5.4/§5.5), not durability guaranteed by any node. +**The claim it must not make:** that a node cannot lose or withhold a playlist. +It can. The property is convergence with a local authority (§6.4/§6.5), not +durability guaranteed by any node. -**One thing §4.2's fallback changes.** Playing a track from group B because -group A's node is offline means operator B, not operator A, sees that play. -Both already host that file for this account and already see its other plays, -so nothing new is learned by anyone — but it is a substitution of *observer*, -it is not obvious from the UI, and so it is written down here rather than -discovered later. +**One thing §5.3's fallback changes.** Playing a track from group B because +group A's node is offline means operator B, not operator A, sees that play. Both +already host that file for this account and already see its other plays, so +nothing new is learned by anyone — but it is a substitution of *observer*, it is +not obvious from the UI, and so it is written down here rather than discovered +later. --- -## 10. Open items — deliberately deferred +## 13. Open items — deliberately deferred | # | Item | Why not now | |---|---|---| -| O1 | Sharing a playlist with other group members | **Explicitly out of scope** (settled before this document). It is not an extension: a shared playlist is group state, sealed under the GEK, with concurrent writers — which makes §5's per-playlist LWW insufficient and an OR-Set over tracks mandatory. If it is ever wanted, reopen §5, not §3 | +| O1 | Sharing a playlist with other group members | **Explicitly out of scope.** Not an extension: a shared playlist is group state, sealed under the GEK, with concurrent writers — which makes §6's per-playlist LWW insufficient and an OR-Set over tracks mandatory. If it is ever wanted, reopen §6, not §3 | | O2 | A second copy wrapped under the account recovery key | `bundle_enc_recovery` is the exact precedent and the column is reserved in §3.3, so this is a client-side change alone when wanted. Not built now because a forgotten passphrase already strands more than playlists | -| O3 | Smart/auto playlists (by artist, by year, recently added) | These are queries over the cached index and need no storage at all. Genuinely a separate feature, and cheaper than this one | -| O4 | Ordering conflicts resolved better than LWW | Reordering is rare and losing a reorder is survivable; losing an added track is not, and per-playlist `rev` already prevents that for one user. Revisit only with O1 | -| O5 | Export / import a playlist as a file | Trivial once §8.1 exists (it is `JSON.stringify` of a merge unit) and worth doing, but it is not what makes playlists work across devices | -| O6 | Tombstone collection driven by an explicit per-node acknowledged `rev` | §5.3's rule is adequate at this scale; a real garbage collector matters at thousands of deletions, which is not a real state | +| O3 | Smart / auto playlists (by artist, by year, recently added) | Queries over the cached index; they need no storage at all. Genuinely a separate feature, and cheaper than this one | +| O4 | Reordering a playlist, and conflicts better than LWW | Losing a reorder is survivable; losing an added track is not, and per-playlist `rev` already prevents that. The drag surface is §10.3's sheet when it comes | +| O5 | Export / import a playlist as a file | Trivial once §14.1 exists (it is `JSON.stringify` of one body) and worth doing, but it is not what makes playlists work across devices | +| O6 | Tombstone collection driven by an explicit per-node acknowledged `rev` | §6.3's rule is adequate at this scale; a real collector matters at thousands of deletions, which is not a real state | +| O7 | A `/playlists` page and sidebar entry | **Dropped from this design, not deferred lightly.** The first draft proposed it; the requirement is a menu in Music's toolbar, and Music is already mounted in both places that matter (§10). A separate page would be a third surface showing the same list, and the first one to go stale | +| O8 | Per-queue persistence — restoring the current queue after a reload | Different object with a different lifetime: the queue is ephemeral and per-device, and `localStorage` is its right home, not §3's blob. Worth doing; not this | + +--- + +## 14. Client side + +### 14.1 Modules + +| file | contents | why separate | +|---|---|---| +| `playlists.js` | manifest + body load / merge / save, tombstones, id-path repair, compress, seal, open, the §5.3 whitelist | pure data layer, **no UI** — the merge is the one part that can be properly tested and must not be entangled with a component | +| `queue-ops.js` | the `{tracks, order, pos}` reducer (§9.3) | same reason, and it is the part that can regress the player | +| `menu.js` | `<Menu>`, `useMenu()` — anchored menu, submenus, coarse-pointer sheets | generic; Videos, Photos and Files want it too | +| `playlist-menu.js` | the toolbar button, its menu, and the create / delete / remove-track / save-queue modals | ~300 lines of modal that do not belong in `music-app.js` | + +Edited: `music-app.js` (toolbar button, per-item menus, the `<button>` → `<div>` +row fix), `music-player.js` (the reducer, the `op` branch, the split failure +counters, the queue panel's save button), `app.js` (`op` through +`handlePlayQueue`, the `groupTransportRef` guard), `hub-client.js` (IDB v2), +`keyderive.js` (the HKDF handle), `icon.js` (a `playlist` glyph), `style.css`. +`search-page.js` needs **nothing** — it already mounts `MusicApp`. + +### 14.2 The IndexedDB detail that will otherwise be missed + +`hub-client.js` opens `meshbay` at `IDB_VERSION = 1` (`hub-client.js:15`) with a +single store, `group_indexes`. Adding `playlists` means **bumping to 2** and +handling it in the existing `onupgradeneeded`, which currently creates one store +and would otherwise never run again. A store that is never created throws on +first access, at a point far from the version constant. + +One store holding the manifest and the bodies, keyed by `kind`, so §8.1's +namespace is the IndexedDB key as well and nothing has to translate between two +schemes. + +### 14.3 Checklist, per `MESHBAY_DESIGN.md` §9.4 + +1. The four new files above, plus the edits. +2. Nothing for caching: every file under `static/` feeds the `/a/<hash>/` + fingerprint. +3. `test_hook_ordering.py` (`STATIC_FILES`) and `test_transport_contracts.py` + (`SPLIT_FILES`) — all four new files. +4. i18n keys in all ten `static/locales/*.js`; `test_locales.py` holds them to + `en.js`'s key set. +5. `npm run sync-ui` in `meshbay-client`, confirmed reported — CI fails if the + copy drifts. +6. No `apps.js` registry entry and no `ALLOWED_APPS` change: this is not a group + application, it is a menu inside one that already exists. --- -## 11. Acceptance before shipping +## 15. Acceptance before shipping + +**No-regression first**, because §9.2 changes the player's queue state and +nothing else in this document can be judged until that is known to be clean: + +1. Baseline counts **before any edit**, per `CLAUDE.md`: the hub suite has + pre-existing browser failures, and a count taken afterwards proves nothing. +2. Play an album, let it auto-advance across at least three tracks, with + prefetch observed warming ahead. Then the same with shuffle on, toggled + mid-album, and confirm the playing track does not restart. +3. `tests/harness/queue_ops.mjs` runs the real reducer through §9.3's six + scenarios, including **two appends in one tick**, and fails if the reducer is + replaced by the three-`useState` version. + +**Then the feature:** -1. `tests/harness/playlist_merge.mjs` runs the real `merge()` through the - five divergences in §8.1, including the resurrect attempt, and **fails +4. `tests/harness/playlist_merge.mjs` runs the real `merge()` through five + divergences — two devices, one node offline for each in turn, a delete on one + side and an edit on the other, a resurrect attempt, a `rev` tie — and **fails with the tombstone rule removed**. Check that, or the test is decoration. -2. A round trip through a real node: seal, `user_blob_store`, restart the - daemon, `user_blob_fetch`, open. Confirms the blob survives the process, - not just the test. -3. Read `bundles.db` back and confirm the playlist plaintext is not in it — - the same check `test_chat_key_storage.py` already makes for epoch keys, - for the same reason. A plaintext table beside it is the obvious thing to - write and would collapse the whole claim silently. -4. Two browsers, one account, one node: edit in each, converge, and confirm - both agree. Then repeat with the node stopped between the two edits, and - confirm neither edit is lost when it comes back. -5. Open `/playlists` with **every** node offline and confirm the page renders - in full — names, artists, durations, entries greyed — because that is the - requirement §4.2 exists for and it cannot be unit-tested meaningfully. -6. Confirm the size cap refuses rather than truncates, and that the refusal - reaches the client as a stated reason rather than a bare `error`. -7. Confirm sign-in still runs Argon2id exactly **once** after §3.4's change - (measure it; the budget is the 650 ms already recorded in draft-v5 §7.1). +5. A round trip through a real node: seal, `user_blob_store`, restart the daemon, + `user_blob_fetch`, open. Confirms the blob survives the process, not just the + test. Repeat for a body and for the manifest. +6. Read `bundles.db` back and confirm no playlist plaintext is in it — the same + check `test_chat_key_storage.py` already makes for epoch keys, for the same + reason. A plaintext table beside it is the obvious thing to write and would + collapse the whole claim silently. +7. Two browsers, one account, one node: edit in each, converge, confirm both + agree. Repeat with the node stopped between the two edits, and confirm + neither edit is lost when it comes back. +8. Open Music with **every** node offline and confirm every menu in §10 draws in + full — playlist names, track titles, artists, durations, entries greyed — + because that is the requirement §5.3 exists for and it cannot be unit-tested + meaningfully. +9. Build a playlist of ~2 000 tracks, confirm the sealed body is under the 1 MB + cap, and **measure the bytes written when one track is starred** — it must be + the Favourites body plus the manifest, not the collection. That number is + what §4.3 is for. +10. Confirm each cap refuses rather than truncates, and that the refusal reaches + the client as a stated reason rather than a bare `error`. +11. Confirm sign-in still runs Argon2id exactly **once** after §3.4's change + (measure it; the budget is the 650 ms already recorded in draft-v5 §7.1). +12. A playlist spanning three groups, one of them stopped: confirm it loads + whole, plays, skips the dead group's tracks **in one step** rather than + erroring on the sixth, and reports the count. +13. Create a playlist from inside a group, then open Search → Music and confirm + the same playlist is in the menu there (requirement 6) — on the same device + immediately, on a second device after it has opened any group. |