diff options
Diffstat (limited to 'docs/playlists.md')
| -rw-r--r-- | docs/playlists.md | 516 |
1 files changed, 516 insertions, 0 deletions
diff --git a/docs/playlists.md b/docs/playlists.md new file mode 100644 index 0000000..aef2366 --- /dev/null +++ b/docs/playlists.md @@ -0,0 +1,516 @@ +# MeshBay — Playlists (design) + +> Status: **proposal**, not implemented. This is M4 in `docs/musicbay.md` §9, +> which deferred it for the right reason: *"a genuinely new category of +> per-account node state, not covered by anything E9 already enumerates — +> needs its own design pass (ownership, sync across devices, whether it's +> node-local or something else)"*. This document is that pass. +> +> Read `docs/musicbay.md` first — Music is built, and this adds nothing to +> its playback path. Read `docs/refactoring-search.md` second: the +> cross-group consolidation this feature needs already exists there, and +> most of the work is recognising that. +> +> **Scope, settled before writing this:** a playlist belongs to **one +> account** and is never shared with other group members. That answer is what +> keeps §5 small; see §10 for what changes if it is ever reversed. +> +> Follows the project convention: every claim names the adversary it holds +> against (§9). + +--- + +## 0. What was asked, in one paragraph + +Somewhere to keep a user's playlists, so that they survive a cache clear and +turn up on that person's other devices — and so that one playlist may hold +albums from **several different groups on several different nodes**, the way +the Search page already searches a consolidated view. With the constraint, +stated up front, that nodes go offline for an evening or for a month and that +this must not corrupt anything. + +--- + +## 1. What this design does not reopen + +- **Views over the index, never a catalogue** (`desktop-client-v1.md` §6.10, + draft-v6 §2.7). A playlist is a list of *references*; it creates no second + identity for a file and no server-side database of content. +- **Nothing about content reaches the hub** (H7, draft-v6 §2.5). §3.1. +- **No new streaming path.** `musicbay.md` §2.2 stands untouched: a track is + fetched through `pipelinedDownload` and handed to `<audio>`. Playlists add + no node-side playback code whatsoever. +- **Node-side derived data lives in the node's own `data_dir`**, never beside + a shared file (`mediacenter.md` §2). The blob in §3.3 goes in `bundles.db`. +- **Identity keys are per node** (`per-node-identity-v1.md`). §3.4 exists + entirely because of this, and getting it wrong is the fastest way to ship a + playlist that cannot be read from the second device it was invented for. + +--- + +## 2. Three problems, and only one of them is hard + +| | Problem | Where it is solved | +|---|---|---| +| A | Somewhere durable to put a few KB per account | §3 — several acceptable answers, one clearly best | +| B | How to **name** a track that lives in one group on one node, inside a list that spans several | §4.2 — where the silent failures live | +| C | Reconciling N copies when nodes are ON and OFF | §5 — the stated fear, and it is well founded *only* for the wrong granularity | + +The request reads as a storage problem. Storage is the easy third of it. + +--- + +## 3. Where the state lives + +### 3.1 Not the hub, and the rule is narrower than "nothing on the hub" + +The hub *does* hold small per-account state: `user_preferences` +(`db/models.py:303`), behind an allowlist of keys that already includes +`music_keep_screen_on` and `default_tab:<group_id>`. So the operative rule is +not "the hub stores nothing about a user". It is: + +> **No content metadata on the hub.** + +A playlist is, literally, a list of blake3 hashes of private-group files — +plus, if it is to render while nodes are offline (§4.2), their titles and +artists. That is the exact object H7 removed from the hub, and it is what +`CLAUDE.md` already pins one size smaller for the video resume position: +*"nothing new learns what you watch"*. + +**An encrypted blob on the hub** is technically trivial and is still refused. +The precedent is exact and recent: keypair bundles *were* hub-stored, and T3 +reduction phases 1 and 2 deliberately moved them onto nodes. Putting a +different per-account blob back would undo a migration that has already been +paid for. The residual cost is real even sealed — the hub learns the +cardinality (how many playlists), the size (roughly how many tracks) and the +timing of every write (when this account listens and edits), and it retains +ciphertext indefinitely under a passphrase-derived key, which is C4's own +argument. + +If a future need makes hub involvement genuinely attractive, the only +acceptable shape is a **version vector** — `playlists_rev: 7` in +`user_preferences`, telling a client that some node is stale — never the +payload. Even that is probably not worth its own migration; §6 gets the same +result with no hub change at all. + +### 3.2 Not node-to-node + +Refused, and not on cost grounds. Nodes do not know each other, share no +authenticated channel, and `replication.py` is legacy public-content code +that has nothing to do with this. Beyond the protocol that would have to be +invented, it leaks the thing this architecture is most careful about: +node A would learn that this account also uses node B — that two unrelated +operators host the same person. Per-node identity exists precisely so that +this correlation is unavailable (`per-node-identity-v1.md`). + +### 3.3 The node, as an opaque per-account blob — a shape already built + +The node already stores an encrypted per-account object it cannot read: the +keypair bundle (`bundle_store.py`, table `keypair_bundles`, MNP ops +`KEYPAIR_BUNDLE_STORE` / `_FETCH` / `_DELETE`). Playlists are the same shape +with a different payload, so this introduces **no new trust boundary**: the +node is not being asked to hold anything of a kind it does not already hold +for this same account. + +A new table in the same `bundles.db`: + +```sql +CREATE TABLE IF NOT EXISTS user_blobs ( + user_id TEXT NOT NULL, + kind TEXT NOT NULL, -- "playlists" today; the column is the + -- reason a second one needs no migration + rev INTEGER NOT NULL, + blob_enc TEXT NOT NULL, + blob_enc_recovery TEXT, -- reserved, §10 O2 — see below + stored_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (user_id, kind) +); +``` + +`blob_enc_recovery` is declared now and left NULL. `bundle_store.py` already +carries `_migrate_keypair_recovery` — a `PRAGMA table_info` dance — for +exactly this column added to exactly this kind of table one release late. +`CREATE TABLE IF NOT EXISTS` never adds a column, which is the node-local +twin of the `create_all()` lesson in `CLAUDE.md`. Declaring the slot costs +nothing today and removes the migration entirely. + +### 3.4 The key — the one thing that must not be got wrong + +Identity keys are **per node**. A blob encrypted under one is unreadable from +every other node, which is the precise opposite of the requirement. + +The only secret this account holds *everywhere* is the **bundle key**: +Argon2id over the passphrase, rederived at every sign-in on every device +(`keyderive.js:145`), and already the key that opens the per-node identity +bundles. So: + +``` +playlist_key = HKDF-SHA256(bundle_key_v2, info = "meshbay:playlists:v1") +``` + +Three consequences, each of which is a line of code somewhere: + +- **`deriveEncryptionKey` must return an HKDF handle as well as the AES-GCM + one.** Today it imports the 32 Argon2 bytes non-extractably as `AES-GCM` + with `['encrypt','decrypt']`, from which nothing can be derived. Import the + *same* `out.hash` a second time as `HKDF` with `['deriveKey']`. One Argon2 + run, two handles, no extra cost on the 650 ms sign-in path. Do **not** add a + second Argon2 call, and do **not** encrypt playlists directly under the + bundle key with a different AAD: `groupbox.py` already sets the convention + ("*Never reuse `chunk_key_aes` with a pseudo-file for this*") and + purpose-separated subkeys are what it looks like here. +- **v2 only.** `bundleKey` is `{v2, v1}` so that a bundle written before the + KDF changed can still be opened. Playlists are new; there is no legacy blob + and therefore no v1 branch. One less thing that can silently take the wrong + fork. +- **The nonce is 96 random bits, never a counter.** Two devices of one account + derive the *same* playlist key — that is the whole point — so a counter + would repeat. This is the same reasoning already recorded for chat subkeys + in `CLAUDE.md`'s module table, and it is safe for the same reason. + +A passphrase change re-derives the key and the blob must be re-encrypted on +the next write, exactly as the keypair bundle already is ("*re-encrypted on +the next backup*"). + +**AAD** = `"user_blob|playlists|<user_id>"`, mirroring +`groupbox.associated_data`. It binds the ciphertext to its owner and its +kind. It does not, and cannot, prevent rollback — §5.4. + +--- + +## 4. The data model + +### 4.1 The blob + +Plaintext-before-sealing, msgpack (same encoding as everything else on MNP): + +``` +{ + "v": 1, + "playlists": { + "<playlist_id>": { + "name": "Evening", + "rev": 7, + "device": "<device_pk_ed25519, first 16 hex>", + "updated_at": 1757000000, -- display only, never a merge input + "deleted": false, + "tracks": [ <entry>, ... ] + }, + ... + } +} +``` + +`playlist_id` is a client-generated UUID, with one reserved value: +`"favorites"`. **Favourites is a playlist**, not a second mechanism — deciding +that now is what stops a parallel store being built next to this one in three +months. + +### 4.2 A track reference, and why each field is there + +``` +{ "group_id": ..., "file_id": <blake3 hex>, "hash_version": 1|2, + "path": "Some Artist/An Album/03 - A Track.flac", + "title": ..., "artist": ..., "album": ..., "duration": 214 } +``` + +A bare hash is not enough, and each field prevents a specific failure: + +- **`group_id`** — the player resolves its connection per track from + `entry.groupId` (`music-player.js:288`). Without it there is nothing to + dial, and a `file_id` alone has no meaning outside a group. +- **`title` / `artist` / `album` / `duration`, denormalised** — this is not + redundancy, it is the core of the design. With them a playlist renders + **completely** with every node offline, unplayable entries greyed out, in + the same spirit as the Search page reporting its `unreachable` list rather + than failing. Without them, an offline playlist is a column of hex strings — + and *that* is the incoherence the request is worried about. It is a + display problem, and it is solved by copying four small strings. +- **`hash_version`** — the index already has two hashing schemes + (`protocol.py:292`: 1 = whole file, 2 = 45 MB sample). A re-hash would + orphan every entry in every playlist, silently and all at once. +- **`path`** — content addressing survives a move; a path survives a + re-encode. Keeping both means either can repair the other: on a sight of + the live index, an entry whose `file_id` is absent but whose `path` matches + has its id rewritten in place (and vice versa), once, on the client. + +Rehydration is by design cheap: a playlist entry is a subset of the +`IndexEntry`-plus-`groupId` shape the player already consumes, so +`onPlayQueue(tracks, startIndex)` takes it unchanged. + +**Availability bonus, close to free.** `source-merge.js` exists because the +same content appears in more than one group. At play time, if the entry's own +`group_id` has no reachable node but the same `file_id` appears in another +cached index whose node is up, play it from there. That is a global playlist +that heals itself when one operator's machine is off, reusing the +fold-on-content-hash logic already written for Search. Default on; see §9 for +the one thing it changes. + +### 4.3 Size + +A thousand tracks at ~200 bytes each is ~200 KB before compression. That is +small, and it is also an unbounded write primitive pointed at someone else's +disk, so: + +- The node **caps and refuses**, never truncates. 256 KB of `blob_enc` is + generous for the shape above. (Noted in passing: `_do_keypair_bundle_store` + (`webrtc_server.py:1056`) has no cap at all today. Out of scope here, worth + its own line somewhere.) +- The client **pads the plaintext up to the next 4 KB** before sealing. The + ciphertext length otherwise tells the operator roughly how many tracks this + account has collected. Cheap, and it is the only metadata this design leaks + to a node that the node cannot already see. + +--- + +## 5. Merge — the part that has to be right + +The stated fear is correct **for a single blob under last-writer-wins**: node +A is off while an edit is made, node B is off while the next one is, and one +edit disappears with nothing to show for it. Four rules remove it. + +### 5.1 The unit is a playlist, not the collection + +The blob is a map keyed by `playlist_id`, and merging is per key. Two +playlists edited on two devices never collide, which is the overwhelmingly +common case for one person with two or three devices. + +### 5.2 `rev`, never the wall clock + +Each playlist carries a monotonic `rev` and the `device` that last wrote it. +Merge takes the higher `rev`; a tie is broken by the lexicographically +smaller `device`, so every device reaches the same answer without talking to +any other. `updated_at` is carried for display and is **never** read by the +merge — clocks across devices are not trustworthy, and a clock-based merge +fails roughly one time in twenty, which is the frequency at which this +codebase's history says a defect ships. + +### 5.3 A deletion is a tombstone, never an absence + +`deleted: true`, kept. Absence must mean "this copy is older than the one +that created it". Otherwise a node rehomed after three weeks **resurrects +every deleted playlist** — this is the single most likely defect in the whole +design, it looks like a sync working correctly right up until it doesn't, and +it deserves its own named test. Tombstones are collected only when every +known node reports a `rev` at or above the deleting one, which for a +single-node account is immediate and for a multi-node one is eventual; a +tombstone is ~40 bytes, so there is no hurry. + +### 5.4 Rollback, and why local-first answers it + +AEAD authenticates a blob; it does not stop a node handing back an older one +it still has (or a fresh one it never received). The defence is that **the +client is the authority**: the merged state lives in the client's own +IndexedDB, and merge takes the maximum `rev` across *local plus every node +answering*. A stale or lying node can only lose the tie. It can never lower +the merged state, because the local copy is one of the inputs. + +This is what makes an offline node a non-event rather than a hazard. Nodes +are backups and a transport. They are not the source of truth, and no node +being reachable at all still leaves every playlist correct and, thanks to +§4.2, fully legible. + +### 5.5 What is genuinely lost, stated plainly + +Device 1 makes an edit, reaches no node, and is then lost or cleared: that +edit is gone. This is the exposure of any offline-first application, it is +not fixable without a durable always-reachable writer (which is §3.1, and +refused), and it is still enormously better than today, where the same edit +is lost on a cache clear regardless of what was online. + +--- + +## 6. Sync — and the point is that it adds no dialing + +A sweep of every group's node costs 10 s per unreachable one; the Search page +does it deliberately, batched three at a time, because the user asked it to. +Playlists must not do that at sign-in. + +**Sync rides on connections that were happening anyway.** Whenever a +transport to any node is open for any other reason — opening a group, the +Search page's sweep, the music pool dialing to play a track — the client +piggybacks a `user_blob_fetch`, merges, and sends `user_blob_store` back if +that node's copy is behind. `ConnectionPool` (`search-page.js:36`) already +holds up to `MAX_POOL_SIZE` (12) live connections and hands them out by +group, so this is a hook, not a new mechanism. + +Two additions on top: + +- **An explicit "Sync now"** on the Playlists page, which does the Search-style + sweep and reports which nodes it could not reach — the same honest + reporting `fetchAllIndexes` already does. +- **On sign-in, nothing.** The local copy is authoritative and complete + (§5.4); the first group opened will reconcile. + +A device that only ever opens one group therefore only ever converges with +one node. That is correct and not a defect: convergence is eventual, and the +copy the user is looking at is right the whole time. + +--- + +## 7. Protocol and node-side implementation + +### 7.1 MNP — additive, MINOR bump (2.1) + +``` +user_blob_store { kind, rev, blob_enc } client → node +user_blob_fetch { kind } client → node +user_blob_resp { kind, rev, blob_enc|null } node → client +``` + +Modelled on `KEYPAIR_BUNDLE_*` in every respect, including that the node +stores and returns an opaque string. `kind` is validated against a small +allowlist (`{"playlists"}`) so the table does not become an arbitrary +key/value store for whatever a client feels like writing. + +Every reply carries `req_id` through the ordinary `_send` path. This is not +optional and does not need re-arguing: `CLAUDE.md` records at length what +arrival-order matching costs, and the victim is never the request that was +answered wrongly. + +### 7.2 Node side + +| Piece | Where | What | +|---|---|---| +| Storage | `meshbay_node/bundle_store.py` | `user_blobs` table (§3.3), `store_user_blob` / `fetch_user_blob`, same shape as `store_keypair` / `fetch_keypair` | +| Handlers | `transport/webrtc_server.py` | `_do_user_blob_store` / `_do_user_blob_fetch`, `self._user_id` from the authenticated session (NS6), never from the message | +| Cap | same | Refuse over 256 KB with a stated reason; refuse an unknown `kind` | +| Audit | same | `user_blob_store` / `user_blob_fetch` events, same as `keypair_bundle_store` already logs | + +`user_id` comes from the session, exactly as `_do_keypair_bundle_store` takes +it — a `user_id` in the message body would let any member read or overwrite +any other member's blob, which is the C5 shape one size down. + +**No hub change. No change to indexing, streaming, transcoding, or the GEK.** + +### 7.3 What the node can and cannot do with it + +It can delete it, lose it with its disk, or serve a stale copy (§5.4 covers +the last). Durability is "the local copy, plus N node copies", with no +guarantee from any single one — which for an account on one node means the +local copy matters. Worth one line in the UI, not a warning dialog. + +--- + +## 8. Client side + +### 8.1 A pure module, deliberately separate from the UI + +`static/playlists.js` holds the data layer and **no UI**: load/merge/save, +the tombstone rules, the id/path repair, seal and open. This is not tidiness. +Source-reading tests are weak evidence and are most of what this repo can do +for the SPA — but a pure function over two objects can be *executed*, the way +`tests/harness/mse_harness.mjs` lifts the real player functions and runs +them. The merge is the one part of this feature that can be properly tested, +so it must not be entangled with a component. + +`tests/harness/playlist_merge.mjs` runs the real `merge()` over scripted +divergences: two devices, one node offline for each in turn, a delete on one +side and an edit on the other, a resurrect attempt, a `rev` tie. Model the +environment, never the code under test. + +### 8.2 Where it appears + +Playlists cross groups, so they do not belong to a group's Music tab: + +- **New route `/playlists`** and `static/playlists-page.js`, a sibling of + `/search`, plus a sidebar entry. It renders from the **cached** indexes + (`getAllCachedIndexes`), so it opens instantly and works with everything + offline; it dials only to play, or on "Sync now". +- **"Add to playlist"** in `music-app.js` (album and track level) and in + `search-page.js`'s music results — both already have the entry in hand, + with `groupId` attached. +- **The player bar needs nothing.** It is already at shell level in `app.js`, + already resolves a connection per track from `entry.groupId`, and already + crosses groups within one queue. This is worth stating loudly, because the + cross-group requirement reads like the hard part and is in fact already + built (`musicbay.md` §9b: *"The player needed no change"*). + +### 8.3 The IndexedDB detail that will otherwise be missed + +`hub-client.js` opens `meshbay` at `IDB_VERSION = 1` with a single store, +`group_indexes`. Adding a `playlists` store means **bumping to 2** and +handling it in the existing `onupgradeneeded`, which currently creates one +store and would otherwise never run again. A store that is never created +throws on first access, at a point far from the version constant. + +### 8.4 Checklist, per `apps.md` + +1. `playlists.js` (data) and `playlists-page.js` (UI). +2. Route + sidebar entry in `app.js`. +3. `webapp.py`'s `_ASSETS` tuple — both new files. +4. `test_hook_ordering.py` (`STATIC_FILES`) and + `test_transport_contracts.py` (`SPLIT_FILES`) — both new files. +5. i18n keys in all ten `static/locales/*.js`; `test_locales.py` holds them + to `en.js`'s key set. +6. `npm run sync-ui` in `meshbay-client`, confirmed reported. + +No `apps.js` registry entry and no `ALLOWED_APPS` change: this is not a group +application. It is a page, like Search. + +--- + +## 9. Security — per adversary + +| Claim | Passive hub | Active hub | Malicious node operator | Another member | +|---|---|---|---|---| +| Playlist contents (which tracks, which groups) | ✅ never transmitted to the hub | ✅ never transmitted to the hub | sealed under a key derived from the passphrase; the node holds ciphertext only | ✅ never served to anyone but the authenticated owner (`user_id` from the session) | +| Existence of playlists / how many | ✅ | ✅ | visible — one row, `stored_at`, and a padded length (§4.3) | ✅ | +| Editing timing | ✅ | ✅ | visible for writes reaching *that* node | ✅ | +| Integrity of the merged state | — | — | can serve stale or nothing; cannot lower the merged `rev` (§5.4) | — | +| Deleting a playlist | — | — | can delete its own copy; other nodes and the local copy survive it | — | +| Which tracks are actually played | — | — | already visible — the node serves the bytes | — | + +**The claim this design supports:** playlists add **no new authorization +boundary and no new key hierarchy**. The node already stores an opaque +per-account object for this same account under this same key material; this +is a second payload of an existing kind, and the hub is not involved at all. + +**The claim it must not make:** that a node cannot lose or withhold a +playlist. It can. The property is convergence with a local authority +(§5.4/§5.5), not durability guaranteed by any node. + +**One thing §4.2's fallback changes.** Playing a track from group B because +group A's node is offline means operator B, not operator A, sees that play. +Both already host that file for this account and already see its other plays, +so nothing new is learned by anyone — but it is a substitution of *observer*, +it is not obvious from the UI, and so it is written down here rather than +discovered later. + +--- + +## 10. Open items — deliberately deferred + +| # | Item | Why not now | +|---|---|---| +| O1 | Sharing a playlist with other group members | **Explicitly out of scope** (settled before this document). It is not an extension: a shared playlist is group state, sealed under the GEK, with concurrent writers — which makes §5's per-playlist LWW insufficient and an OR-Set over tracks mandatory. If it is ever wanted, reopen §5, not §3 | +| O2 | A second copy wrapped under the account recovery key | `bundle_enc_recovery` is the exact precedent and the column is reserved in §3.3, so this is a client-side change alone when wanted. Not built now because a forgotten passphrase already strands more than playlists | +| O3 | Smart/auto playlists (by artist, by year, recently added) | These are queries over the cached index and need no storage at all. Genuinely a separate feature, and cheaper than this one | +| O4 | Ordering conflicts resolved better than LWW | Reordering is rare and losing a reorder is survivable; losing an added track is not, and per-playlist `rev` already prevents that for one user. Revisit only with O1 | +| O5 | Export / import a playlist as a file | Trivial once §8.1 exists (it is `JSON.stringify` of a merge unit) and worth doing, but it is not what makes playlists work across devices | +| O6 | Tombstone collection driven by an explicit per-node acknowledged `rev` | §5.3's rule is adequate at this scale; a real garbage collector matters at thousands of deletions, which is not a real state | + +--- + +## 11. Acceptance before shipping + +1. `tests/harness/playlist_merge.mjs` runs the real `merge()` through the + five divergences in §8.1, including the resurrect attempt, and **fails + with the tombstone rule removed**. Check that, or the test is decoration. +2. A round trip through a real node: seal, `user_blob_store`, restart the + daemon, `user_blob_fetch`, open. Confirms the blob survives the process, + not just the test. +3. Read `bundles.db` back and confirm the playlist plaintext is not in it — + the same check `test_chat_key_storage.py` already makes for epoch keys, + for the same reason. A plaintext table beside it is the obvious thing to + write and would collapse the whole claim silently. +4. Two browsers, one account, one node: edit in each, converge, and confirm + both agree. Then repeat with the node stopped between the two edits, and + confirm neither edit is lost when it comes back. +5. Open `/playlists` with **every** node offline and confirm the page renders + in full — names, artists, durations, entries greyed — because that is the + requirement §4.2 exists for and it cannot be unit-tested meaningfully. +6. Confirm the size cap refuses rather than truncates, and that the refusal + reaches the client as a stated reason rather than a bare `error`. +7. Confirm sign-in still runs Argon2id exactly **once** after §3.4's change + (measure it; the budget is the 650 ms already recorded in draft-v5 §7.1). |