path: root/docs/playlists.md

# MeshBay — Playlists (design)

> Status: **proposal**, not implemented. This was deferred out of the Music
> application's design,
> which deferred it for the right reason: *"a genuinely new category of
> per-account node state, not covered by anything E9 already enumerates —
> needs its own design pass (ownership, sync across devices, whether it's
> node-local or something else)"*. This document is that pass.
>
> Read `MESHBAY_DESIGN.md` §9.8 first — Music is built, and this adds nothing
> to its playback path. Read §9.11 second: the
> cross-group consolidation this feature needs already exists there, and
> most of the work is recognising that.
>
> **Scope, settled before writing this:** a playlist belongs to **one
> account** and is never shared with other group members. That answer is what
> keeps §5 small; see §10 for what changes if it is ever reversed.
>
> Follows the project convention: every claim names the adversary it holds
> against (§9).

---

## 0. What was asked, in one paragraph

Somewhere to keep a user's playlists, so that they survive a cache clear and
turn up on that person's other devices — and so that one playlist may hold
albums from **several different groups on several different nodes**, the way
the Search page already searches a consolidated view. With the constraint,
stated up front, that nodes go offline for an evening or for a month and that
this must not corrupt anything.

---

## 1. What this design does not reopen

- **Views over the index, never a catalogue** (`MESHBAY_DESIGN.md` §6.5, §9.1,
  draft-v6 §2.7). A playlist is a list of *references*; it creates no second
  identity for a file and no server-side database of content.
- **Nothing about content reaches the hub** (H7, draft-v6 §2.5). §3.1.
- **No new streaming path.** `MESHBAY_DESIGN.md` §9.8 stands untouched: a track is
  fetched through `pipelinedDownload` and handed to `<audio>`. Playlists add
  no node-side playback code whatsoever.
- **Node-side derived data lives in the node's own `data_dir`**, never beside
  a shared file (`MESHBAY_DESIGN.md` §6.5). The blob in §3.3 goes in `bundles.db`.
- **Identity keys are per node** (`MESHBAY_DESIGN.md` §3.2). §3.4 exists
  entirely because of this, and getting it wrong is the fastest way to ship a
  playlist that cannot be read from the second device it was invented for.

---

## 2. Three problems, and only one of them is hard

| | Problem | Where it is solved |
|---|---|---|
| A | Somewhere durable to put a few KB per account | §3 — several acceptable answers, one clearly best |
| B | How to **name** a track that lives in one group on one node, inside a list that spans several | §4.2 — where the silent failures live |
| C | Reconciling N copies when nodes are ON and OFF | §5 — the stated fear, and it is well founded *only* for the wrong granularity |

The request reads as a storage problem. Storage is the easy third of it.

---

## 3. Where the state lives

### 3.1 Not the hub, and the rule is narrower than "nothing on the hub"

The hub *does* hold small per-account state: `user_preferences`
(`db/models.py:303`), behind an allowlist of keys that already includes
`music_keep_screen_on` and `default_tab:<group_id>`. So the operative rule is
not "the hub stores nothing about a user". It is:

> **No content metadata on the hub.**

A playlist is, literally, a list of blake3 hashes of private-group files —
plus, if it is to render while nodes are offline (§4.2), their titles and
artists. That is the exact object H7 removed from the hub, and it is what
`CLAUDE.md` already pins one size smaller for the video resume position:
*"nothing new learns what you watch"*.

**An encrypted blob on the hub** is technically trivial and is still refused.
The precedent is exact and recent: keypair bundles *were* hub-stored, and T3
reduction phases 1 and 2 deliberately moved them onto nodes. Putting a
different per-account blob back would undo a migration that has already been
paid for. The residual cost is real even sealed — the hub learns the
cardinality (how many playlists), the size (roughly how many tracks) and the
timing of every write (when this account listens and edits), and it retains
ciphertext indefinitely under a passphrase-derived key, which is C4's own
argument.

If a future need makes hub involvement genuinely attractive, the only
acceptable shape is a **version vector** — `playlists_rev: 7` in
`user_preferences`, telling a client that some node is stale — never the
payload. Even that is probably not worth its own migration; §6 gets the same
result with no hub change at all.

### 3.2 Not node-to-node

Refused, and not on cost grounds. Nodes do not know each other, share no
authenticated channel, and `replication.py` is legacy public-content code
that has nothing to do with this. Beyond the protocol that would have to be
invented, it leaks the thing this architecture is most careful about:
node A would learn that this account also uses node B — that two unrelated
operators host the same person. Per-node identity exists precisely so that
this correlation is unavailable (`MESHBAY_DESIGN.md` §3.2).

### 3.3 The node, as an opaque per-account blob — a shape already built

The node already stores an encrypted per-account object it cannot read: the
keypair bundle (`bundle_store.py`, table `keypair_bundles`, MNP ops
`KEYPAIR_BUNDLE_STORE` / `_FETCH` / `_DELETE`). Playlists are the same shape
with a different payload, so this introduces **no new trust boundary**: the
node is not being asked to hold anything of a kind it does not already hold
for this same account.

A new table in the same `bundles.db`:

```sql
CREATE TABLE IF NOT EXISTS user_blobs (
    user_id     TEXT NOT NULL,
    kind        TEXT NOT NULL,          -- "playlists" today; the column is the
                                        -- reason a second one needs no migration
    rev         INTEGER NOT NULL,
    blob_enc    TEXT NOT NULL,
    blob_enc_recovery TEXT,             -- reserved, §10 O2 — see below
    stored_at   TEXT NOT NULL DEFAULT (datetime('now')),
    PRIMARY KEY (user_id, kind)
);
```

`blob_enc_recovery` is declared now and left NULL. `bundle_store.py` already
carries `_migrate_keypair_recovery` — a `PRAGMA table_info` dance — for
exactly this column added to exactly this kind of table one release late.
`CREATE TABLE IF NOT EXISTS` never adds a column, which is the node-local
twin of the `create_all()` lesson in `CLAUDE.md`. Declaring the slot costs
nothing today and removes the migration entirely.

### 3.4 The key — the one thing that must not be got wrong

Identity keys are **per node**. A blob encrypted under one is unreadable from
every other node, which is the precise opposite of the requirement.

The only secret this account holds *everywhere* is the **bundle key**:
Argon2id over the passphrase, rederived at every sign-in on every device
(`keyderive.js:145`), and already the key that opens the per-node identity
bundles. So:

```
playlist_key = HKDF-SHA256(bundle_key_v2, info = "meshbay:playlists:v1")
```

Three consequences, each of which is a line of code somewhere:

- **`deriveEncryptionKey` must return an HKDF handle as well as the AES-GCM
  one.** Today it imports the 32 Argon2 bytes non-extractably as `AES-GCM`
  with `['encrypt','decrypt']`, from which nothing can be derived. Import the
  *same* `out.hash` a second time as `HKDF` with `['deriveKey']`. One Argon2
  run, two handles, no extra cost on the 650 ms sign-in path. Do **not** add a
  second Argon2 call, and do **not** encrypt playlists directly under the
  bundle key with a different AAD: `groupbox.py` already sets the convention
  ("*Never reuse `chunk_key_aes` with a pseudo-file for this*") and
  purpose-separated subkeys are what it looks like here.
- **v2 only.** `bundleKey` is `{v2, v1}` so that a bundle written before the
  KDF changed can still be opened. Playlists are new; there is no legacy blob
  and therefore no v1 branch. One less thing that can silently take the wrong
  fork.
- **The nonce is 96 random bits, never a counter.** Two devices of one account
  derive the *same* playlist key — that is the whole point — so a counter
  would repeat. This is the same reasoning already recorded for chat subkeys
  in `CLAUDE.md`'s module table, and it is safe for the same reason.

A passphrase change re-derives the key and the blob must be re-encrypted on
the next write, exactly as the keypair bundle already is ("*re-encrypted on
the next backup*").

**AAD** = `"user_blob|playlists|<user_id>"`, mirroring
`groupbox.associated_data`. It binds the ciphertext to its owner and its
kind. It does not, and cannot, prevent rollback — §5.4.

---

## 4. The data model

### 4.1 The blob

Plaintext-before-sealing, msgpack (same encoding as everything else on MNP):

```
{
  "v": 1,
  "playlists": {
    "<playlist_id>": {
      "name":       "Evening",
      "rev":        7,
      "device":     "<device_pk_ed25519, first 16 hex>",
      "updated_at": 1757000000,        -- display only, never a merge input
      "deleted":    false,
      "tracks": [ <entry>, ... ]
    },
    ...
  }
}
```

`playlist_id` is a client-generated UUID, with one reserved value:
`"favorites"`. **Favourites is a playlist**, not a second mechanism — deciding
that now is what stops a parallel store being built next to this one in three
months.

### 4.2 A track reference, and why each field is there

```
{ "group_id": ..., "file_id": <blake3 hex>, "hash_version": 1|2,
  "path": "Some Artist/An Album/03 - A Track.flac",
  "title": ..., "artist": ..., "album": ..., "duration": 214 }
```

A bare hash is not enough, and each field prevents a specific failure:

- **`group_id`** — the player resolves its connection per track from
  `entry.groupId` (`music-player.js:288`). Without it there is nothing to
  dial, and a `file_id` alone has no meaning outside a group.
- **`title` / `artist` / `album` / `duration`, denormalised** — this is not
  redundancy, it is the core of the design. With them a playlist renders
  **completely** with every node offline, unplayable entries greyed out, in
  the same spirit as the Search page reporting its `unreachable` list rather
  than failing. Without them, an offline playlist is a column of hex strings —
  and *that* is the incoherence the request is worried about. It is a
  display problem, and it is solved by copying four small strings.
- **`hash_version`** — the index already has two hashing schemes
  (`protocol.py:292`: 1 = whole file, 2 = 45 MB sample). A re-hash would
  orphan every entry in every playlist, silently and all at once.
- **`path`** — content addressing survives a move; a path survives a
  re-encode. Keeping both means either can repair the other: on a sight of
  the live index, an entry whose `file_id` is absent but whose `path` matches
  has its id rewritten in place (and vice versa), once, on the client.

Rehydration is by design cheap: a playlist entry is a subset of the
`IndexEntry`-plus-`groupId` shape the player already consumes, so
`onPlayQueue(tracks, startIndex)` takes it unchanged.

**Availability bonus, close to free.** `source-merge.js` exists because the
same content appears in more than one group. At play time, if the entry's own
`group_id` has no reachable node but the same `file_id` appears in another
cached index whose node is up, play it from there. That is a global playlist
that heals itself when one operator's machine is off, reusing the
fold-on-content-hash logic already written for Search. Default on; see §9 for
the one thing it changes.

### 4.3 Size

A thousand tracks at ~200 bytes each is ~200 KB before compression. That is
small, and it is also an unbounded write primitive pointed at someone else's
disk, so:

- The node **caps and refuses**, never truncates. 256 KB of `blob_enc` is
  generous for the shape above. (Noted in passing: `_do_keypair_bundle_store`
  (`webrtc_server.py:1056`) has no cap at all today. Out of scope here, worth
  its own line somewhere.)
- The client **pads the plaintext up to the next 4 KB** before sealing. The
  ciphertext length otherwise tells the operator roughly how many tracks this
  account has collected. Cheap, and it is the only metadata this design leaks
  to a node that the node cannot already see.

---

## 5. Merge — the part that has to be right

The stated fear is correct **for a single blob under last-writer-wins**: node
A is off while an edit is made, node B is off while the next one is, and one
edit disappears with nothing to show for it. Four rules remove it.

### 5.1 The unit is a playlist, not the collection

The blob is a map keyed by `playlist_id`, and merging is per key. Two
playlists edited on two devices never collide, which is the overwhelmingly
common case for one person with two or three devices.

### 5.2 `rev`, never the wall clock

Each playlist carries a monotonic `rev` and the `device` that last wrote it.
Merge takes the higher `rev`; a tie is broken by the lexicographically
smaller `device`, so every device reaches the same answer without talking to
any other. `updated_at` is carried for display and is **never** read by the
merge — clocks across devices are not trustworthy, and a clock-based merge
fails roughly one time in twenty, which is the frequency at which this
codebase's history says a defect ships.

### 5.3 A deletion is a tombstone, never an absence

`deleted: true`, kept. Absence must mean "this copy is older than the one
that created it". Otherwise a node rehomed after three weeks **resurrects
every deleted playlist** — this is the single most likely defect in the whole
design, it looks like a sync working correctly right up until it doesn't, and
it deserves its own named test. Tombstones are collected only when every
known node reports a `rev` at or above the deleting one, which for a
single-node account is immediate and for a multi-node one is eventual; a
tombstone is ~40 bytes, so there is no hurry.

### 5.4 Rollback, and why local-first answers it

AEAD authenticates a blob; it does not stop a node handing back an older one
it still has (or a fresh one it never received). The defence is that **the
client is the authority**: the merged state lives in the client's own
IndexedDB, and merge takes the maximum `rev` across *local plus every node
answering*. A stale or lying node can only lose the tie. It can never lower
the merged state, because the local copy is one of the inputs.

This is what makes an offline node a non-event rather than a hazard. Nodes
are backups and a transport. They are not the source of truth, and no node
being reachable at all still leaves every playlist correct and, thanks to
§4.2, fully legible.

### 5.5 What is genuinely lost, stated plainly

Device 1 makes an edit, reaches no node, and is then lost or cleared: that
edit is gone. This is the exposure of any offline-first application, it is
not fixable without a durable always-reachable writer (which is §3.1, and
refused), and it is still enormously better than today, where the same edit
is lost on a cache clear regardless of what was online.

---

## 6. Sync — and the point is that it adds no dialing

A sweep of every group's node costs 10 s per unreachable one; the Search page
does it deliberately, batched three at a time, because the user asked it to.
Playlists must not do that at sign-in.

**Sync rides on connections that were happening anyway.** Whenever a
transport to any node is open for any other reason — opening a group, the
Search page's sweep, the music pool dialing to play a track — the client
piggybacks a `user_blob_fetch`, merges, and sends `user_blob_store` back if
that node's copy is behind. `ConnectionPool` (`search-page.js:36`) already
holds up to `MAX_POOL_SIZE` (12) live connections and hands them out by
group, so this is a hook, not a new mechanism.

Two additions on top:

- **An explicit "Sync now"** on the Playlists page, which does the Search-style
  sweep and reports which nodes it could not reach — the same honest
  reporting `fetchAllIndexes` already does.
- **On sign-in, nothing.** The local copy is authoritative and complete
  (§5.4); the first group opened will reconcile.

A device that only ever opens one group therefore only ever converges with
one node. That is correct and not a defect: convergence is eventual, and the
copy the user is looking at is right the whole time.

---

## 7. Protocol and node-side implementation

### 7.1 MNP — additive, MINOR bump (2.1)

```
user_blob_store  { kind, rev, blob_enc }        client → node
user_blob_fetch  { kind }                       client → node
user_blob_resp   { kind, rev, blob_enc|null }   node   → client
```

Modelled on `KEYPAIR_BUNDLE_*` in every respect, including that the node
stores and returns an opaque string. `kind` is validated against a small
allowlist (`{"playlists"}`) so the table does not become an arbitrary
key/value store for whatever a client feels like writing.

Every reply carries `req_id` through the ordinary `_send` path. This is not
optional and does not need re-arguing: `CLAUDE.md` records at length what
arrival-order matching costs, and the victim is never the request that was
answered wrongly.

### 7.2 Node side

| Piece | Where | What |
|---|---|---|
| Storage | `meshbay_node/bundle_store.py` | `user_blobs` table (§3.3), `store_user_blob` / `fetch_user_blob`, same shape as `store_keypair` / `fetch_keypair` |
| Handlers | `transport/webrtc_server.py` | `_do_user_blob_store` / `_do_user_blob_fetch`, `self._user_id` from the authenticated session (NS6), never from the message |
| Cap | same | Refuse over 256 KB with a stated reason; refuse an unknown `kind` |
| Audit | same | `user_blob_store` / `user_blob_fetch` events, same as `keypair_bundle_store` already logs |

`user_id` comes from the session, exactly as `_do_keypair_bundle_store` takes
it — a `user_id` in the message body would let any member read or overwrite
any other member's blob, which is the C5 shape one size down.

**No hub change. No change to indexing, streaming, transcoding, or the GEK.**

### 7.3 What the node can and cannot do with it

It can delete it, lose it with its disk, or serve a stale copy (§5.4 covers
the last). Durability is "the local copy, plus N node copies", with no
guarantee from any single one — which for an account on one node means the
local copy matters. Worth one line in the UI, not a warning dialog.

---

## 8. Client side

### 8.1 A pure module, deliberately separate from the UI

`static/playlists.js` holds the data layer and **no UI**: load/merge/save,
the tombstone rules, the id/path repair, seal and open. This is not tidiness.
Source-reading tests are weak evidence and are most of what this repo can do
for the SPA — but a pure function over two objects can be *executed*, the way
`tests/harness/mse_harness.mjs` lifts the real player functions and runs
them. The merge is the one part of this feature that can be properly tested,
so it must not be entangled with a component.

`tests/harness/playlist_merge.mjs` runs the real `merge()` over scripted
divergences: two devices, one node offline for each in turn, a delete on one
side and an edit on the other, a resurrect attempt, a `rev` tie. Model the
environment, never the code under test.

### 8.2 Where it appears

Playlists cross groups, so they do not belong to a group's Music tab:

- **New route `/playlists`** and `static/playlists-page.js`, a sibling of
  `/search`, plus a sidebar entry. It renders from the **cached** indexes
  (`getAllCachedIndexes`), so it opens instantly and works with everything
  offline; it dials only to play, or on "Sync now".
- **"Add to playlist"** in `music-app.js` (album and track level) and in
  `search-page.js`'s music results — both already have the entry in hand,
  with `groupId` attached.
- **The player bar needs nothing.** It is already at shell level in `app.js`,
  already resolves a connection per track from `entry.groupId`, and already
  crosses groups within one queue. This is worth stating loudly, because the
  cross-group requirement reads like the hard part and is in fact already
  built (the cross-group merge already covers it — `MESHBAY_DESIGN.md` §9.11).

### 8.3 The IndexedDB detail that will otherwise be missed

`hub-client.js` opens `meshbay` at `IDB_VERSION = 1` with a single store,
`group_indexes`. Adding a `playlists` store means **bumping to 2** and
handling it in the existing `onupgradeneeded`, which currently creates one
store and would otherwise never run again. A store that is never created
throws on first access, at a point far from the version constant.

### 8.4 Checklist, per `MESHBAY_DESIGN.md` §9.4

1. `playlists.js` (data) and `playlists-page.js` (UI).
2. Route + sidebar entry in `app.js`.
3. `webapp.py`'s `_ASSETS` tuple — both new files.
4. `test_hook_ordering.py` (`STATIC_FILES`) and
   `test_transport_contracts.py` (`SPLIT_FILES`) — both new files.
5. i18n keys in all ten `static/locales/*.js`; `test_locales.py` holds them
   to `en.js`'s key set.
6. `npm run sync-ui` in `meshbay-client`, confirmed reported.

No `apps.js` registry entry and no `ALLOWED_APPS` change: this is not a group
application. It is a page, like Search.

---

## 9. Security — per adversary

| Claim | Passive hub | Active hub | Malicious node operator | Another member |
|---|---|---|---|---|
| Playlist contents (which tracks, which groups) | ✅ never transmitted to the hub | ✅ never transmitted to the hub | sealed under a key derived from the passphrase; the node holds ciphertext only | ✅ never served to anyone but the authenticated owner (`user_id` from the session) |
| Existence of playlists / how many | ✅ | ✅ | visible — one row, `stored_at`, and a padded length (§4.3) | ✅ |
| Editing timing | ✅ | ✅ | visible for writes reaching *that* node | ✅ |
| Integrity of the merged state | — | — | can serve stale or nothing; cannot lower the merged `rev` (§5.4) | — |
| Deleting a playlist | — | — | can delete its own copy; other nodes and the local copy survive it | — |
| Which tracks are actually played | — | — | already visible — the node serves the bytes | — |

**The claim this design supports:** playlists add **no new authorization
boundary and no new key hierarchy**. The node already stores an opaque
per-account object for this same account under this same key material; this
is a second payload of an existing kind, and the hub is not involved at all.

**The claim it must not make:** that a node cannot lose or withhold a
playlist. It can. The property is convergence with a local authority
(§5.4/§5.5), not durability guaranteed by any node.

**One thing §4.2's fallback changes.** Playing a track from group B because
group A's node is offline means operator B, not operator A, sees that play.
Both already host that file for this account and already see its other plays,
so nothing new is learned by anyone — but it is a substitution of *observer*,
it is not obvious from the UI, and so it is written down here rather than
discovered later.

---

## 10. Open items — deliberately deferred

| # | Item | Why not now |
|---|---|---|
| O1 | Sharing a playlist with other group members | **Explicitly out of scope** (settled before this document). It is not an extension: a shared playlist is group state, sealed under the GEK, with concurrent writers — which makes §5's per-playlist LWW insufficient and an OR-Set over tracks mandatory. If it is ever wanted, reopen §5, not §3 |
| O2 | A second copy wrapped under the account recovery key | `bundle_enc_recovery` is the exact precedent and the column is reserved in §3.3, so this is a client-side change alone when wanted. Not built now because a forgotten passphrase already strands more than playlists |
| O3 | Smart/auto playlists (by artist, by year, recently added) | These are queries over the cached index and need no storage at all. Genuinely a separate feature, and cheaper than this one |
| O4 | Ordering conflicts resolved better than LWW | Reordering is rare and losing a reorder is survivable; losing an added track is not, and per-playlist `rev` already prevents that for one user. Revisit only with O1 |
| O5 | Export / import a playlist as a file | Trivial once §8.1 exists (it is `JSON.stringify` of a merge unit) and worth doing, but it is not what makes playlists work across devices |
| O6 | Tombstone collection driven by an explicit per-node acknowledged `rev` | §5.3's rule is adequate at this scale; a real garbage collector matters at thousands of deletions, which is not a real state |

---

## 11. Acceptance before shipping

1. `tests/harness/playlist_merge.mjs` runs the real `merge()` through the
   five divergences in §8.1, including the resurrect attempt, and **fails
   with the tombstone rule removed**. Check that, or the test is decoration.
2. A round trip through a real node: seal, `user_blob_store`, restart the
   daemon, `user_blob_fetch`, open. Confirms the blob survives the process,
   not just the test.
3. Read `bundles.db` back and confirm the playlist plaintext is not in it —
   the same check `test_chat_key_storage.py` already makes for epoch keys,
   for the same reason. A plaintext table beside it is the obvious thing to
   write and would collapse the whole claim silently.
4. Two browsers, one account, one node: edit in each, converge, and confirm
   both agree. Then repeat with the node stopped between the two edits, and
   confirm neither edit is lost when it comes back.
5. Open `/playlists` with **every** node offline and confirm the page renders
   in full — names, artists, durations, entries greyed — because that is the
   requirement §4.2 exists for and it cannot be unit-tested meaningfully.
6. Confirm the size cap refuses rather than truncates, and that the refusal
   reaches the client as a stated reason rather than a bare `error`.
7. Confirm sign-in still runs Argon2id exactly **once** after §3.4's change
   (measure it; the budget is the 650 ms already recorded in draft-v5 §7.1).