diff options
Diffstat (limited to 'docs/refactor-groups.md')
| -rw-r--r-- | docs/refactor-groups.md | 916 |
1 files changed, 0 insertions, 916 deletions
diff --git a/docs/refactor-groups.md b/docs/refactor-groups.md deleted file mode 100644 index 9e0ecf1..0000000 --- a/docs/refactor-groups.md +++ /dev/null @@ -1,916 +0,0 @@ -# Groups Refactor — Per-Root Permissions & App Plugin Architecture - -> **Superseded by `MESHBAY_DESIGN.md`.** This was the roots and app-settings refactor; its design -> content now lives in §6.2 (RO/RW, eject), §9.3 (enablement and settings). -> -> It is kept because code comments, tests and other documents cite its -> sections and its labels, and because it records reasoning a synthesis -> compresses. **Where it disagrees with `MESHBAY_DESIGN.md`, the design -> document is right; where either disagrees with the code, the code is.** -> `MESHBAY_DESIGN.md` §16 maps every section reference here onto its -> replacement, and §13 defines every label. - -> Status: **Complete** (2026-09-07). All three phases built, reviewed and -> tested against a running node. -> -> This is the most significant refactoring of the project. It changes how roots -> are permissioned, how group applications are configured, and how the Settings -> and Create Group pages are structured. -> -> §7b, §7c and §7d record what each phase's review found and where the plan -> below was wrong. Several entries are rules rather than one-off fixes; §7d -> also lists what a person still has to test by hand. - ---- - -## 0. Summary of changes - -| Area | Before | After | -|---|---|---| -| Root permissions | One root marked `upload=True`; binary `member_upload` toggle per group | Each root is **RO** (default) or **RW**; multiple RW roots allowed; fully RO group is valid | -| Upload policy | Separate section in Settings; `member_upload` signed op | **Removed.** RO/RW on the root is the mechanism. Files shows Upload only on RW roots. Chat disables attachments when its configured directory is not on a RW root | -| Root metadata | `name, path, kind, upload, direct` | `name, path, kind, writable, removable, direct` | -| Removable flag | Not tracked | Per-root boolean, set by operator. Enables the **eject/plug** button for safe device removal | -| Safe eject | Auto-detected only (`path.is_dir()`) | Operator-initiated eject button in Settings AND Files root view. `ejected` state distinct from `available`. Indexer freezes entries, no data loss | -| Files app | Can be disabled | **Always enabled**, transparently. Not shown in the app toggle list | -| App selection at group creation | Checkbox list of all apps | **Removed.** Files is enabled automatically; other apps are configured later in Settings | -| Settings layout | Monolithic: apps checkboxes, TMDB, MusicBrainz, directories, uploads — all in `group-settings.js` | **Structured:** Shared directories (top, expanded) → per-app sections (each with toggle + icon + title, collapsed, settings hidden until enabled) → Scan/Danger/Devices/Members | -| App settings code | All inlined in `group-settings.js` (1338 lines) | Each app has `ui/<APP_NAME>-app-settings.js`; loaded by discovery (presence of the file) | -| App enablement UI | One "Applications" section with checkboxes | Each app is a collapsible section with its own toggle in the title. The toggle is the enablement control | -| Folder picker | Flat `<select>` with depth-indented names | **Folder tree popup**: modal, root icons, `[+]` expand, sub-directory selection | -| Server-side app ops | Per-app functions in `ops.py` (`set_video_root`, `set_audio_root`, `set_photo_roots`, ...) | Generic `set_app_directory()` / `set_app_directories()` + app-specific wrappers where needed | -| CLI | `group add --dir --upload-dir`; `member upload` concept | `group add --dir [--writable]`; `root add/remove/set` with `--writable`/`--read-only`/`--removable` | -| MNP | 1.0 | 1.1 (additive: new fields on roots, new generic app-directory messages). 1.0 peers still work | - ---- - -## 1. Design decisions - -### 1.1 RO/RW replaces upload + member_upload - -The current model has two orthogonal mechanisms: (a) one root is the upload target, -(b) `member_upload` toggles whether non-operators can upload there. The new model -collapses both into one property per root: **writable**. - -- `writable = false` (default): the root is read-only for everyone, including the - operator via the UI. Content is placed there out-of-band (filesystem, rsync, USB). -- `writable = true`: any group member may upload to this root (into the quarantine - subdirectory, same protections as today — allowlist, no overwrite, size cap). - -Multiple roots can be writable. Zero can be writable (fully read-only group). The -operator controls which roots are RW by toggling a switch in the shared directories -table. - -**What this removes:** -- The `member_upload` toggle and its `OP_MEMBER_UPLOAD` signed op -- The `member_upload` / `member_upload_ack` MNP message types (deprecated, still - parsed for backward compat) -- The Uploads section in Settings -- The concept of "the upload root" (singular) - -**What this preserves:** -- The quarantine directory, filename allowlist, no-overwrite check, size cap -- The `_do_file_upload` handler in `webrtc_server.py` — now checks `writable` on - the target root instead of checking `upload` + `member_upload` -- The operator's ability to create a read-only group (set all roots RO) - -### 1.2 Files is always enabled - -`files` is removed from the toggleable app list. It is always present in -`enabled_apps` and cannot be disabled. The current ability to hide it was misleading: -MNP still permits root exploration regardless. The tab bar always shows Files. - -`apps.js` keeps `files` in `APPS` but marks it `alwaysEnabled: true`. The Settings -page skips it when rendering app toggle sections. - -### 1.3 Per-app settings files - -Each app that has configurable settings exports a settings component from -`ui/<APP_NAME>-app-settings.js`. The file is optional — an app with no settings -(like Files today) has no settings file and gets only a toggle. - -The `APPS` registry in `apps.js` gains an optional `Settings` field per entry, -imported from the corresponding settings file. The group-settings page iterates -`APPS`, skips `files`, and renders a collapsible section for each, with: - -- The app's monochrome icon + localized title in the section header -- A toggle switch in the header (disabled by default) -- The app's `Settings` component below, **hidden until the toggle is on** -- A Save button per app section (some saves trigger caching — TMDB, MusicBrainz) - -**Discovery mechanism:** In the browser context, "file discovery" is registration in -`apps.js`. Adding a new app means: write `<APP>-app.js` + `<APP>-app-settings.js`, -add one entry to `APPS` in `apps.js`, add the key to `ALLOWED_APPS` on the node. -Server-side enforcement via `ALLOWED_APPS` prevents client-side hacks from enabling -an unrecognized app. - -### 1.4 Folder tree widget - -A reusable modal popup (`FolderTreePicker`) that: - -- Appears centered on screen, semi-transparent backdrop -- Lists root directories at the top level, with the same folder icons as Files -- Each root can be expanded via `[+]` / collapsed via `[-]`, explorer-tree style -- Sub-directories load from the existing `nodeDirs` data (already available from the - file index, no new endpoint needed) -- Supports **single-select** mode (Chat) and **multi-select** mode (Videos, Music, - Photos) -- Shows the root's RO/RW badge next to each root name -- For apps that require RW (Chat): RO roots and their children are greyed out / - unselectable, with a tooltip explaining why -- OK / Cancel buttons at the bottom -- Returns the selected path(s) relative to the root (e.g., `Movies/Action`) - -The widget replaces the current flat `<select>` dropdowns in all app settings. It is -also usable in other UI contexts (the video player's folder navigation already does -something similar ad-hoc). - -### 1.5 Shared directories table - -A reusable component (`SharedDirectoriesTable`) used in both the group Settings page -and the Create Group wizard (developed once, shared). Features: - -- Borderless table, one row per root -- Columns: **Name** (with folder icon), **Path** (truncated with tooltip on hover), - **RW toggle** (switch, default off), **Removable toggle** (checkbox), - **Eject/Plug button** (visible only when removable is checked — see §1.5b), - **Delete button** (trash icon, with confirmation) -- Ejected roots show a distinct visual state: greyed-out row, eject icon replaced by - a plug icon -- The first root in the Create Group wizard defaults to RW -- A concise explanatory sentence at the top: "At least one directory is required. - Read-write directories accept uploads from group members." -- Add button: opens the native folder picker (Electron) or a path input (web, admin - only) -- Cannot delete the last root (refused with explanation) -- Each change is a signed operator op (`ROOT_ADD`, `ROOT_REMOVE`, or new - `ROOT_UPDATE` for toggling writable/removable on an existing root) - -### 1.5b Safe eject for removable devices - -**Problem.** An operator stores data on a USB drive. Unplugging it without warning -triggers the watchdog — file deletions propagate as though the operator erased an -entire library. The existing `refresh_availability()` auto-detects this and freezes -entries (good), but there is no way to eject cleanly before unplugging, and no way -to re-plug without a full rescan. - -**The `ejected` state.** A root has two independent runtime states: - -- `ejected` (bool, default `false`): operator-controlled, persisted in `roster.db`. - Set by clicking the eject button; cleared by clicking plug. -- `available` (bool, runtime): computed as `not ejected and is_live()`. This is - what clients and the indexer see. - -The distinction matters: when the operator clicks "eject" but hasn't physically -unplugged yet, `is_live()` returns `true` but `available` is `false` because -`ejected` is `true`. Without this, `refresh_availability()` would immediately -flip it back to available. - -**Eject flow:** - -1. Operator clicks the eject button (⏏) on a removable root -2. Confirmation dialog: "Eject *Movies*? Files from this directory will be - temporarily hidden to all members. You can safely unplug the device." -3. On confirm: `PUT /api/groups/{gid}/roots/{name}/eject` → `ops.eject_root()` -4. `ops.eject_root()`: sets `ejected = true` in `roster.db`, marks root - `available = false`, stops the watchdog observer for that root -5. The indexer **freezes** all entries from that root (existing behavior — no - deletions, no index updates, cached data preserved) -6. `index_sync` update propagates to connected peers: the root's `available` is - now `false` -7. All apps filter out entries from unavailable roots (Files already does this - partially — needs to be complete across Videos, Music, Photos) -8. The operator can now safely unplug the device - -**Plug flow:** - -1. Operator plugs the device back in and clicks the plug button (🔌) -2. `PUT /api/groups/{gid}/roots/{name}/plug` → `ops.plug_root()` -3. `ops.plug_root()`: checks `is_live()` first — if the path is not accessible, - refuses with "Directory not found. Is the device connected?" -4. On success: sets `ejected = false`, marks root `available = true`, restarts - the watchdog observer -5. The indexer **rescans the root** — its frozen entries are dropped and the - directory is read again. (The plan said "a reconciliation, not a full - rescan"; it is a rescan, deliberately. It is the same path a root coming - back from `refresh_availability` already took, and a device people carry - around can come back arbitrarily different — the hash cache means unchanged - files are not re-read, which is where the cost would have been.) -6. `index_sync` update propagates — entries reappear in all apps - -**The flag is persisted, and restored at startup.** `ejected` lives in -`roster.db` (`root_ejected:<folded name>`), not in `node.toml`: it is runtime -state, and an operator's hand-written config must not be rewritten because a USB -drive was unplugged. It has to survive a restart — a restart is exactly what an -operator does after noticing a drive fell off, and a flag that only lived in -memory would let the scan that follows read the empty mount point as an erased -library. `daemon._build_roots()` merges the two sources; it is the only place -that builds a `RootSet` for a group. - -**Auto-detection safety net.** If a `removable` root's path suddenly disappears -(operator unplugged without clicking eject): - -- `refresh_availability()` detects `is_live() = false` -- Because `removable = true`, it sets `ejected = true` automatically (as if the - operator had clicked eject), and reports it through the indexer's - `on_root_ejected` callback so the daemon writes it to `roster.db` — an - auto-eject that only existed in memory would be undone by the next restart -- Entries freeze, no deletions propagate -- The root stays in "ejected" state until the operator explicitly plugs it back - -For non-removable roots, the existing behavior is unchanged: `available` flips -based on `is_live()`, entries freeze when unavailable, rescan when available again. - -**Eject button in Files app.** In addition to the Settings table, an eject button -appears in the Files app root-level view, next to each removable root's name. This -provides quick access without navigating to Settings. Same confirmation dialog, -same API call. Ejected roots show as greyed-out with a plug icon to re-enable. - -**What is NOT deleted on eject:** - -- Index entries (frozen, not removed) -- TMDB / MusicBrainz cached metadata -- Video thumbnails in `media_cache` -- Chat message history referencing files on that root -- App directory configurations pointing to that root (but flagged as temporarily - invalid — the app shows a warning, not an error) - -**MNP message:** `ROOT_EJECT` / `ROOT_EJECT_ACK` and `ROOT_PLUG` / `ROOT_PLUG_ACK` -— signed operator ops, same pattern as `ROOT_UPDATE`. Broadcast to all connected -peers so they see the availability change immediately without waiting for the next -`index_sync`. - -### 1.6 Server-side normalization (ops.py) - -Two generic functions replace the per-app specific ones: - -```python -def set_app_directory(state, group_id, app_key, path, *, require_writable=False): - """Set a single directory for an app. Validates path is within a named root. - If require_writable, refuses paths under RO roots.""" - -def set_app_directories(state, group_id, app_key, paths, *, require_writable=False): - """Set multiple directories for an app. Same validation.""" -``` - -Existing functions (`set_video_root`, `set_audio_root`, `set_photo_roots`) become -thin wrappers calling the generic versions, preserving the current MNP message types -and roster keys for backward compat. New apps use the generic functions directly. - -### 1.7 Chat settings additions - -- **Directory picker** (single, RW-only): selects the directory for chat file - attachments. If no RW root exists, the picker shows an explanation and the - attachment button is disabled in the chat UI. If the selected directory's root is - later set to RO, the setting is flagged as invalid and attachments are disabled - until corrected. -- **Link preview toggle** (new, server-side): the operator can disable link previews - for the group. Stored in `roster.db` as `chat_link_preview` (default: enabled). - The node's `linkpreview.py` checks this setting before unfurling. The toggle is a - `ToggleSwitch` in the Chat settings section. - -### 1.8 Videos/Music/Photos settings changes - -**Videos:** -- Directory picker changes from single to **multi-directory** (via folder tree widget) -- TMDB settings section moved here from the monolithic settings -- TMDB API key field: no longer says "optional" or mentions the default key. - Instead, a prompt to sign up on TMDB with a direct link to generate a key -- Per-app Save button triggers TMDB cache sweep - -**Music:** -- Directory picker changes from single to **multi-directory** -- MusicBrainz settings section moved here -- Per-app Save button triggers MusicBrainz cache sweep - -**Photos:** -- Multi-directory picker (already multi, just moves to the folder tree widget) -- No third-party service settings - -### 1.9 Handshake ack changes - -The `handshake_ack` payload gains per-root metadata: - -```python -# Current -"roots": [{"name": "Movies", "path": "/mnt/movies", ...}] -"member_upload": True - -# New -"roots": [{"name": "Movies", "path": "/mnt/movies", "writable": False, "removable": True, ...}] -# member_upload removed (deprecated, still parsed by old clients) -``` - -For backward compatibility with MNP 1.0 peers: -- A 1.0 client that does not see `writable` on roots falls back to the old model - (root with `upload=True` is writable, `member_upload` from the ack controls access) -- A 1.1 node continues to send `member_upload` as a computed value: `True` if any - root is writable, `False` otherwise — so old clients behave sensibly -- `member_upload` is no longer writable via MNP ops; the node computes it from roots - -### 1.10 CLI changes - -As built. The group is a `--group` option rather than a positional, matching -every other verb in this CLI, and the negative flags are spelled `--no-writable` -/ `--no-removable` rather than `--read-only`, so each pair reads as one setting. - -``` -# Group creation (first root defaults to RW) -meshbay-node group add <name> --dir <path> # first root, RW -meshbay-node group add <name> --dir <path> --no-writable # first root, RO - -# Root management (--group is optional with one group configured) -meshbay-node root list [--group <name>] -meshbay-node root add <path> [--name <name>] [--writable] [--removable] -meshbay-node root remove <name> [--yes] -meshbay-node root set <name> --writable | --no-writable -meshbay-node root set <name> --removable | --no-removable -meshbay-node root eject <name> # safe eject -meshbay-node root plug <name> # re-plug - -# Deprecated (accepted with a warning) ---upload-dir → "use --writable on the target root instead" -member upload → removed; use 'root set --no-writable' / '--writable' -``` - -### 1.11 HelloWorld proof-of-concept - -A minimal app that validates the plugin architecture end to end: - -- `static/helloworld-app.js`: renders a greeting and lists files in its configured - directory -- `static/helloworld-app-settings.js`: single-directory picker (via folder tree - widget), no other settings -- Entry in `apps.js` with `key: "helloworld"`, icon, label, Component, Settings -- `ALLOWED_APPS` extended on the node -- No dedicated `helloworld.py` — uses the generic `set_app_directory()` function, - which is the whole point - -The HelloWorld app is **not shipped in production**. It lives in the tree as a -reference implementation and can be excluded from the build. Its value is proving -that the plugin mechanism works: zero changes to `group-settings.js`, -`group-page.js`, `webrtc_server.py` or `ops.py` to add it. - -### 1.12 Migration (existing nodes) - -A script in `QE/migration/` (not versioned) handles Fedora and Ubuntu nodes: - -**node.toml:** -- `upload = true` → `writable = true` -- `upload = false` (or absent) → `writable = false` -- Add `removable = false` to all roots that lack it -- Remove `upload_dir` from `[[groups]]` blocks (if present) - -**roster.db:** -- If `member_upload = "off"` for a group: set all that group's roots to - `writable = false` (the intent was "no uploads") -- Rename `video_root` → `video_directories` (wrap single value in a list) -- Rename `audio_root` → `audio_directories` (same) -- `photo_roots` → `photo_directories` (rename only) -- Remove `member_upload` rows -- Add `chat_link_preview = "true"` default for groups with chat enabled -- Add `chat_directory` for groups that had an upload root (default: the upload - root's path) - -**Protocol version:** -- MNP version file bumped to 1.1 - -**The script is idempotent** — running it twice is safe. - ---- - -## 2. Phase 1 — Root RO/RW model + Shared Directories UI - -**Goal:** Change the data model from `upload` to `writable`/`removable`, build the -shared directories table, restructure the top of Settings and the Create Group -wizard. Remove the Uploads section. Files always enabled. - -**Testable after this phase:** Create a group with RO/RW roots, toggle RW in -Settings, add/remove roots in the new table, see the Upload button appear/disappear -in Files based on the current root's writable flag, eject a removable root and verify -files disappear from all apps without data loss, plug it back and verify files -reappear, CLI works with new syntax including `root eject/plug`. - -### 2.1 Backend changes - -| File | Change | -|---|---| -| `config.py` | `RootSpec`: add `writable: bool = False`, `removable: bool = False`. Remove `upload` field. `__post_init__` migration: `upload=True` → `writable=True`. Parse new fields from node.toml | -| `roots.py` | `Root` dataclass: add `writable`, `removable`, `ejected`. Remove `upload`. `available` becomes a computed property: `not self.ejected and self.is_live()`. `_settle_upload_root()` removed. `RootSet.build()`: validate at least one root exists (no RW minimum). `refresh_availability()`: when a `removable` root's path disappears, auto-set `ejected=True` (safety net). Collision checks unchanged | -| `ops.py` | `add_root()`: accept `writable`, `removable` params. `remove_root()`: refuse removing last root (unchanged). `attach_group()`: first root defaults to `writable=True`. Remove `set_member_upload()`. New: `update_root()` for toggling writable/removable on an existing root (signed op `OP_ROOT_UPDATE`). New: `eject_root()` — sets `ejected=True`, stops watchdog for that root. New: `plug_root()` — checks `is_live()`, sets `ejected=False`, triggers reconciliation | -| `roster.py` | No schema change (generic key/value). Remove `member_upload` handling from `_apply_group_settings()` | -| `ui/app.py` | `POST /api/groups/{gid}/roots`: accept `writable`, `removable`. New: `PATCH /api/groups/{gid}/roots/{name}` → `ops.update_root()`. New: `PUT /api/groups/{gid}/roots/{name}/eject` → `ops.eject_root()`. New: `PUT /api/groups/{gid}/roots/{name}/plug` → `ops.plug_root()`. Remove `PUT /api/groups/{gid}/member-upload` | -| `daemon.py` (CLI) | New `root` subcommand: `add`, `remove`, `set`, `list`. `group add --dir` defaults to `writable=True`. Deprecate `--upload-dir` with warning. Remove `member upload` command | -| `webrtc_server.py` | Handshake ack: add `writable`/`removable`/`ejected` per root. Compute `member_upload` for backward compat. `_do_file_upload`: check `root.writable` instead of `root.upload` + `member_upload_allowed`. Handle `ROOT_UPDATE`, `ROOT_EJECT`, `ROOT_PLUG` MNP messages. Broadcast root availability changes to all connected peers. `ALLOWED_APPS`: add `"files"` to always-enabled set | -| `protocol.py` | New message types: `ROOT_UPDATE` / `ROOT_UPDATE_ACK`, `ROOT_EJECT` / `ROOT_EJECT_ACK`, `ROOT_PLUG` / `ROOT_PLUG_ACK`. `ROOT_ADD` gains `writable`, `removable` fields | -| `indexer.py` | `eject_root()`: stop the watchdog observer for that root, do NOT touch entries. `plug_root()`: restart observer, trigger reconciliation pass. `refresh_availability()`: auto-eject removable roots whose path disappears (set `ejected=True` instead of just flipping `available`) | -| `handshake.py` | MNP version → 1.1 (minor, additive) | - -### 2.2 Frontend changes - -| File | Change | -|---|---| -| `group-settings.js` | New `SharedDirectoriesTable` component (reusable). Includes eject/plug button per removable root. Move to top of settings (after Invite/Pair). Remove the Uploads toggle section. Remove the old Directories section (root management part — app root pickers stay for now). Calls loopback API for add/remove/update/eject/plug root | -| `create-group-page.js` | Replace app checkboxes with nothing (Files auto-enabled). Replace directory section with `SharedDirectoriesTable` (same component). First root defaults `writable=true`. Step 2: remove `set enabled_apps` call (Files is automatic) | -| `files-app.js` | Upload button visibility: check `currentRoot.writable && currentRoot.available`. Upload target: the root currently being browsed. Hide upload affordances on RO roots. Eject/plug button next to each removable root name in root-level view. Ejected roots greyed out with plug icon. Entries from unavailable roots filtered out of all views | -| `chat-app.js` | Attachment button: disabled if no RW root exists or if the chat directory's root is RO or unavailable. Tooltip explaining why | -| `group-page.js` | `nodeRoots` state: include `writable`/`removable`/`ejected` from handshake ack. Handle `ROOT_EJECT_ACK`/`ROOT_PLUG_ACK` broadcasts to update root state live. Remove `memberUpload` state. Remove `onUploadPolicy` callback | -| `video-app.js` | Filter entries: exclude files from unavailable roots | -| `music-app.js` | Filter entries: exclude files from unavailable roots | -| `photos-app.js` | Filter entries: exclude files from unavailable roots | -| `apps.js` | Add `alwaysEnabled: true` to `files` entry | -| `transport.js` | Parse new root fields from handshake ack | - -### 2.3 Localization - -New keys in `locales/*.js`: -- `sharedDirectories`, `sharedDirectoriesHint` ("At least one directory is - required...") -- `readOnly`, `readWrite`, `removableDevice` -- `uploadNotAvailableRO` (tooltip: "This directory is read-only") -- `ejectRoot`, `ejectRootConfirm` ("Eject *{name}*? Files from this directory - will be temporarily hidden...") -- `plugRoot`, `plugRootFailed` ("Directory not found. Is the device connected?") -- `rootEjected` (status label shown on ejected roots) -- Deprecation: `uploadToggle*` keys can be removed - ---- - -## 3. Phase 2 — App Settings Plugin Architecture + Folder Tree Widget - -**Goal:** Split app settings into per-app files, build the folder tree widget, -restructure Settings with per-app collapsible sections. Normalize server-side -ops. - -**Testable after this phase:** Each app has its own settings section with toggle. -Folder tree popup works for directory selection. TMDB/MusicBrainz settings are in -their app sections. Chat has link preview toggle. Videos/Music use multi-directory. - -### 3.1 Folder tree widget - -| File | Change | -|---|---| -| `static/folder-tree.js` (new) | `FolderTreePicker` component. Props: `roots` (with writable/removable), `dirs` (flat list from index), `mode` ("single"/"multi"), `requireWritable` (bool), `selected` (current selection), `onSelect` callback. Renders a modal with tree-view of directories. Builds tree structure from flat `nodeDirs` paths | - -**Behavior:** -- Modal overlay with semi-transparent backdrop, centered panel -- Root level: each root with folder icon, name, RO/RW badge, removable badge -- `[+]` / `[-]` toggle to expand/collapse children -- Directories sorted alphabetically at each level -- Single mode: clicking a directory selects it (highlight), deselects previous -- Multi mode: clicking a directory toggles its selection (checkbox visual) -- `requireWritable=true`: RO roots and all their children are greyed out and - unclickable, with a brief explanation at the top of the modal -- Selected path shown at the bottom of the modal -- OK (disabled if nothing valid selected) and Cancel buttons -- Escape key closes - -### 3.2 Per-app settings files - -| File | Content | -|---|---| -| `static/chat-app-settings.js` (new) | `ChatSettings` component. Single-directory picker (folder tree, `requireWritable=true`). Link preview toggle (`ToggleSwitch`). Save button. Validates: selected directory must be on a RW root | -| `static/video-app-settings.js` (new) | `VideoSettings` component. Multi-directory picker (folder tree). TMDB section: toggle + API key + language (moved from `group-settings.js`). TMDB key prompt: "Sign up on TMDB to generate your API key" with link, no "optional" wording. Save triggers TMDB sweep | -| `static/music-app-settings.js` (new) | `MusicSettings` component. Multi-directory picker. MusicBrainz section: toggle (moved from `group-settings.js`). Save triggers MusicBrainz sweep | -| `static/photos-app-settings.js` (new) | `PhotoSettings` component. Multi-directory picker. No third-party settings. Save button | -| `static/apps.js` | Each `APPS` entry gains `Settings` field imported from the corresponding settings file. `files` has no `Settings` (no per-app config). `files` has `alwaysEnabled: true` (from Phase 1) | - -### 3.3 Settings page restructure - -| File | Change | -|---|---| -| `group-settings.js` | Remove all inlined app settings (TMDB section, MusicBrainz section, app root pickers). Remove the Applications checkbox section. New rendering loop: for each app in `APPS` where `!app.alwaysEnabled && app.Settings`, render a `CollapsibleSection` with: icon + title in header, toggle switch in header, `app.Settings` component inside (visible only when enabled). Section collapsed by default. App toggle triggers signed `OP_APPS_ENABLED` op | - -**Final settings layout:** - -``` -1. Invite form (if admin & invite-only) -2. Pair operator (if nodeAdmin & not paired) -── App configuration ── -3. Shared directories (expanded, SharedDirectoriesTable) -4. 💬 Chat [toggle] (collapsed) - └─ Directory picker, Link preview toggle, Save -5. 🎬 Videos [toggle] (collapsed) - └─ Directory picker (multi), TMDB settings, Save -6. 🎵 Music [toggle] (collapsed) - └─ Directory picker (multi), MusicBrainz settings, Save -7. 📷 Photos [toggle] (collapsed) - └─ Directory picker (multi), Save -8. [Future apps discovered from APPS registry] -── Node tuning ── -9. Scan tuning (reconcile, debounce) -── Danger zone ── -10. Leave / Delete group -── Identity ── -11. My devices -12. Members table -``` - -### 3.4 Server-side normalization - -| File | Change | -|---|---| -| `ops.py` | New generic: `set_app_directory(state, group_id, app_key, path, require_writable=False)` and `set_app_directories(state, group_id, app_key, paths, require_writable=False)`. Both validate path(s) within named roots and check writable if required. Existing `set_video_root` → wrapper calling `set_app_directories("video", ...)`. Same for audio, photo. New: `set_chat_directory(state, group_id, path)` → `set_app_directory("chat", path, require_writable=True)`. New: `set_chat_link_preview(state, group_id, enabled)` | -| `webrtc_server.py` | Handle new MNP messages: `CHAT_DIRECTORY` / `ACK`, `CHAT_LINK_PREVIEW` / `ACK`. Refactor `VIDEO_ROOT` handler to use generic. Handshake ack: add `chat_directory`, `chat_link_preview` | -| `protocol.py` | New message types: `CHAT_DIRECTORY`, `CHAT_LINK_PREVIEW` (and acks) | -| `ui/app.py` | New endpoints: `PUT /api/groups/{gid}/chat-directory`, `PUT /api/groups/{gid}/chat-link-preview`. Generic: `PUT /api/groups/{gid}/app-directories/{app_key}` | -| `linkpreview.py` | Check `chat_link_preview` setting before unfurling | - -### 3.5 Handshake ack additions - -```python -"chat_directory": "/Shared/uploads", -"chat_link_preview": True, -"video_directories": ["/Movies", "/Series"], # replaces video_root (single) -"audio_directories": ["/Music"], # replaces audio_root (single) -"photo_directories": ["/Photos", "/Camera"], # replaces photo_roots -``` - -Old field names (`video_root`, `audio_root`, `photo_roots`) still sent for backward -compat with MNP 1.0 clients. New clients read the `*_directories` form. - ---- - -## 4. Phase 3 — HelloWorld, CLI polish & Migration - -**Goal:** Prove the plugin architecture with a HelloWorld app, finalize CLI changes, -write and test the migration script. - -### 4.1 HelloWorld app - -| File | Content | -|---|---| -| `static/helloworld-app.js` (new) | Minimal component: renders "Hello, World!" heading + lists files from its configured directory. Uses `entries` from `commonProps`, filtered by the configured path | -| `static/helloworld-app-settings.js` (new) | `HelloWorldSettings` component: single-directory picker via `FolderTreePicker`, Save button. Uses `set_app_directory("helloworld", path)` on the generic endpoint | -| `apps.js` | New entry: `{ key: "helloworld", icon: "👋", labelKey: "helloWorld", Component: HelloWorldApp, Settings: HelloWorldSettings }` | -| `webrtc_server.py` | Add `"helloworld"` to `ALLOWED_APPS` | - -**Validation:** enabling HelloWorld in Settings, picking a directory, saving, and -seeing the file list in the HelloWorld tab — with **zero changes** to -`group-settings.js`, `group-page.js`, or `ops.py`. If this works, any future app -can be added the same way. - -### 4.2 CLI final polish - -- `root` subcommand fully tested: `add`, `remove`, `set`, `list` -- `group add --upload-dir` prints deprecation warning and maps to `--dir <path> - --writable` -- `member upload` command removed (prints migration guidance) -- Help text updated to reflect RO/RW model -- `test_cli_dispatch.py` updated for all new verbs - -### 4.3 Migration script - -`QE/migration/migrate_groups_v2.sh` (or `.py`) — not versioned, for Fedora/Ubuntu -nodes only. - -**node.toml transformations:** -``` -upload = true → writable = true -upload = false → writable = false -(no upload field) → writable = false -(add removable = false to every root that lacks it) -(remove upload_dir lines from [[groups]] blocks) -``` - -**roster.db transformations:** -```sql --- Convert member_upload=off to all-RO roots (handled via ops on restart) --- Rename app directory keys -UPDATE group_settings SET key = 'video_directories' WHERE key = 'video_root'; -UPDATE group_settings SET key = 'audio_directories' WHERE key = 'audio_root'; -UPDATE group_settings SET key = 'photo_directories' WHERE key = 'photo_roots'; --- Wrap single values in JSON arrays for video/audio --- Add chat defaults -INSERT INTO group_settings (group_id, key, value) - SELECT group_id, 'chat_link_preview', 'true' - FROM group_settings WHERE key = 'enabled_apps' AND value LIKE '%chat%'; --- Remove member_upload rows -DELETE FROM group_settings WHERE key = 'member_upload'; -``` - -**Idempotency:** every transformation is guarded (`IF NOT EXISTS`, check before -rename, etc.). Safe to run twice. - -**Rollback:** the script backs up `node.toml` and `roster.db` before any change. - -### 4.4 Windows notes - -No migration script needed for Windows (manual setup). Functional non-regression -testing only: -- Drive letter roots (`D:\Movies`, `E:\Music`) work with RO/RW -- Removable flag on USB drives -- Folder tree widget handles backslash paths -- Create Group wizard with Windows paths - ---- - -## 5. Risk assessment - -| Risk | Mitigation | -|---|---| -| Upload regression | Phase 1 is self-contained: test uploads on RW roots, verify refused on RO roots, before touching app settings | -| Eject data loss | Eject freezes entries (existing `freeze-not-empty` path). Ejected roots are never rescanned. Auto-eject safety net for surprise unplugs on removable roots. Explicit reconciliation (not full rescan) on plug | -| Indexer race on eject | Watchdog observer is stopped synchronously before `ejected=true` is set. No window where the watchdog sees a missing path and processes deletions | -| MNP backward compat | Computed `member_upload` in ack for 1.0 clients. New fields additive. Old field names kept alongside new ones | -| Monolithic `group-settings.js` diff | Phase 2 extracts code into new files; the old code is deleted, not refactored. Clear before/after | -| Folder tree perf with large indexes | `nodeDirs` is already computed. Tree construction is O(n) on directory count, not file count. Lazy child rendering on expand | -| Windows path handling | Existing NFC normalization and path handling unchanged. New fields (`writable`, `removable`) are path-independent booleans. Drive letters work as root paths | -| Migration data loss | Script backs up before changes. Idempotent. Tested on a staging node before production | - ---- - -## 6. Files touched (by phase) - -### Phase 1 -``` -packages/meshbay-node/src/meshbay_node/config.py -packages/meshbay-node/src/meshbay_node/roots.py -packages/meshbay-node/src/meshbay_node/ops.py -packages/meshbay-node/src/meshbay_node/roster.py -packages/meshbay-node/src/meshbay_node/ui/app.py -packages/meshbay-node/src/meshbay_node/daemon.py -packages/meshbay-node/src/meshbay_node/transport/webrtc_server.py -packages/meshbay-common/src/meshbay_common/protocol.py -packages/meshbay-common/src/meshbay_common/handshake.py -packages/meshbay-hub/src/meshbay_hub/static/group-settings.js -packages/meshbay-hub/src/meshbay_hub/static/create-group-page.js -packages/meshbay-hub/src/meshbay_hub/static/group-page.js -packages/meshbay-hub/src/meshbay_hub/static/files-app.js -packages/meshbay-hub/src/meshbay_hub/static/chat-app.js -packages/meshbay-hub/src/meshbay_hub/static/apps.js -packages/meshbay-hub/src/meshbay_hub/static/transport.js -packages/meshbay-node/src/meshbay_node/indexer/indexer.py -packages/meshbay-hub/src/meshbay_hub/static/video-app.js -packages/meshbay-hub/src/meshbay_hub/static/music-app.js -packages/meshbay-hub/src/meshbay_hub/static/photos-app.js -packages/meshbay-hub/src/meshbay_hub/static/locales/*.js -``` - -### Phase 2 -``` -packages/meshbay-hub/src/meshbay_hub/static/folder-tree.js (new) -packages/meshbay-hub/src/meshbay_hub/static/chat-app-settings.js (new) -packages/meshbay-hub/src/meshbay_hub/static/video-app-settings.js (new) -packages/meshbay-hub/src/meshbay_hub/static/music-app-settings.js (new) -packages/meshbay-hub/src/meshbay_hub/static/photos-app-settings.js (new) -packages/meshbay-hub/src/meshbay_hub/static/group-settings.js (extract) -packages/meshbay-hub/src/meshbay_hub/static/apps.js -packages/meshbay-node/src/meshbay_node/ops.py -packages/meshbay-node/src/meshbay_node/ui/app.py -packages/meshbay-node/src/meshbay_node/transport/webrtc_server.py -packages/meshbay-node/src/meshbay_node/linkpreview.py -packages/meshbay-common/src/meshbay_common/protocol.py -packages/meshbay-hub/src/meshbay_hub/static/locales/*.js -``` - -### Phase 3 -``` -packages/meshbay-hub/src/meshbay_hub/static/helloworld-app.js (new) -packages/meshbay-hub/src/meshbay_hub/static/helloworld-app-settings.js (new) -packages/meshbay-hub/src/meshbay_hub/static/apps.js -packages/meshbay-node/src/meshbay_node/transport/webrtc_server.py -packages/meshbay-node/src/meshbay_node/daemon.py -tests/test_cli_dispatch.py -QE/migration/migrate_groups_v2.py (new, not versioned) -``` - ---- - -## 7. Estimated effort - -| Phase | Scope | Approx. size | -|---|---|---| -| Phase 1 | Data model + shared dirs UI + eject/plug + CLI + remove uploads | ~1800 lines changed/added | -| Phase 2 | Folder tree + 4 app-settings files + settings restructure + server normalization | ~1800 lines changed/added | -| Phase 3 | HelloWorld + CLI polish + migration script | ~500 lines | - -Each phase is one focused Claude session. Test between phases. - ---- - -## 7b. Phase 1 review (2026-09-06) - -What the plan above got wrong, and what was actually built. The first three -entries are **rules for phases 2 and 3**, not one-off fixes: each describes a -shape the same code can take again. - -### The rules - -**An operator is not sitting at their node.** The shared directories table read -its roots exclusively from the loopback API (`platform.node.available`), which -resolves to "not available" in a browser. So the section rendered for nobody on -the web — while the Uploads controls it replaced *had* worked there — and the -`transport.updateRoot` / `ejectRoot` / `plugRoot` methods written next to it -were unreachable. §2.2 of the plan said "calls loopback API", and that was the -mistake: MNP is the path that must exist, and loopback is the fallback for a -local node with no live connection. Every operator-facing control phase 2 adds -(the folder tree, four app settings panes, the link-preview toggle) needs the -MNP route first. Pinned by `test_upload_controls_hidden.py`. - -**A reply that carries state nobody could have predicted has to be handed on.** -`transport.js` resolves an admin `*_ack` against the pending request and -returns, deliberately: every caller already updates local state from the value -it chose. The root acks are not like that — they carry the node's whole roots -table, including things only it knows (availability, the name it settled on, -the eject a failed plug left in place). Returning left the operator who clicked -Eject as the single client that never saw it happen, while every *other* peer -got the broadcast. Any phase-2 op returning computed state has the same shape. - -**A control that writes needs to name where.** The node was given a `root` -field on `file_upload` and no client ever sent it, so every upload went to -`writable_roots[0]` while the Files toolbar offered the button based on the root -being browsed. With two writable roots, uploading from one wrote into the other. -This is the failure `_settle_upload_root`'s deleted docstring existed to -prevent, reintroduced by removing it. Chat's attachments have the same problem -one level up and get an explicit `attachRoot` until §1.7 gives them a -configured directory. - -### The rest - -- **`ejected` was written to `roster.db` and never read back**, and the - auto-eject path did not persist at all. Both fixed; see §1.5b. -- **`PUT /api/groups/{gid}/member-upload` became a stub returning `200 - {"deprecated": true}`.** A route that answers OK and changes nothing is - indistinguishable from a working one to whoever calls it. Removed. -- **The wizard ignored the first root's RW switch** — `ops.attach_group` always - wrote `writable = true`. It takes the flag now. -- **`refresh_availability` was the only reader of a root's config.** The reload - path compared roots on `(name, path)`, so an operator editing `writable` in - `node.toml` and reloading saw nothing happen. The comparison includes the - flags. -- **The table had no Path column** (§1.5 asked for one). Two libraries whose - folders share a basename are indistinguishable without it, and the basename is - the identity — so it is the one thing that has to be visible. -- **Phase 1 shipped no tests.** 29 of the suite's failures were its own. The - gap that mattered was not the broken helpers but that eject, plug, per-root - upload refusal and the `node.toml` rewrite had no coverage at all: - `test_root_eject.py`, `test_root_writable_policy.py` and the new cases in - `test_ops.py` / `test_node_status.py` / `test_security_regressions.py` are - that. `test_member_upload_policy.py` is gone — it tested a removed feature. -- **`chat-app.js` was in §2.2's file list and was never touched.** -- **Eight of the ten locales were missing the new keys.** `test_locales.py` - holds them to `en.js`, so this was a failing test rather than a silent gap — - but it is worth noting that adding a key means adding it ten times. - -### Still open, deliberately - -- **The operator can no longer have a directory only they may write to.** RW is - open to every member; RO refuses everyone including the operator. This - reverses draft-v6's structural decision 9, which is annotated there. It is a - real capability removed, and if it turns out to be wanted the answer is a - third state on the root, not the old group-wide switch. -- `test_ops.py::test_a_backslash_path_written_into_node_toml_stays_parseable` - fails on any non-Windows machine and always has — it builds a - `PurePosixPath` from a Windows path. Unrelated to this refactor, left alone. - ---- - -## 7c. Phase 2 as built (2026-09-06) - -The plan held. Four things were done differently, and one of them is a rule. - -### The rule - -**A settings key added to the client must be added ten times.** `test_locales` -holds the nine other catalogues to `en.js`, so a missing key is a failing test -rather than a silent gap — but Phase 2 added 27 keys, and doing them one file -at a time is how the Phase 1 gap happened. Write the table, generate the -insert. - -**And a second one, which cost a bug in this phase:** `node --check foo.js` -does **not** reliably report a module syntax error. It accepted a file with -`${/* ... */''}` — htm template syntax, pasted into a plain object literal — -and reported success. Copying to `.mjs` first forces the module parser, which -reports it. `test_spa_syntax.py` now does that for every module; the suite had -no syntax check at all before, which is how the file was committed. - -### Done differently - -- **Almost no migration script.** The plan (§4.3) called for one to rename - `video_root` → `video_directories` in `roster.db`. Instead the roster falls - back to the old key when the new one is unset, and the first save through the - new path leaves it behind. A script that has to be run by hand on the machine - where it matters is a step that does not happen; a fallback is one that - cannot be skipped. `node.toml` needs nothing either — `upload = true` is read - as `writable`. - - **One transformation genuinely cannot be a fallback**, and an earlier draft - of this section wrongly said the script was unnecessary altogether. A group - whose operator had turned `member_upload` *off* has that switch ignored after - the upgrade, because nothing consults it any more — and its root still says - `upload = true`, so it accepts uploads from every member again. Nobody is - told. "Uploads are off for this group" and "this root is writable" are two - different sentences that happened to disagree, and only the operator knows - which they meant; there is nothing to infer. `QE/migration/check_upload_policy.py` - reads both files, reports the groups affected, and prints the `root set - --no-writable` line for each. Read-only, exits non-zero when something needs - a decision, so it can gate a deploy. -- **`music`, not `audio`.** The app's registry key was `music` while its - storage said `audio_root` and its ops said `set_audio_root`. One identifier - per app now — the registry key — with the correspondence in exactly one - table (`Roster.LEGACY_DIR_KEYS`). -- **One storage shape.** `set_app_directory` (single) writes a one-element - list, so there is no scalar form anywhere below the wire. `video_root` and - friends survive on the handshake ack only, *derived* from the list rather - than stored beside it — a second stored value drifts within one run, which - reads as "it works after a restart". -- **The panes call the transport themselves.** The plan had every pane report - through one `onSave`, which would have made the page a dispatcher naming - every app's settings keys — the thing the phase exists to remove. The line - is: what every app has (directories) the page does, generically; what one - app alone has (a TMDB key, a link-preview switch) the pane does with the - transport it is handed. An app that only wants directories touches neither - file, which is `test_app_settings_plugin.py`'s subject. - -### Worth knowing - -- `settings-ui.js` exists because `group-settings` → `apps` → a pane → - `group-settings` is an import cycle, and ES modules answer that with a - temporal-dead-zone `ReferenceError` at first render — the component simply - does not appear, which is the fault already recorded in CLAUDE.md about hook - ordering. The shared widgets live outside both. -- The folder picker asks the node for **nothing**. The tree is derived from - paths the client already holds, so it shows what the group's index contains - and no more — a folder the node never indexed does not exist as far as the - group is concerned. There is no folder-browsing protocol and this does not - add one. -- `_ASSETS` in `webapp.py` had to grow by six. Modules reached through the - registry rather than imported by name are exactly the ones nothing else would - notice changing, and a stale one is served from a browser cache with no - version bump. `test_asset_versioning` caught it. -- **What Phase 3 still owes:** the HelloWorld app (§4.1) — which is the actual - proof of the above, since every test here reads source rather than adding an - app and watching it work — plus the CLI polish and the Windows pass. - ---- - -## 7d. Phase 3 as built (2026-09-07) - -### HelloWorld earned its place - -It was written last and immediately found two things no amount of source -reading had: `group-settings.js` fell back to the *whole* registry when a group -had no `enabled_apps` yet — which would have enabled a hidden app for everyone -— and `group-page.js` wrote out `videoDirectories` / `musicDirectories` / -`photoDirectories` by hand, so a fifth app would have needed that file edited. -Both are fixed by making the code less app-specific, and the plugin claim is -now true rather than nearly true. - -That is the argument for keeping it: every other test of the architecture reads -source for the *absence* of app names, which proves nobody wrote a special case -— not that a new app works. Deleting HelloWorld would leave the claim resting -entirely on tests that read text. - -**It is hidden behind `?dev=1`**, not excluded from the build as §4.1 imagined. -There is no build step to exclude it from, and an unregistered app proves -nothing, since registration is exactly what is claimed to be sufficient. The -flag is the same opt-in shape as `transport.js`'s `?trace=1`. - -### The CLI - -**`--upload-dir` is gone, not deprecated.** It was documented as the old -spelling first, which was wrong and was caught in review. It wrote `upload_dir` -into a *brand-new* `[[groups]]` block, and `GroupConfig.__post_init__` reads -that key by forcing every other root read-only and appending that path as the -one writable one — so `group add --dir X --writable --upload-dir Y` silently -made X read-only. Two mechanisms deciding which directories accept uploads, one -of them invisible, in a group created after the model that replaced it. The -*read* path stays, because an existing node.toml must keep working; that is the -only legitimate use, and nothing writes the key any more. - -`member upload` reached the generic usage line for the other `member` verbs — -"usage: meshbay-node member upload <username>" — which advertises a removed -feature and sends the operator looking for a username it would then reject. It -names `root set --writable` now. `--upload-dir` still works, so an existing -script keeps working, but its help and the man page say it is the old spelling. - -### Windows - -`test_windows_root_shapes.py` covers what can be covered from here: drive -letters and UNC through `as_posix()` into TOML, a drive root having no basename -to derive a name from, and a case-insensitive collision — which on NTFS and -exFAT is one directory indexed as two roots. All pass. - -**What still needs a person on Windows**, and cannot be faked: - -- `ReadDirectoryChangesW` dropping events under load — the reason periodic - reconciliation is mandatory, and the reason eject exists at all -- `MAX_PATH` against a deep library, on download and on upload -- whether an eject actually lets the drive be removed, and a plug picks it back - up — the eject/plug pair is the least-exercised thing in all three phases -- the folder-tree picker against backslash paths in the UI -- the Create Group wizard with a drive-letter root - -### What the whole refactor still owes - -Nothing in the plan. Two things it did not think of: - -- **`_do_dir_delete` was never checked against RO/RW.** `_do_file_upload` and - `_do_dir_create` both gained the `writable` check; deletion is operator-only - and so is not the same hole, but the asymmetry is worth a look. -- **`index_delta` carries roots but not `dirs`.** A folder created by another - member does not reach a connected client's folder picker until a full - `index_sync`. Small, and the picker offers root names from the roots table - regardless, so nothing is unreachable — but it is the same class as the bug - §7d's roots fix closed. |