summaryrefslogtreecommitdiffstats
path: root/devel-phases.md
diff options
context:
space:
mode:
authorChristophe Besson <cbesson@gmail.com>2026-09-01 16:05:16 +0200
committerChristophe Besson <cbesson@gmail.com>2026-09-01 16:05:16 +0200
commit8a6294b0412a86f378c6e2e937c28de64a903c91 (patch)
tree20d1977d9148a2c86c62925c5be7e571815c9057 /devel-phases.md
parent854a9441ccb734c7fbb1e0ff8570b9ef659c09b0 (diff)
downloadmeshbay-8a6294b0412a86f378c6e2e937c28de64a903c91.tar.gz
docs: move root docs into docs/ and archive superseded drafts
Move the remaining root-level .md files (except CLAUDE.md) into docs/: devel-phases.md, devel-phases-next.md, first-review.md, second-review.md, tmp-decisions.md. Update all inbound references in CLAUDE.md (now docs/-prefixed) and strip the now-redundant docs/ prefix from links inside the moved files. Consolidate the superseded material into docs/old-draft.md: architecture drafts v1-v4, POC v1, and the Phase 1-12 development log, each under an ARCHIVED banner with a preamble pointing at the current specs. Delete the merged originals plus the unreferenced French translations (v1-fr, v2-fr, poc-v1-fr). Repoint the surviving file-links in first-review.md, second-review.md and meshbay-draft-v5.md at old-draft.md; prose "draft-v3 §x" mentions are left as-is since the content now lives in the archive. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J74kj44q6REczub8XR3DRy
Diffstat (limited to 'devel-phases.md')
-rw-r--r--devel-phases.md404
1 files changed, 0 insertions, 404 deletions
diff --git a/devel-phases.md b/devel-phases.md
deleted file mode 100644
index b5bef49..0000000
--- a/devel-phases.md
+++ /dev/null
@@ -1,404 +0,0 @@
-# MeshBay — Development Phases
-
-> Reference: architecture spec in `docs/meshbay-draft-v3.md`
-> POC results: `poc/spike-results.md`
-
----
-
-## Phase 1 — Foundations ✅ DONE
-
-**Goal:** validate all blocking concepts before writing production code.
-
-### Deliverables
-
-| Item | Status | Notes |
-|---|---|---|
-| POC Spike 1 — Crypto primitives | ✅ | All 22 tests pass. ChaCha20 1MB in 1.1ms. |
-| POC Spike 2 — Hub skeleton | ✅ | 12/12 endpoints. JWT EdDSA offline verify in 884µs. |
-| POC Spike 3 — Node registration | ✅ | Full handshake. jti bug found and fixed. |
-| POC Spike 4 — NAT traversal | ✅ | Cone NAT on SFR. UDP P2P works. UPnP disabled (SFR). |
-| POC Spike 5 — Encrypted transfer | ✅ | 1MB P2P. 3.2ms encrypt, 3.9ms decrypt. 4.3MB/s. |
-| POC Spike 6 — GEK distribution | ✅ | X25519+HKDF wrap/unwrap. 0.48ms/0.59ms. Hub opaque. |
-| Security cleanup meshbay.org | ✅ | UFW: 22/80/443 only. No services exposed. |
-| Git monorepo | ✅ | 3 packages: meshbay-common, meshbay-hub, meshbay-node. |
-| Draft v3 | ✅ | POC findings integrated. All corrections applied. |
-| CLAUDE.md conventions | ✅ | Python 3.12+, uv, ruff, SemVer, commit format. |
-
-### Key findings from POC
-
-- jti mandatory in all JWTs (Ed25519 is deterministic — same payload = same token)
-- Argon2id at 64MB/3iter = 78ms — increase to 256MB for production (~500ms target)
-- NAT order: STUN/hole-punching is priority 2, not UPnP (UPnP disabled on tested SFR box)
-- TCP+TLS for v1 transport; QUIC in v2
-- GEK wrapping: ephemeral X25519 + HKDF(salt=pk_eph) + ChaCha20-Poly1305(aad=pk_recipient)
-
----
-
-## Phase 2 — Node v1 ✅ DONE
-
-**Goal:** working Mesh Node: indexes a directory, registers with hub,
-serves encrypted chunks over TCP+TLS, local web UI on localhost:18000.
-
-**Transport:** TCP+TLS 1.3 (QUIC in v2). Self-signed cert per node.
-Node identity verified via Ed25519 PK from hub, not TLS cert chain (client uses CERT_NONE).
-
-### Milestones
-
-| # | Component | File(s) | Tests |
-|---|---|---|---|
-| 2.1 | Keystore | `meshbay_node/keystore.py` | 10/10 |
-| 2.2 | Hub client | `meshbay_node/hub_client.py` | 6/6 |
-| 2.3 | Directory indexer | `meshbay_node/indexer/indexer.py` | 5/5 |
-| 2.4 | Mesh Group Index | `meshbay_node/indexer/group_index.py` | 5/5 |
-| 2.5 | TCP+TLS chunk server | `meshbay_node/transport/server.py` | 3/3 |
-| 2.6 | TCP+TLS chunk client | `meshbay_node/transport/client.py` | included above |
-| 2.7 | TLS cert helper | `meshbay_node/transport/tls_cert.py` | — |
-| 2.8 | Config (TOML + env) | `meshbay_node/config.py` | — |
-| 2.9 | Local web UI | `meshbay_node/ui/app.py` | — |
-| 2.10 | Daemon + CLI | `meshbay_node/daemon.py` | — |
-
-**Total: 29/29 tests passing**
-
-### Python dependencies (dev venv — `/home/cbesson/meshbay/.venv`)
-
-Installed packages (freeze) as of Phase 2 completion:
-
-```
-aioice==0.10.2 # ICE/STUN for NAT traversal
-blake3==1.0.9 # fast content hashing
-cryptography==50.0.0 # Ed25519, X25519, ChaCha20, Argon2id, AES-GCM
-fastapi==0.141.1 # local web UI + hub POC
-httpx==0.28.1 # hub client HTTP
-meshbay-common==0.1.0 # editable install
-meshbay-node==0.1.0 # editable install
-msgpack==1.2.1 # wire serialisation
-PyJWT==2.13.0 # JWT EdDSA
-pytest==9.1.1 # test runner
-pytest-asyncio==1.4.0 # async test support
-uvicorn==0.52.1 # ASGI server (local UI)
-watchdog==6.0.0 # filesystem watcher
-zstandard==0.25.0 # zstd compression
-```
-
-Also installed transitively: pydantic 2.13.4, starlette 1.6.0, anyio 4.14.2, uvloop 0.22.1.
-
-### meshbay_common/crypto.py — Argon2id NOTE
-
-Current params: `iterations=3, memory_cost=65536` (64MB) → ~78ms on dev laptop.
-**Must increase to `memory_cost=262144` (256MB) before production keystore use.**
-Run `meshbay-node calibrate-argon2` on target hardware to tune.
-
-### Out of scope for Phase 2
-
-Multiple groups, chat, QUIC, module sandbox, mobile pairing, HLS streaming.
-
----
-
-## Phase 3 — Hub v1 production ✅ DONE
-
-**Goal:** replace POC in-memory hub with a production-ready service on meshbay.org.
-PostgreSQL persistence, HTTPS via Caddy, all endpoints hardened, legal IP logging, deploy.
-
-### Environment
-
-- **Server:** meshbay.org — Ubuntu 26.04 LTS, Python 3.14.4, OVH VPS
-- **Database:** PostgreSQL 16 (to install)
-- **Proxy:** Caddy (to install — handles Let's Encrypt automatically)
-- **Service:** systemd `meshbay-hub.service`
-
-### Python dependencies to add (Phase 3)
-
-```
-sqlalchemy>=2.0 # async ORM (SQLAlchemy 2.x)
-alembic>=1.13 # DB migrations
-asyncpg>=0.30 # PostgreSQL async driver
-aiosqlite>=0.20 # SQLite async driver (tests only)
-slowapi>=0.1 # rate limiting (FastAPI middleware)
-```
-
-### Milestones
-
-| # | Component | File(s) | Status |
-|---|---|---|---|
-| 3.1 | DB models | `meshbay_hub/db/models.py` | ✅ |
-| 3.2 | DB engine + session | `meshbay_hub/db/engine.py` | ✅ |
-| 3.3 | Alembic migrations | `meshbay_hub/db/migrations/` | ✅ initial_schema |
-| 3.4 | Hub config | `meshbay_hub/config.py` | ✅ |
-| 3.5 | Auth (JWT + Argon2id) | `meshbay_hub/auth.py` | ✅ |
-| 3.6 | API deps | `meshbay_hub/api/deps.py` | ✅ |
-| 3.7 | Hub info router | `meshbay_hub/api/hub.py` | ✅ |
-| 3.8 | Users router | `meshbay_hub/api/users.py` | ✅ |
-| 3.9 | Nodes router | `meshbay_hub/api/nodes.py` | ✅ |
-| 3.10 | Groups router | `meshbay_hub/api/groups.py` | ✅ |
-| 3.11 | Rate limiting | `meshbay_hub/api/middleware.py` | ✅ slowapi |
-| 3.12 | App factory + lifespan | `meshbay_hub/app.py` | ✅ |
-| 3.13 | Hub daemon CLI | `meshbay_hub/daemon.py` | ✅ |
-| 3.14 | PostgreSQL 16 | meshbay.org | ✅ DB: meshbay_hub |
-| 3.15 | Caddy + Let's Encrypt | meshbay.org Caddyfile | ✅ HTTPS auto-cert |
-| 3.16 | Systemd service | `/etc/systemd/system/meshbay-hub.service` | ✅ |
-| 3.17 | Deploy + smoke test | https://meshbay.org | ✅ 11/11 endpoints |
-
-**Total: 40 local tests (SQLite) + 11/11 smoke tests HTTPS production**
-
-### Phase 3 deployment details (meshbay.org)
-
-- PostgreSQL 16, user `meshbay`, DB `meshbay_hub`
-- Caddy auto-handles Let's Encrypt for `meshbay.org` and `www.meshbay.org`
-- `www.meshbay.org` → 301 redirect → `meshbay.org`
-- TLS setup documented in `docs/HTTPS.md`
-- Hub listens on `127.0.0.1:8000`, Caddy proxies 80/443
-- Systemd service: `meshbay-hub.service` (restart-on-failure)
-- Hub config: `~/.config/meshbay/hub.toml`
-- Hub keypair: `~/.config/meshbay/hub_private.pem` (chmod 600)
-- Source deployed at: `~/meshbay-hub/` (common_pkg + hub_pkg)
-
-### Additional Python dependencies added in Phase 3
-
-```
-sqlalchemy==2.0.51 # async ORM
-alembic==1.19.1 # DB migrations
-asyncpg==0.31.0 # PostgreSQL async driver
-aiosqlite # SQLite async (tests only)
-slowapi==0.1.10 # rate limiting
-hatchling==1.31.0 # build backend (needed for pip install)
-```
-
-### Notes
-
-- Initial DB schema created via `init_db()` (`create_all`) — Alembic tracks future changes
-- IP logging records: account_create, login, login_fail, group_create, node_announce
-- Rate limiting active on /v1/users/register and /v1/users/login (slowapi)
-- Revocation, CSAM hash matching, moderation deferred to Phase 5
-
-### Phase 3 scope
-
-**In scope:**
-- All POC endpoints from Spike 2+6, production-ready
-- PostgreSQL via SQLAlchemy async + Alembic migrations
-- IP logging (creation, login, group events) — 1-year retention, legal compliance
-- Access token (JWT, 1h) + refresh token (30d, stored hashed)
-- Rate limiting on auth endpoints
-- Hub config file `/etc/meshbay/hub.toml` or env vars
-- HTTPS via Caddy + auto Let's Encrypt on meshbay.org
-- Systemd service with restart-on-failure
-
-**Deferred to later:**
-- Revocation push (WebSocket signaling to nodes)
-- CSAM hash matching (NCMEC/IWF integration)
-- Moderation flow (blocklist + takedown)
-- MHP federation
-- RPM/DEB packaging
-
-### Testing strategy
-
-- Unit tests with SQLite in-memory (`aiosqlite`) — no PostgreSQL needed locally
-- API tests via `httpx.AsyncClient` + `ASGITransport` — no network
-- All tests run in the existing `.venv` after adding Phase 3 deps
-
----
-
-## Phase 4 — Integration & Web Client ✅ DONE
-
-**Goal:** end-to-end working product in a browser: login via hub, discover groups,
-browse files on a node, download and stream public content.
-
-### Architecture decision
-
-Browsers cannot make raw TCP connections — the node must speak HTTP.
-- Node adds an **HTTP file API** (port 19001) serving public content via standard `fetch()`
-- Private content (GEK decrypt in browser) deferred to Phase 5 (requires WebCrypto + ChaCha20 WASM)
-- Hub serves the web application at `meshbay.org/app/`
-
-### Milestones
-
-| # | Component | File(s) | Status |
-|---|---|---|---|
-| 4.1 | Node HTTP file API | `meshbay_node/transport/http_server.py` | ✅ 7/7 tests |
-| 4.2 | Hub web app (HTML/JS) | `meshbay_hub/api/webapp.py` + `static/app.js` | ✅ live on meshbay.org |
-| 4.3 | Group listing endpoint | `meshbay_hub/api/groups.py` GET /v1/groups | ✅ |
-| 4.4 | End-to-end integration test | local node ↔ hub ↔ browser | ✅ smoke test |
-| 4.5 | HLS streaming (public) | node `/hls/{id}/playlist.m3u8` + `.ts` via ffmpeg | ✅ included in 4.1 |
-
-**Total: 47/47 tests. https://meshbay.org live with web client.**
-
-### Phase 4 deployment
-
-- `https://meshbay.org/` — web app (HTML/JS SPA)
-- `https://meshbay.org/app.js` — JS client
-- `https://meshbay.org/v1/groups` — public group listing (no auth)
-- Node HTTP API (port 19001): `/index`, `/file/{id}`, `/hls/{id}/*.m3u8`, `/hls/{id}/*.ts`
-- HLS streaming uses ffmpeg for on-the-fly segmentation
-- Public content: no auth for index, auth required for chunks
-- Private content (GEK decrypt in browser): deferred to Phase 5
-
-### Dependencies added in Phase 4
-
-```
-# Node (runtime)
-ffmpeg (system package) — HLS segmentation via subprocess
-```
-
-### Phase 4 scope
-
-**In scope (public content only):**
-- Node HTTP API: serve public Mesh Group Index (JSON) + file chunks (binary)
-- Hub web app: login, group discovery, file browser, download link
-- HLS basic streaming: node segments video on-the-fly, browser plays natively
-- JWT auth passed as query param or header to node HTTP API
-
-**Deferred to Phase 5:**
-- Private group content in browser (requires ChaCha20-Poly1305 via WASM)
-- Chat UI
-- Multi-group node
-- Android client
-
----
-
-## Phase 5 — QUIC, Federation, Moderation ✅ DONE (Mobile deferred)
-
-**Goal:** full decentralization, mobile support, community infrastructure.
-
-| # | Component | Notes |
-|---|---|---|
-| # | Component | File(s) | Status |
-|---|---|---|---|
-| 5.1 | QUIC transport (MNP v2) | `transport/quic_server.py` + `quic_client.py` | ✅ 3/3 tests |
-| 5.2 | MHP federation | `meshbay_hub/api/federation.py` | ✅ GET/POST /mhp/* |
-| 5.3 | Mesh Relay | `meshbay_hub/api/relay.py` | ✅ register+list+approve |
-| 5.4 | Android client | — | ⏳ deferred (different tech) |
-| 5.5 | Content replication | — | ⏳ deferred |
-| 5.6 | iOS client | — | ⏳ after Android |
-| 5.7 | Revocation push | `api/revocation.py` + `node/revocation.py` | ✅ 3/3 tests |
-| 5.8 | CSAM hash matching | `meshbay_hub/csam.py` | ✅ CSAMChecker + admin API |
-| 5.9 | Moderation | `api/moderation.py` | ✅ 6/6 tests |
-| 5.10 | RPM/DEB packaging | `packaging/` | ✅ spec + control + systemd |
-
-**Total: 59/59 tests.**
-
-### Phase 5 bugs fixed
-
-- **QUIC**: `asyncio.Event` race condition in client recv loop (quic_event_received
- overwrote `_stream_events[0]` created by `_recv`). Fixed with `asyncio.Queue`
- (no shared mutable state between coroutines).
-- **QUIC**: `verify_peer` parameter renamed to `verify_mode` in aioquic 1.3.0.
-- **QUIC**: `connect()` returns protocol directly (not `(transport, proto)` tuple).
-
-### Phase 5 dependencies added
-
-```
-aioquic==1.3.0 # QUIC transport (Cloudflare-maintained)
-websockets # hub→node revocation push
-```
-
-### Deferred to future
-
-- Android/iOS client (Kotlin/Flutter — different tech stack, dedicated effort)
-- Content replication (node-to-node) → Phase 6
-- MHP federation persistence (currently in-memory) → Phase 6
-
----
-
-## Phase 6 — Chat, Multi-group, Federation persistence, Replication ✅ DONE
-
-**Goal:** complete the product with group chat (Double Ratchet), multi-group node support,
-persistent MHP federation, content replication, and browser private group decryption.
-
-### Milestones
-
-| # | Component | File(s) | Status |
-|---|---|---|---|
-| 6.1 | Double Ratchet chat | `meshbay_common/ratchet.py` | ✅ 11/11 tests |
-| 6.2 | Multi-group node | `meshbay_node/config.py` [[groups]] | ✅ |
-| 6.3 | MHP federation persistence | `FederatedGroup` + `SwarmSource` DB tables | ✅ |
-| 6.4 | Content replication | `node/replication.py` + hub `/v1/swarm/*` | ✅ |
-| 6.5 | Browser private group | `webcrypto.py` + `static/crypto.js` | ✅ 4/4 tests |
-| 6.6 | Dérivation clés depuis password | `meshbay_common/keyderive.py` + `static/keyderive.js` | ✅ 7/7 tests |
-| 6.7 | Bundle clés chiffré (web) | Hub: `keypair_bundle` field + migration Alembic | ✅ |
-| 6.8 | Scripts démo opérationnels | `QE/demo-v1/` + `QE/demo-v2/` (non versionné) | ✅ testés |
-| 6.9 | QUICKSTART réécrit | `docs/QUICKSTART.md` | ✅ |
-| 6.6 | Dérivation clés depuis password | `meshbay_common/keyderive.py` + `static/keyderive.js` | ✅ 7/7 tests |
-| 6.7 | Bundle clés chiffré (web) | Hub: `keypair_bundle` field + migration Alembic | ✅ |
-| 6.8 | Scripts démo opérationnels | `QE/demo-v1/` (non versionné) | ✅ testés |
-| 6.9 | QUICKSTART réécrit | `docs/QUICKSTART.md` | ✅ |
-
-**Total: 81/81 tests.**
-
----
-
-## Phase 7 — Node v2: production, streaming, chat ✅ DONE
-
-**Goal:** multi-group node, Sender Keys chat, QUIC 0-RTT, jti denylist push, HLS streaming.
-
-Commit: fc56585 — 26 files, +2155/−159 lines, 109 tests.
-
-See `devel-phases-next.md` for details.
-
----
-
-## Phase 8 — Hub v2: admin, federation, security ✅ DONE
-
-**Goal:** admin roles, email encryption, refresh token rotation, rate limiting, healthcheck.
-
-Commit: 46918ec — 20 files, +508/−90 lines, 117 tests.
-Deployed to meshbay.org. All security review items S1/S2/S5 resolved.
-
-See `devel-phases-next.md` for details.
-
----
-
-## Phase 9 — Web client: WebRTC transport + core SPA ✅ DONE
-
-**Goal:** browser connects P2P to a node behind residential NAT via WebRTC DataChannel.
-Full SPA: login, groups, file browser, download, video playback, chat, i18n, dark/light.
-
-### Milestones
-
-| # | Component | Status |
-|---|---|---|
-| 9.1–9.5 | WebRTC DataChannel spike + E2E NAT validation | ✅ |
-| 9.6–9.12 | Preact SPA (login, groups, files, video, chat, i18n, settings) | ✅ |
-| 9.13 | Tests: 132 passing | ✅ |
-| 9.14–9.16 | Performance: pipelining, binary wire format, I/O reduction | ✅ |
-| 9.17 | Large file download: File System Access API (stream to disk) | ✅ |
-
-**Total: 132/132 tests. Deployed to meshbay.org + Orange node (2026-08-11).**
-
-### Key technical decisions
-
-- **Transport:** WebRTC DataChannel (aiortc on node) — browsers can't use QUIC for NAT traversal
-- **Wire format:** length-prefixed msgpack, binary chunk fields (no base64)
-- **UI:** Preact + htm ESM (vendored, no build step, no CDN, no npm)
-- **Crypto:** WebCrypto SubtleCrypto AES-256-GCM for E2E chunk decryption in browser
-- **Large files:** File System Access API (`showSaveFilePicker`) — stream to disk, ~8 MB RAM
-- **Indexer:** 2s debounce + path-based dedup for file copy events
-
-### NAT traversal validated
-
-Two ISPs (SFR + Orange residential NAT), Chrome + Firefox, IPv4 STUN + IPv6 direct.
-No TURN relay needed. See `devel-phases-next.md` for detailed test matrix.
-
-### QE deployment state (2026-08-11)
-
-- **Hub (meshbay.org):** running as `meshbay-hub.service`, DB has 3 users
- (admin, cbesson, grenet), 1 group (`d3bbd90b`), `admin_usernames = ["admin"]`
-- **Node (Orange host via `ssh cbesson@localhost -p 2222`):** running as
- `nohup .venv/bin/python3 QE/demo-v3/run_node_simple.py`, user grenet,
- connected via WS to hub, WebRTC + QUIC dual transport
-- **Credentials:** `QE/demo-v3/creds.json` (not versioned)
-- **Shared dir on node:** `~/meshbay/QE/demo-v3/shared/`
-- All 3 users password: see creds.json
-
-### Dependencies added
-
-- `aiortc>=1.9` in meshbay-node (WebRTC DataChannel)
-- `preact` + `htm` vendored as `static/vendor/htm-preact.js` (ESM, ~3 KB gzipped)
-
----
-
-## Conventions
-
-- Commits: `feat(node):`, `fix(hub):`, `chore(common):`, `docs:`, `test(node):`
-- Branch per feature/fix, merge to `main` (when remote configured)
-- **Always close test UFW ports after any spike on meshbay.org**
-- **Never commit key material** (keystore.enc, hub_private.pem, *.key, node_state.json, unlock.key)
-- meshbay.org is internet-facing: only run known-safe services, close ports after tests