diff options
| author | Christophe Besson <cbesson@gmail.com> | 2026-09-01 16:05:16 +0200 |
|---|---|---|
| committer | Christophe Besson <cbesson@gmail.com> | 2026-09-01 16:05:16 +0200 |
| commit | 8a6294b0412a86f378c6e2e937c28de64a903c91 (patch) | |
| tree | 20d1977d9148a2c86c62925c5be7e571815c9057 /devel-phases.md | |
| parent | 854a9441ccb734c7fbb1e0ff8570b9ef659c09b0 (diff) | |
| download | meshbay-8a6294b0412a86f378c6e2e937c28de64a903c91.tar.gz | |
docs: move root docs into docs/ and archive superseded drafts
Move the remaining root-level .md files (except CLAUDE.md) into docs/:
devel-phases.md, devel-phases-next.md, first-review.md, second-review.md,
tmp-decisions.md. Update all inbound references in CLAUDE.md (now docs/-prefixed)
and strip the now-redundant docs/ prefix from links inside the moved files.
Consolidate the superseded material into docs/old-draft.md: architecture
drafts v1-v4, POC v1, and the Phase 1-12 development log, each under an
ARCHIVED banner with a preamble pointing at the current specs. Delete the
merged originals plus the unreferenced French translations (v1-fr, v2-fr,
poc-v1-fr). Repoint the surviving file-links in first-review.md,
second-review.md and meshbay-draft-v5.md at old-draft.md; prose "draft-v3 §x"
mentions are left as-is since the content now lives in the archive.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J74kj44q6REczub8XR3DRy
Diffstat (limited to 'devel-phases.md')
| -rw-r--r-- | devel-phases.md | 404 |
1 files changed, 0 insertions, 404 deletions
diff --git a/devel-phases.md b/devel-phases.md deleted file mode 100644 index b5bef49..0000000 --- a/devel-phases.md +++ /dev/null @@ -1,404 +0,0 @@ -# MeshBay — Development Phases - -> Reference: architecture spec in `docs/meshbay-draft-v3.md` -> POC results: `poc/spike-results.md` - ---- - -## Phase 1 — Foundations ✅ DONE - -**Goal:** validate all blocking concepts before writing production code. - -### Deliverables - -| Item | Status | Notes | -|---|---|---| -| POC Spike 1 — Crypto primitives | ✅ | All 22 tests pass. ChaCha20 1MB in 1.1ms. | -| POC Spike 2 — Hub skeleton | ✅ | 12/12 endpoints. JWT EdDSA offline verify in 884µs. | -| POC Spike 3 — Node registration | ✅ | Full handshake. jti bug found and fixed. | -| POC Spike 4 — NAT traversal | ✅ | Cone NAT on SFR. UDP P2P works. UPnP disabled (SFR). | -| POC Spike 5 — Encrypted transfer | ✅ | 1MB P2P. 3.2ms encrypt, 3.9ms decrypt. 4.3MB/s. | -| POC Spike 6 — GEK distribution | ✅ | X25519+HKDF wrap/unwrap. 0.48ms/0.59ms. Hub opaque. | -| Security cleanup meshbay.org | ✅ | UFW: 22/80/443 only. No services exposed. | -| Git monorepo | ✅ | 3 packages: meshbay-common, meshbay-hub, meshbay-node. | -| Draft v3 | ✅ | POC findings integrated. All corrections applied. | -| CLAUDE.md conventions | ✅ | Python 3.12+, uv, ruff, SemVer, commit format. | - -### Key findings from POC - -- jti mandatory in all JWTs (Ed25519 is deterministic — same payload = same token) -- Argon2id at 64MB/3iter = 78ms — increase to 256MB for production (~500ms target) -- NAT order: STUN/hole-punching is priority 2, not UPnP (UPnP disabled on tested SFR box) -- TCP+TLS for v1 transport; QUIC in v2 -- GEK wrapping: ephemeral X25519 + HKDF(salt=pk_eph) + ChaCha20-Poly1305(aad=pk_recipient) - ---- - -## Phase 2 — Node v1 ✅ DONE - -**Goal:** working Mesh Node: indexes a directory, registers with hub, -serves encrypted chunks over TCP+TLS, local web UI on localhost:18000. - -**Transport:** TCP+TLS 1.3 (QUIC in v2). Self-signed cert per node. -Node identity verified via Ed25519 PK from hub, not TLS cert chain (client uses CERT_NONE). - -### Milestones - -| # | Component | File(s) | Tests | -|---|---|---|---| -| 2.1 | Keystore | `meshbay_node/keystore.py` | 10/10 | -| 2.2 | Hub client | `meshbay_node/hub_client.py` | 6/6 | -| 2.3 | Directory indexer | `meshbay_node/indexer/indexer.py` | 5/5 | -| 2.4 | Mesh Group Index | `meshbay_node/indexer/group_index.py` | 5/5 | -| 2.5 | TCP+TLS chunk server | `meshbay_node/transport/server.py` | 3/3 | -| 2.6 | TCP+TLS chunk client | `meshbay_node/transport/client.py` | included above | -| 2.7 | TLS cert helper | `meshbay_node/transport/tls_cert.py` | — | -| 2.8 | Config (TOML + env) | `meshbay_node/config.py` | — | -| 2.9 | Local web UI | `meshbay_node/ui/app.py` | — | -| 2.10 | Daemon + CLI | `meshbay_node/daemon.py` | — | - -**Total: 29/29 tests passing** - -### Python dependencies (dev venv — `/home/cbesson/meshbay/.venv`) - -Installed packages (freeze) as of Phase 2 completion: - -``` -aioice==0.10.2 # ICE/STUN for NAT traversal -blake3==1.0.9 # fast content hashing -cryptography==50.0.0 # Ed25519, X25519, ChaCha20, Argon2id, AES-GCM -fastapi==0.141.1 # local web UI + hub POC -httpx==0.28.1 # hub client HTTP -meshbay-common==0.1.0 # editable install -meshbay-node==0.1.0 # editable install -msgpack==1.2.1 # wire serialisation -PyJWT==2.13.0 # JWT EdDSA -pytest==9.1.1 # test runner -pytest-asyncio==1.4.0 # async test support -uvicorn==0.52.1 # ASGI server (local UI) -watchdog==6.0.0 # filesystem watcher -zstandard==0.25.0 # zstd compression -``` - -Also installed transitively: pydantic 2.13.4, starlette 1.6.0, anyio 4.14.2, uvloop 0.22.1. - -### meshbay_common/crypto.py — Argon2id NOTE - -Current params: `iterations=3, memory_cost=65536` (64MB) → ~78ms on dev laptop. -**Must increase to `memory_cost=262144` (256MB) before production keystore use.** -Run `meshbay-node calibrate-argon2` on target hardware to tune. - -### Out of scope for Phase 2 - -Multiple groups, chat, QUIC, module sandbox, mobile pairing, HLS streaming. - ---- - -## Phase 3 — Hub v1 production ✅ DONE - -**Goal:** replace POC in-memory hub with a production-ready service on meshbay.org. -PostgreSQL persistence, HTTPS via Caddy, all endpoints hardened, legal IP logging, deploy. - -### Environment - -- **Server:** meshbay.org — Ubuntu 26.04 LTS, Python 3.14.4, OVH VPS -- **Database:** PostgreSQL 16 (to install) -- **Proxy:** Caddy (to install — handles Let's Encrypt automatically) -- **Service:** systemd `meshbay-hub.service` - -### Python dependencies to add (Phase 3) - -``` -sqlalchemy>=2.0 # async ORM (SQLAlchemy 2.x) -alembic>=1.13 # DB migrations -asyncpg>=0.30 # PostgreSQL async driver -aiosqlite>=0.20 # SQLite async driver (tests only) -slowapi>=0.1 # rate limiting (FastAPI middleware) -``` - -### Milestones - -| # | Component | File(s) | Status | -|---|---|---|---| -| 3.1 | DB models | `meshbay_hub/db/models.py` | ✅ | -| 3.2 | DB engine + session | `meshbay_hub/db/engine.py` | ✅ | -| 3.3 | Alembic migrations | `meshbay_hub/db/migrations/` | ✅ initial_schema | -| 3.4 | Hub config | `meshbay_hub/config.py` | ✅ | -| 3.5 | Auth (JWT + Argon2id) | `meshbay_hub/auth.py` | ✅ | -| 3.6 | API deps | `meshbay_hub/api/deps.py` | ✅ | -| 3.7 | Hub info router | `meshbay_hub/api/hub.py` | ✅ | -| 3.8 | Users router | `meshbay_hub/api/users.py` | ✅ | -| 3.9 | Nodes router | `meshbay_hub/api/nodes.py` | ✅ | -| 3.10 | Groups router | `meshbay_hub/api/groups.py` | ✅ | -| 3.11 | Rate limiting | `meshbay_hub/api/middleware.py` | ✅ slowapi | -| 3.12 | App factory + lifespan | `meshbay_hub/app.py` | ✅ | -| 3.13 | Hub daemon CLI | `meshbay_hub/daemon.py` | ✅ | -| 3.14 | PostgreSQL 16 | meshbay.org | ✅ DB: meshbay_hub | -| 3.15 | Caddy + Let's Encrypt | meshbay.org Caddyfile | ✅ HTTPS auto-cert | -| 3.16 | Systemd service | `/etc/systemd/system/meshbay-hub.service` | ✅ | -| 3.17 | Deploy + smoke test | https://meshbay.org | ✅ 11/11 endpoints | - -**Total: 40 local tests (SQLite) + 11/11 smoke tests HTTPS production** - -### Phase 3 deployment details (meshbay.org) - -- PostgreSQL 16, user `meshbay`, DB `meshbay_hub` -- Caddy auto-handles Let's Encrypt for `meshbay.org` and `www.meshbay.org` -- `www.meshbay.org` → 301 redirect → `meshbay.org` -- TLS setup documented in `docs/HTTPS.md` -- Hub listens on `127.0.0.1:8000`, Caddy proxies 80/443 -- Systemd service: `meshbay-hub.service` (restart-on-failure) -- Hub config: `~/.config/meshbay/hub.toml` -- Hub keypair: `~/.config/meshbay/hub_private.pem` (chmod 600) -- Source deployed at: `~/meshbay-hub/` (common_pkg + hub_pkg) - -### Additional Python dependencies added in Phase 3 - -``` -sqlalchemy==2.0.51 # async ORM -alembic==1.19.1 # DB migrations -asyncpg==0.31.0 # PostgreSQL async driver -aiosqlite # SQLite async (tests only) -slowapi==0.1.10 # rate limiting -hatchling==1.31.0 # build backend (needed for pip install) -``` - -### Notes - -- Initial DB schema created via `init_db()` (`create_all`) — Alembic tracks future changes -- IP logging records: account_create, login, login_fail, group_create, node_announce -- Rate limiting active on /v1/users/register and /v1/users/login (slowapi) -- Revocation, CSAM hash matching, moderation deferred to Phase 5 - -### Phase 3 scope - -**In scope:** -- All POC endpoints from Spike 2+6, production-ready -- PostgreSQL via SQLAlchemy async + Alembic migrations -- IP logging (creation, login, group events) — 1-year retention, legal compliance -- Access token (JWT, 1h) + refresh token (30d, stored hashed) -- Rate limiting on auth endpoints -- Hub config file `/etc/meshbay/hub.toml` or env vars -- HTTPS via Caddy + auto Let's Encrypt on meshbay.org -- Systemd service with restart-on-failure - -**Deferred to later:** -- Revocation push (WebSocket signaling to nodes) -- CSAM hash matching (NCMEC/IWF integration) -- Moderation flow (blocklist + takedown) -- MHP federation -- RPM/DEB packaging - -### Testing strategy - -- Unit tests with SQLite in-memory (`aiosqlite`) — no PostgreSQL needed locally -- API tests via `httpx.AsyncClient` + `ASGITransport` — no network -- All tests run in the existing `.venv` after adding Phase 3 deps - ---- - -## Phase 4 — Integration & Web Client ✅ DONE - -**Goal:** end-to-end working product in a browser: login via hub, discover groups, -browse files on a node, download and stream public content. - -### Architecture decision - -Browsers cannot make raw TCP connections — the node must speak HTTP. -- Node adds an **HTTP file API** (port 19001) serving public content via standard `fetch()` -- Private content (GEK decrypt in browser) deferred to Phase 5 (requires WebCrypto + ChaCha20 WASM) -- Hub serves the web application at `meshbay.org/app/` - -### Milestones - -| # | Component | File(s) | Status | -|---|---|---|---| -| 4.1 | Node HTTP file API | `meshbay_node/transport/http_server.py` | ✅ 7/7 tests | -| 4.2 | Hub web app (HTML/JS) | `meshbay_hub/api/webapp.py` + `static/app.js` | ✅ live on meshbay.org | -| 4.3 | Group listing endpoint | `meshbay_hub/api/groups.py` GET /v1/groups | ✅ | -| 4.4 | End-to-end integration test | local node ↔ hub ↔ browser | ✅ smoke test | -| 4.5 | HLS streaming (public) | node `/hls/{id}/playlist.m3u8` + `.ts` via ffmpeg | ✅ included in 4.1 | - -**Total: 47/47 tests. https://meshbay.org live with web client.** - -### Phase 4 deployment - -- `https://meshbay.org/` — web app (HTML/JS SPA) -- `https://meshbay.org/app.js` — JS client -- `https://meshbay.org/v1/groups` — public group listing (no auth) -- Node HTTP API (port 19001): `/index`, `/file/{id}`, `/hls/{id}/*.m3u8`, `/hls/{id}/*.ts` -- HLS streaming uses ffmpeg for on-the-fly segmentation -- Public content: no auth for index, auth required for chunks -- Private content (GEK decrypt in browser): deferred to Phase 5 - -### Dependencies added in Phase 4 - -``` -# Node (runtime) -ffmpeg (system package) — HLS segmentation via subprocess -``` - -### Phase 4 scope - -**In scope (public content only):** -- Node HTTP API: serve public Mesh Group Index (JSON) + file chunks (binary) -- Hub web app: login, group discovery, file browser, download link -- HLS basic streaming: node segments video on-the-fly, browser plays natively -- JWT auth passed as query param or header to node HTTP API - -**Deferred to Phase 5:** -- Private group content in browser (requires ChaCha20-Poly1305 via WASM) -- Chat UI -- Multi-group node -- Android client - ---- - -## Phase 5 — QUIC, Federation, Moderation ✅ DONE (Mobile deferred) - -**Goal:** full decentralization, mobile support, community infrastructure. - -| # | Component | Notes | -|---|---|---| -| # | Component | File(s) | Status | -|---|---|---|---| -| 5.1 | QUIC transport (MNP v2) | `transport/quic_server.py` + `quic_client.py` | ✅ 3/3 tests | -| 5.2 | MHP federation | `meshbay_hub/api/federation.py` | ✅ GET/POST /mhp/* | -| 5.3 | Mesh Relay | `meshbay_hub/api/relay.py` | ✅ register+list+approve | -| 5.4 | Android client | — | ⏳ deferred (different tech) | -| 5.5 | Content replication | — | ⏳ deferred | -| 5.6 | iOS client | — | ⏳ after Android | -| 5.7 | Revocation push | `api/revocation.py` + `node/revocation.py` | ✅ 3/3 tests | -| 5.8 | CSAM hash matching | `meshbay_hub/csam.py` | ✅ CSAMChecker + admin API | -| 5.9 | Moderation | `api/moderation.py` | ✅ 6/6 tests | -| 5.10 | RPM/DEB packaging | `packaging/` | ✅ spec + control + systemd | - -**Total: 59/59 tests.** - -### Phase 5 bugs fixed - -- **QUIC**: `asyncio.Event` race condition in client recv loop (quic_event_received - overwrote `_stream_events[0]` created by `_recv`). Fixed with `asyncio.Queue` - (no shared mutable state between coroutines). -- **QUIC**: `verify_peer` parameter renamed to `verify_mode` in aioquic 1.3.0. -- **QUIC**: `connect()` returns protocol directly (not `(transport, proto)` tuple). - -### Phase 5 dependencies added - -``` -aioquic==1.3.0 # QUIC transport (Cloudflare-maintained) -websockets # hub→node revocation push -``` - -### Deferred to future - -- Android/iOS client (Kotlin/Flutter — different tech stack, dedicated effort) -- Content replication (node-to-node) → Phase 6 -- MHP federation persistence (currently in-memory) → Phase 6 - ---- - -## Phase 6 — Chat, Multi-group, Federation persistence, Replication ✅ DONE - -**Goal:** complete the product with group chat (Double Ratchet), multi-group node support, -persistent MHP federation, content replication, and browser private group decryption. - -### Milestones - -| # | Component | File(s) | Status | -|---|---|---|---| -| 6.1 | Double Ratchet chat | `meshbay_common/ratchet.py` | ✅ 11/11 tests | -| 6.2 | Multi-group node | `meshbay_node/config.py` [[groups]] | ✅ | -| 6.3 | MHP federation persistence | `FederatedGroup` + `SwarmSource` DB tables | ✅ | -| 6.4 | Content replication | `node/replication.py` + hub `/v1/swarm/*` | ✅ | -| 6.5 | Browser private group | `webcrypto.py` + `static/crypto.js` | ✅ 4/4 tests | -| 6.6 | Dérivation clés depuis password | `meshbay_common/keyderive.py` + `static/keyderive.js` | ✅ 7/7 tests | -| 6.7 | Bundle clés chiffré (web) | Hub: `keypair_bundle` field + migration Alembic | ✅ | -| 6.8 | Scripts démo opérationnels | `QE/demo-v1/` + `QE/demo-v2/` (non versionné) | ✅ testés | -| 6.9 | QUICKSTART réécrit | `docs/QUICKSTART.md` | ✅ | -| 6.6 | Dérivation clés depuis password | `meshbay_common/keyderive.py` + `static/keyderive.js` | ✅ 7/7 tests | -| 6.7 | Bundle clés chiffré (web) | Hub: `keypair_bundle` field + migration Alembic | ✅ | -| 6.8 | Scripts démo opérationnels | `QE/demo-v1/` (non versionné) | ✅ testés | -| 6.9 | QUICKSTART réécrit | `docs/QUICKSTART.md` | ✅ | - -**Total: 81/81 tests.** - ---- - -## Phase 7 — Node v2: production, streaming, chat ✅ DONE - -**Goal:** multi-group node, Sender Keys chat, QUIC 0-RTT, jti denylist push, HLS streaming. - -Commit: fc56585 — 26 files, +2155/−159 lines, 109 tests. - -See `devel-phases-next.md` for details. - ---- - -## Phase 8 — Hub v2: admin, federation, security ✅ DONE - -**Goal:** admin roles, email encryption, refresh token rotation, rate limiting, healthcheck. - -Commit: 46918ec — 20 files, +508/−90 lines, 117 tests. -Deployed to meshbay.org. All security review items S1/S2/S5 resolved. - -See `devel-phases-next.md` for details. - ---- - -## Phase 9 — Web client: WebRTC transport + core SPA ✅ DONE - -**Goal:** browser connects P2P to a node behind residential NAT via WebRTC DataChannel. -Full SPA: login, groups, file browser, download, video playback, chat, i18n, dark/light. - -### Milestones - -| # | Component | Status | -|---|---|---| -| 9.1–9.5 | WebRTC DataChannel spike + E2E NAT validation | ✅ | -| 9.6–9.12 | Preact SPA (login, groups, files, video, chat, i18n, settings) | ✅ | -| 9.13 | Tests: 132 passing | ✅ | -| 9.14–9.16 | Performance: pipelining, binary wire format, I/O reduction | ✅ | -| 9.17 | Large file download: File System Access API (stream to disk) | ✅ | - -**Total: 132/132 tests. Deployed to meshbay.org + Orange node (2026-08-11).** - -### Key technical decisions - -- **Transport:** WebRTC DataChannel (aiortc on node) — browsers can't use QUIC for NAT traversal -- **Wire format:** length-prefixed msgpack, binary chunk fields (no base64) -- **UI:** Preact + htm ESM (vendored, no build step, no CDN, no npm) -- **Crypto:** WebCrypto SubtleCrypto AES-256-GCM for E2E chunk decryption in browser -- **Large files:** File System Access API (`showSaveFilePicker`) — stream to disk, ~8 MB RAM -- **Indexer:** 2s debounce + path-based dedup for file copy events - -### NAT traversal validated - -Two ISPs (SFR + Orange residential NAT), Chrome + Firefox, IPv4 STUN + IPv6 direct. -No TURN relay needed. See `devel-phases-next.md` for detailed test matrix. - -### QE deployment state (2026-08-11) - -- **Hub (meshbay.org):** running as `meshbay-hub.service`, DB has 3 users - (admin, cbesson, grenet), 1 group (`d3bbd90b`), `admin_usernames = ["admin"]` -- **Node (Orange host via `ssh cbesson@localhost -p 2222`):** running as - `nohup .venv/bin/python3 QE/demo-v3/run_node_simple.py`, user grenet, - connected via WS to hub, WebRTC + QUIC dual transport -- **Credentials:** `QE/demo-v3/creds.json` (not versioned) -- **Shared dir on node:** `~/meshbay/QE/demo-v3/shared/` -- All 3 users password: see creds.json - -### Dependencies added - -- `aiortc>=1.9` in meshbay-node (WebRTC DataChannel) -- `preact` + `htm` vendored as `static/vendor/htm-preact.js` (ESM, ~3 KB gzipped) - ---- - -## Conventions - -- Commits: `feat(node):`, `fix(hub):`, `chore(common):`, `docs:`, `test(node):` -- Branch per feature/fix, merge to `main` (when remote configured) -- **Always close test UFW ports after any spike on meshbay.org** -- **Never commit key material** (keystore.enc, hub_private.pem, *.key, node_state.json, unlock.key) -- meshbay.org is internet-facing: only run known-safe services, close ports after tests |