diff options
Diffstat (limited to 'devel-phases.md')
| -rw-r--r-- | devel-phases.md | 171 |
1 files changed, 123 insertions, 48 deletions
diff --git a/devel-phases.md b/devel-phases.md index 201f5f6..1d5424b 100644 --- a/devel-phases.md +++ b/devel-phases.md @@ -36,61 +36,132 @@ ## Phase 2 — Node v1 ✅ DONE -**Goal:** a working Mesh Node that indexes a directory, registers with the hub, -serves encrypted chunks over TCP+TLS, and has a minimal local web UI. +**Goal:** working Mesh Node: indexes a directory, registers with hub, +serves encrypted chunks over TCP+TLS, local web UI on localhost:18000. -**Transport:** TCP+TLS (QUIC added in v2). -**TLS:** self-signed cert per node. Client verifies node identity via Ed25519 PK from hub, -not via cert chain. Client uses `CERT_NONE` ssl context. +**Transport:** TCP+TLS 1.3 (QUIC in v2). Self-signed cert per node. +Node identity verified via Ed25519 PK from hub, not TLS cert chain (client uses CERT_NONE). ### Milestones -| # | Component | File(s) | Status | +| # | Component | File(s) | Tests | |---|---|---|---| -| 2.1 | Keystore | `meshbay_node/keystore.py` | ✅ 10/10 tests | -| 2.2 | Hub client | `meshbay_node/hub_client.py` | ✅ 6/6 tests | -| 2.3 | Directory indexer | `meshbay_node/indexer/indexer.py` | ✅ 10/10 tests | -| 2.4 | Mesh Group Index | `meshbay_node/indexer/group_index.py` | ✅ included above | -| 2.5 | TCP+TLS chunk server | `meshbay_node/transport/server.py` | ✅ 3/3 tests | -| 2.6 | Daemon + config | `meshbay_node/daemon.py`, `config.py` | ✅ | -| 2.7 | Local web UI | `meshbay_node/ui/app.py` | ✅ | +| 2.1 | Keystore | `meshbay_node/keystore.py` | 10/10 | +| 2.2 | Hub client | `meshbay_node/hub_client.py` | 6/6 | +| 2.3 | Directory indexer | `meshbay_node/indexer/indexer.py` | 5/5 | +| 2.4 | Mesh Group Index | `meshbay_node/indexer/group_index.py` | 5/5 | +| 2.5 | TCP+TLS chunk server | `meshbay_node/transport/server.py` | 3/3 | +| 2.6 | TCP+TLS chunk client | `meshbay_node/transport/client.py` | included above | +| 2.7 | TLS cert helper | `meshbay_node/transport/tls_cert.py` | — | +| 2.8 | Config (TOML + env) | `meshbay_node/config.py` | — | +| 2.9 | Local web UI | `meshbay_node/ui/app.py` | — | +| 2.10 | Daemon + CLI | `meshbay_node/daemon.py` | — | -### Node v1 scope +**Total: 29/29 tests passing** -**In scope:** -- Single group, single shared directory -- GEK loaded from hub bundle (hub must be running) -- Serve file chunks to authenticated clients (JWT bearer) -- Mesh Group Index: encrypted, delta updates -- Local web UI: status, file list, group info -- Keystore: Argon2id + AES-256-GCM, 3 unlock modes (password / file / env var) -- Config file: `~/.config/meshbay/node.toml` -- Systemd service unit +### Python dependencies (dev venv — `/home/cbesson/meshbay/.venv`) + +Installed packages (freeze) as of Phase 2 completion: + +``` +aioice==0.10.2 # ICE/STUN for NAT traversal +blake3==1.0.9 # fast content hashing +cryptography==50.0.0 # Ed25519, X25519, ChaCha20, Argon2id, AES-GCM +fastapi==0.141.1 # local web UI + hub POC +httpx==0.28.1 # hub client HTTP +meshbay-common==0.1.0 # editable install +meshbay-node==0.1.0 # editable install +msgpack==1.2.1 # wire serialisation +PyJWT==2.13.0 # JWT EdDSA +pytest==9.1.1 # test runner +pytest-asyncio==1.4.0 # async test support +uvicorn==0.52.1 # ASGI server (local UI) +watchdog==6.0.0 # filesystem watcher +zstandard==0.25.0 # zstd compression +``` + +Also installed transitively: pydantic 2.13.4, starlette 1.6.0, anyio 4.14.2, uvloop 0.22.1. + +### meshbay_common/crypto.py — Argon2id NOTE -**Out of scope for v1:** -- Multiple groups or directories -- Chat / Double Ratchet -- QUIC transport -- Module sandbox -- Mobile pairing -- Streaming (HLS) — chunks are raw, player integration later +Current params: `iterations=3, memory_cost=65536` (64MB) → ~78ms on dev laptop. +**Must increase to `memory_cost=262144` (256MB) before production keystore use.** +Run `meshbay-node calibrate-argon2` on target hardware to tune. + +### Out of scope for Phase 2 + +Multiple groups, chat, QUIC, module sandbox, mobile pairing, HLS streaming. --- -## Phase 3 — Hub v1 production +## Phase 3 — Hub v1 production 🚧 IN PROGRESS -**Goal:** replace POC in-memory hub with a production-ready service. +**Goal:** replace POC in-memory hub with a production-ready service on meshbay.org. +PostgreSQL persistence, HTTPS via Caddy, all endpoints hardened, legal IP logging, deploy. -| # | Component | Notes | -|---|---|---| -| 3.1 | PostgreSQL persistence | SQLAlchemy async + asyncpg, Alembic migrations | -| 3.2 | HTTPS via Caddy | Auto Let's Encrypt, ProxyPass to uvicorn on 127.0.0.1:8000 | -| 3.3 | Full user/group management | All POC endpoints hardened, input validation, rate limiting | -| 3.4 | IP logging | Legal compliance: creation, login, group events (1-year retention) | -| 3.5 | Revocation | Signed revocation tokens, hub → node push via WebSocket | -| 3.6 | Moderation | blake3 blocklist, DMCA takedown endpoint, escalation flow | -| 3.7 | CSAM hash matching | NCMEC/IWF integration on public content registration | -| 3.8 | Systemd service + RPM/DEB | meshbay-hub package, `/etc/meshbay/hub.toml` | +### Environment + +- **Server:** meshbay.org — Ubuntu 26.04 LTS, Python 3.14.4, OVH VPS +- **Database:** PostgreSQL 16 (to install) +- **Proxy:** Caddy (to install — handles Let's Encrypt automatically) +- **Service:** systemd `meshbay-hub.service` + +### Python dependencies to add (Phase 3) + +``` +sqlalchemy>=2.0 # async ORM (SQLAlchemy 2.x) +alembic>=1.13 # DB migrations +asyncpg>=0.30 # PostgreSQL async driver +aiosqlite>=0.20 # SQLite async driver (tests only) +slowapi>=0.1 # rate limiting (FastAPI middleware) +``` + +### Milestones + +| # | Component | File(s) | Status | +|---|---|---|---| +| 3.1 | DB models | `meshbay_hub/db/models.py` | 🚧 | +| 3.2 | DB engine + session | `meshbay_hub/db/engine.py` | 🚧 | +| 3.3 | Alembic migrations | `meshbay_hub/db/migrations/` | 🚧 | +| 3.4 | Hub config | `meshbay_hub/config.py` | 🚧 | +| 3.5 | Auth (JWT + Argon2id) | `meshbay_hub/auth.py` | 🚧 | +| 3.6 | API deps | `meshbay_hub/api/deps.py` | 🚧 | +| 3.7 | Hub info router | `meshbay_hub/api/hub.py` | 🚧 | +| 3.8 | Users router | `meshbay_hub/api/users.py` | 🚧 | +| 3.9 | Nodes router | `meshbay_hub/api/nodes.py` | 🚧 | +| 3.10 | Groups router | `meshbay_hub/api/groups.py` | 🚧 | +| 3.11 | IP logging middleware | `meshbay_hub/api/middleware.py` | 🚧 | +| 3.12 | App factory | `meshbay_hub/app.py` | 🚧 | +| 3.13 | Hub daemon CLI | `meshbay_hub/daemon.py` | 🚧 | +| 3.14 | Install PostgreSQL | meshbay.org | 🚧 | +| 3.15 | Caddy + TLS | meshbay.org Caddyfile | 🚧 | +| 3.16 | Systemd service | `meshbay-hub.service` | 🚧 | +| 3.17 | Deploy + smoke test | meshbay.org | 🚧 | + +### Phase 3 scope + +**In scope:** +- All POC endpoints from Spike 2+6, production-ready +- PostgreSQL via SQLAlchemy async + Alembic migrations +- IP logging (creation, login, group events) — 1-year retention, legal compliance +- Access token (JWT, 1h) + refresh token (30d, stored hashed) +- Rate limiting on auth endpoints +- Hub config file `/etc/meshbay/hub.toml` or env vars +- HTTPS via Caddy + auto Let's Encrypt on meshbay.org +- Systemd service with restart-on-failure + +**Deferred to later:** +- Revocation push (WebSocket signaling to nodes) +- CSAM hash matching (NCMEC/IWF integration) +- Moderation flow (blocklist + takedown) +- MHP federation +- RPM/DEB packaging + +### Testing strategy + +- Unit tests with SQLite in-memory (`aiosqlite`) — no PostgreSQL needed locally +- API tests via `httpx.AsyncClient` + `ASGITransport` — no network +- All tests run in the existing `.venv` after adding Phase 3 deps --- @@ -104,7 +175,7 @@ not via cert chain. Client uses `CERT_NONE` ssl context. | 4.2 | End-to-end integration test | Node ↔ Hub ↔ Client full flow | | 4.3 | HLS/DASH streaming | Node segments on-the-fly, per-segment GEK-derived key | | 4.4 | Chat (Double Ratchet) | Group messaging with attachments, Signal-like | -| 4.5 | Multi-group support | Node hosts N groups across M directories | +| 4.5 | Multi-group + multi-dir | Node hosts N groups across M directories | --- @@ -120,13 +191,17 @@ not via cert chain. Client uses `CERT_NONE` ssl context. | 5.4 | Android client | Kotlin/Flutter, hub account, group access, node pairing | | 5.5 | Content replication | Node-to-node, admin-authorized, no hub involvement | | 5.6 | iOS client | After Android stabilizes | -| 5.7 | Argon2id calibration CLI | `meshbay-node --calibrate-argon2` tuning command | +| 5.7 | Revocation push | WebSocket hub→node signed token broadcast | +| 5.8 | CSAM hash matching | NCMEC/IWF integration on public content registration | +| 5.9 | Moderation flow | blake3 blocklist, DMCA takedown, escalation | +| 5.10 | RPM/DEB packaging | meshbay-hub, meshbay-node, python3-meshbay-common | --- -## Conventions reminder +## Conventions - Commits: `feat(node):`, `fix(hub):`, `chore(common):`, `docs:`, `test(node):` -- Branch per feature/fix, merge to `main` via PR (when remote is set up) -- Close test UFW ports after any spike on meshbay.org -- Never commit key material (keystore.enc, hub_private.pem, *.key, node_state.json) +- Branch per feature/fix, merge to `main` (when remote configured) +- **Always close test UFW ports after any spike on meshbay.org** +- **Never commit key material** (keystore.enc, hub_private.pem, *.key, node_state.json, unlock.key) +- meshbay.org is internet-facing: only run known-safe services, close ports after tests |