diff options
Diffstat (limited to 'docs/meshbay-draft-v1.md')
| -rw-r--r-- | docs/meshbay-draft-v1.md | 442 |
1 files changed, 0 insertions, 442 deletions
diff --git a/docs/meshbay-draft-v1.md b/docs/meshbay-draft-v1.md deleted file mode 100644 index 1eb1758..0000000 --- a/docs/meshbay-draft-v1.md +++ /dev/null @@ -1,442 +0,0 @@ -# MeshBay — Architecture Draft v1 - -> Status: preliminary draft — many points still open, marked [TBD] - ---- - -## 1. Project Overview - -MeshBay is a decentralized, peer-to-peer platform for file sharing, video streaming, and group messaging. It combines identity federation (via Mesh Hubs) with truly peer-to-peer data exchange (via Mesh Nodes), aiming to be resilient, censorship-resistant, and user-friendly. - -**Core principles:** -- Data never transits through a central server — only identity and routing do -- End-to-end encryption for all private content (files, indexes, messages) -- The node operator is the legal host and is fully responsible for their content -- The hub is a lightweight registrar, not a content host or indexer -- Open source, self-hostable at every level - -**Domain:** meshbay.org - ---- - -## 2. Terminology - -| Term | Role | -|---|---| -| **Mesh Hub** | Identity authority and group registry server | -| **Mesh Node** | Local program on the host user's machine | -| **Mesh Client** | Web browser or Android app (end user) | -| **Mesh Relay** | Community-operated TURN fallback relay | -| **MNP** | Mesh Node Protocol — P2P protocol between nodes and clients | -| **MHP** | Mesh Bay Hub Protocol — inter-hub federation protocol | -| **GEK** | Group Encryption Key — symmetric key for group content | -| **Mesh Directory** | Public registry of groups (hub level) | -| **Mesh Group Index** | Encrypted file listing for a group (node level) | - ---- - -## 3. System Components - -### 3.1 Mesh Hub - -A lightweight server acting as a registrar. It is intentionally kept minimal to reduce legal exposure and operational burden. - -**What the hub stores:** -- User accounts: username, hashed email, `PK_user` (public key fingerprint), hub ID, status -- Group registry: name, `PK_group`, hosting node address, visibility, member list with encrypted GEK bundles -- Revocation lists (users and groups) -- Registered peer hubs (explicit allowlist — no auto-discovery) - -**What the hub never stores:** -- File content or metadata -- Private group indexes -- Message content -- Node IP addresses (handled by ephemeral signaling service) - -**Hub interactions — when is it called?** - -| Event | Hub load | Frequency | -|---|---|---| -| Account creation | Hash credential, store PK | Once | -| Login | Verify credentials, issue signed JWT | Per session (~30-day validity) | -| Group creation | Register name, PK_group, node | Once per group | -| Member add/remove | Store/remove encrypted GEK bundle | On admin action | -| Group discovery | Return node address + PK_node + GEK bundle | Per initial access | -| NAT signaling | Relay a few WebSocket messages (<1 KB) | Per new P2P connection | -| Public search | Delegate query to nodes at request time | On demand | -| MHP federation sync | Exchange public directory updates | Background, periodic | -| Revocation | Issue signed revocation token | Rare | - -**The hub is never in the data path after initial connection setup.** - -**JWT as offline passport:** -The hub issues a JWT signed with its Ed25519 private key. Nodes verify this JWT locally using the hub's known public key — no hub roundtrip required per request. JWT validity: ~30 days. - -**Tech stack:** -- Language: Python -- Framework: FastAPI + Uvicorn -- Database: PostgreSQL + SQLAlchemy + Alembic -- Deployment: behind Apache reverse proxy (ProxyPass) -- Authentication: own system (JWT signed with Ed25519, no OAuth dependency) - -**Account creation:** [TBD] — email only at first, phone number associable later. Via Android app, both collected by default. Fusionable accounts. - -### 3.2 Mesh Node - -A local program running on the host user's machine. The node is the actual host of all content. - -**Responsibilities:** -- Watch and index shared directories (Mesh Group Index) -- Serve files and video streams to group members -- Manage all cryptographic keys locally (keystore, password-protected) -- Handle P2P connections and NAT traversal -- Run the MNP protocol -- Host the Python module sandbox -- Serve the local web UI (localhost) -- Receive and redistribute ephemeral video from mobile [future] - -**Platform:** Linux primary, cross-platform from the start (Windows/macOS). Python ensures portability. - -**Tech stack:** -- Language: Python (primary), Rust extensions only if strictly necessary for hot paths -- QUIC: `aioquic` -- ICE/STUN: `aioice` -- WebRTC (future): `aiortc` -- Crypto: `cryptography` (PyCA, OpenSSL-backed, hardware-accelerated) -- Serialization: `msgpack` -- Compression: `zstandard` (zstd) -- File watching: `watchdog` -- Local DB: SQLite -- Local web UI: served by the node on localhost (port [TBD]) - -**Node pairing with mobile:** QR code from local web UI [future]. - -### 3.3 Mesh Client - -Web browser or Android app. Consumes content from the node; manages account via the hub. - -**Hub-side operations (via hub):** -- Account creation and login -- Public group search and discovery -- Group membership management - -**Node-side operations (direct P2P):** -- File browsing (Mesh Group Index) -- Message feed reading (with attachments, Signal-like) -- File download -- Video streaming (VOD) -- [Future] Ephemeral video feed - -**Client modes** [to be designed]: -- Explorer mode: browse files in a group -- Feed mode: message thread with attachments -- Hub/Node UI split to be defined - -### 3.4 Mesh Relay - -Community-operated TURN relay. Used only as last-resort fallback when all P2P connection methods fail. Traffic is always E2E encrypted — the relay sees only opaque QUIC packets and cannot read content. - -Not operated by meshbay.org. A protocol for relay registration with hubs is [TBD]. - ---- - -## 4. Group Model - -Groups are the core organizational unit. - -| Parameter | Options | -|---|---| -| Visibility | Public / Private | -| Join policy | Open / On request / By invitation only | -| Admin | The hosting node operator (legal host) | - -A public group functions like a themed forum: shared files, message thread, member list. It can be open entry, request-based, or invitation-only regardless of its public visibility. - -A private group's content (files, index, messages) is always E2E encrypted with the GEK. Only members with the GEK can decrypt anything. - -**Group addressing** [TBD]: -``` -meshbay.org/u/username/groupname — public group via hub -meshbay.org/g/groupname — direct public group -group://<PK_group_fingerprint>@<node_addr> — hub-less direct access -``` - ---- - -## 5. Cryptographic Architecture - -### 5.1 Key Hierarchy - -``` -User Identity Key Ed25519 Signing, authentication -User Exchange Key X25519 Key agreement -Group Identity Key Ed25519 Group metadata signing (held by admin node) -Group Encryption Key ChaCha20 Content and index encryption (symmetric, 256-bit) -Session Keys X25519/HKDF Perfect forward secrecy per P2P connection -``` - -All private keys are stored exclusively on the node (or client device), in a password-protected local keystore. The hub never sees any private key. - -### 5.2 GEK Management - -**Group creation:** -1. Admin node generates GEK (ChaCha20-Poly1305, 256-bit, CSPRNG) -2. GEK is encrypted for each member using X25519 key agreement + HKDF -3. Encrypted GEK bundles stored on hub (or node — [TBD]) - -**Member addition:** -- GEK encrypted with new member's `PK_user` and distributed - -**Member revocation:** -- Admin node generates a new GEK -- Re-encrypts for all remaining members -- New content encrypted with new GEK -- Former member retains ability to decrypt previously received content (acceptable trade-off — full re-encryption not planned) - -### 5.3 On-the-Fly Encryption for File Transfer - -Files are stored in plaintext on the host's disk. The node encrypts at read time before transmission. - -``` -Disk (plaintext) → [Node] → zstd compress → GEK encrypt (per-chunk) → QUIC session → [Client] → QUIC decrypt → GEK decrypt → plaintext -``` - -**Chunking strategy:** -- Chunk size: 1 MB (amortizes AEAD overhead, allows seeking) -- Per-chunk key derivation: - `chunk_key = HKDF(GEK, "file:" || blake3(file) || "chunk:" || index)` -- Each chunk independently decryptable (enables video seeking) -- Compress before encrypt (zstd compression is useless after encryption) - -**Chunk authentication:** -Each chunk (or batch) is signed with the node's Ed25519 key. The client verifies before decryption. Prevents data injection by a compromised relay. - -### 5.4 Transport Security - -- Primary protocol: **QUIC** (TLS 1.3 integrated, UDP-based, multiplexed) -- Per-connection session keys via X25519 ECDH + HKDF -- The QUIC layer is independent from the GEK application layer — two independent encryption layers - -### 5.5 Chat Encryption - -Group messaging uses the **Double Ratchet algorithm** (as in Signal): -- Forward secrecy and break-in recovery per message -- Each message independently encrypted -- Implementation: existing Python or Rust library [TBD] - ---- - -## 6. Network and Connectivity - -### 6.1 NAT Traversal — Attempt Order - -``` -1. IPv6 available on both sides → direct connection, no NAT issue -2. UPnP / NAT-PMP on router → node opens port automatically -3. ICE + STUN / UDP hole punching → works for ~80-85% of cases -4. Mesh Relay (TURN fallback) → community-operated, E2E encrypted traffic -``` - -**Signaling** (steps 3/4): coordinated via hub WebSocket, <1 KB per attempt, stateless after connection established. - -**Step 4 coverage:** ~15-20% of connections (symmetric NAT on both sides, CGNAT). The relay sees only encrypted QUIC packets. - -### 6.2 MNP — Mesh Node Protocol - -Application-level protocol over QUIC. Defined blocks: - -- **Handshake**: key exchange, group membership verification (JWT presentation) -- **Index sync**: encrypted delta Mesh Group Index on connection -- **File transfer**: chunk request/response with hash verification -- **VOD streaming**: HLS/DASH segments, encrypted per-segment with GEK-derived keys -- **Messaging**: Double Ratchet messages encapsulated in MNP frames -- **[Future] Ephemeral stream**: `ephemeral_stream` type with TTL metadata - -### 6.3 Public Content Delivery - -Public files are identified by their `blake3` hash. Multiple nodes can serve the same file: - -1. Node A has public file X (hash H) -2. Any node that obtains X and chooses to mirror it registers with the hub: "I serve hash H" -3. Hub maintains: `{ blake3_hash → [node_A, node_B, ...] }` -4. Client requests X → hub returns source list → client fetches in parallel chunks from multiple nodes - -**Public content transport:** TLS only (no GEK). Content signed with the original node's Ed25519 key for authenticity verification by clients, even when served from a mirror. Door left open for GEK on "registered-users-only public" groups in a future revision. - ---- - -## 7. Indexes - -### 7.1 Mesh Directory (hub level) - -Public registry of groups. Exchanged between hubs via MHP. - -Format: msgpack, signed by hub's Ed25519 key. - -Fields per entry: group name, `PK_group`, hosting hub, description, content type tags, join policy. - -### 7.2 Mesh Group Index (node level) - -File listing for a group. Generated and maintained by the hosting node. - -Format: msgpack → zstd compressed → GEK encrypted (private groups) or plaintext signed (public groups). - -Entry structure: -```python -{ - "id": "<blake3_hash>", - "name": "filename.mkv", - "path": "Movies/2024/", # relative to shared directory - "size": 4294967296, - "type": "video", # video | audio | image | document | archive | other - "duration": 7245, # seconds, for media - "thumb_hash":"<blake3>", # thumbnail hash (thumbnail also GEK-encrypted) - "added_at": 1720000000 -} -``` - -**Delta updates:** each update carries `{base_version, additions, deletions}` — no full re-encryption on every change. - -**Transit:** nodes push index deltas to connected members on change. Members pull full index on first connection. Hub stores no index content — only the node address for routing. - -### 7.3 Search - -**Private groups:** search is entirely local on the client device. The client maintains a local encrypted cache of all indexes for groups it belongs to. No network call, no hub involvement, instant results. - -**Public groups:** client queries node(s) directly at request time. Hub provides routing (which node hosts which group) but performs no content lookup itself. - -**Hub web UI search:** delegates query to relevant nodes at request time. Hub stores nothing from this interaction. In-memory micro-cache of results: **60-second TTL maximum, RAM only, never written to disk, public content only.** This qualifies as technical caching (EU DSA Article 13) and does not constitute indexing. - ---- - -## 8. Hub Federation (MHP) - -### 8.1 Hub Hierarchy - -``` -Root Hub (meshbay.org) - ├── Full Hub (self-hosted, CA-delegated) - │ └── issues user credentials, manages its own groups - │ └── can federate with other Full Hubs via MHP - └── Mirror Hub - └── hosts public Mesh Directory only (no user accounts) -``` - -A Full Hub receives a certificate signed by the Root Hub (or a parent Full Hub) proving its authority. Clients verify the chain. A Mirror Hub can only replicate public directory data. - -### 8.2 MHP Design Principles - -- Explicit peer selection: each hub maintains an allowlist of trusted peer hubs -- No automatic hub discovery -- Exchanged data: Mesh Directory (public groups), revocation lists, cross-hub user credentials -- Cross-hub authentication: user from Hub A presents JWT signed by Hub A; Hub B verifies using Hub A's public key (fetched once on first interaction, cached) - -### 8.3 Cross-Hub Client Access - -Client from Hub A accessing a group on Hub B: -1. Hub A's public directory or direct link leads client to Hub B -2. Client presents Hub A JWT to Hub B directly -3. Hub B verifies JWT signature using Hub A's public key -4. Hub B issues a short-lived local token for this session -5. Client proceeds to node as normal - ---- - -## 9. Moderation - -### 9.1 Public Content - -``` -Report #1 → automatic suspension of public access to content - → node operator notified -One republication allowed -Report #2 → escalated to hub moderators -Confirmed → group revoked on local hub - → revocation propagated to federated hubs via MHP -``` - -Mechanism: blake3 hash of content added to hub blocklist. Node receives signed revocation notice and cuts public access. - -### 9.2 CSAM - -Hash matching against NCMEC/IWF database on all public content at registration time. Participation demonstrates good faith and significantly reduces legal exposure. No scanning of private/encrypted content. - -### 9.3 Copyright - -DMCA/legal notice framework (takedown on notification). No automated technical blocking — too complex, too many false positives (fair use, regional variations). Hub can revoke on confirmed legal request. - -### 9.4 Private Content - -Not directly moderatable (E2E encrypted by design). Only action available: revoke user or group at hub level on formal legal request. Hub issues a signed revocation token that all group members' nodes can verify. - ---- - -## 10. Python Module System - -The node can load extension modules (Python) that run in a sandboxed subprocess. - -**Module manifest** (declared capabilities): -```python -{ - "name": "group-chat", - "version": "1.0.0", - "permissions": ["read_index", "send_message", "receive_events"] -} -``` - -**Available APIs (restricted):** -- `read_index()` — read current group index (read-only) -- `send_message(content)` — post a message to the group thread -- `receive_events(handler)` — subscribe to group events (new file, new message) - -**Not available:** -- Arbitrary network access -- Filesystem access outside the group context -- System calls - -**First official module:** group chat thread (Signal-like, with attachments). Bundled with node. - ---- - -## 11. Legal Framework - -**Node operator:** primary legal host of content. Fully responsible for what they share. Node software clearly communicates this at setup. - -**Hub operator:** registrar, not content host. Stores minimal PII. Operates takedown mechanism. Participates in CSAM hash matching. Analogous to a domain registrar in legal exposure terms. - -**Protocol/software author:** protected by substantial non-infringing uses. No active facilitation of infringement. - -**Hub data minimization:** -- Email stored hashed after verification [TBD] -- No IP address logging (or auto-deletion after 24h) -- No content metadata stored -- Node current address managed by ephemeral signaling service only - ---- - -## 12. Future Features (noted, not designed) - -- **Content replication between nodes:** node-to-node, admin-authorized, no hub involvement -- **Mobile video push:** mobile films → pushes to hosting node → distributed as ephemeral stream with TTL to group members. MNP `ephemeral_stream` type reserved. -- **Mesh Relay registration protocol:** community TURN relays registered with hubs -- **Node mobile pairing:** QR code from local web UI -- **Multi-source download:** parallel chunk fetching from multiple nodes for same public file (swarm) -- **iOS client** -- **At-rest encryption on node:** optional, for nodes deployed on remote servers - ---- - -## 13. Open Questions [TBD] - -1. **GEK bundle storage:** on hub or on node only? Hub = easier discovery; node only = more decentralized -2. **Group address scheme:** final URL format -3. **Hub local web UI scope for V1:** config only, or also group browsing? -4. **Account creation:** email only to start, phone associable — confirm -5. **Chat implementation:** bundled module or core feature? -6. **QUIC library maturity:** `aioquic` production readiness assessment needed -7. **Double Ratchet library:** identify best Python implementation -8. **Relay registration protocol:** design when community relays are introduced -9. **Cross-hub directory exchange:** frequency, conflict resolution -10. **Node port for local web UI:** to assign -11. **JWT expiry and refresh strategy** -12. **Keystore format and unlock mechanism on node startup** |