diff options
70 files changed, 8192 insertions, 95 deletions
@@ -32,6 +32,7 @@ meshbay/ │ ├── meshbay-common/ # Shared crypto + protocol — python3-meshbay-common RPM (LGPL; rest AGPL) │ ├── meshbay-hub/ # Hub server (FastAPI + PostgreSQL) — meshbay-hub RPM │ ├── meshbay-node/ # Node daemon + local UI — meshbay-node RPM +│ ├── meshbay-cli/ # Command-line client for members (Python) — meshbay-cli package │ ├── meshbay-client/ # Desktop client (Electron) │ └── meshbay-android/ # Android client (WebView shell, Kotlin) — README.md ├── poc/ # POC spike scripts (reference, not production) @@ -59,7 +60,7 @@ source .venv/bin/activate # All dependencies are declared in the pyproject.toml files — a single pip install is enough pip install -e "packages/meshbay-common[dev]" -e "packages/meshbay-hub[dev]" \ - -e "packages/meshbay-node[dev]" + -e "packages/meshbay-node[dev]" -e "packages/meshbay-cli[dev]" ``` The key deps (aioquic, watchdog, fastapi, blake3, etc.) are in the `pyproject.toml` files @@ -99,7 +100,7 @@ in the right `pyproject.toml`. ### Package versions (SemVer) - Format: `MAJOR.MINOR.PATCH` - Pre-1.0: breaking changes bump MINOR, not MAJOR -- All three packages share the same version number (released together) +- All Python packages share the same version number (released together) ### Protocol versions (independent) @@ -143,11 +144,12 @@ AGPL-3.0-or-later.** The protocol layer is `meshbay-common` (`COPYING.LESSER` + `// SPDX-License-Identifier: LGPL-3.0-or-later`: `keyderive.js`, `crypto.js`, `playlist-crypto.js`, `transport.js`, `transport-*.js`; `meshbay-client/src/` `keyring.js`, `transcripts.js`, `argon2-wasm.js`; Android `keys/Kdf.kt`, -`Keyring.kt`, `Transcripts.kt`. A file without that line has its package's -licence: the AGPL (`LICENSE` at the root, copied into `meshbay-hub/` and -`meshbay-node/` because a wheel's `license-files` cannot reach outside its -package; into `static/licenses/` with the LGPL and GPL, so that every client -carries all three). +`Keyring.kt`, `Transcripts.kt`; `meshbay-cli`'s `client/keys.py` (first line +`# SPDX-License-Identifier: LGPL-3.0-or-later`). A file without that line has +its package's licence: the AGPL (`LICENSE` at the root, copied into +`meshbay-hub/`, `meshbay-node/` and `meshbay-cli/` because a wheel's +`license-files` cannot reach outside its package; into `static/licenses/` with +the LGPL and GPL, so that every client carries all three). - **An LGPL file depends only on LGPL files, permissive vendored code or the platform.** One import of an AGPL module and a client using the layer is under @@ -155,9 +157,9 @@ carries all three). `window.meshbay`, a `Secrets` store) is reached as an injected interface, never imported. `test_licensing.py` checks the closure. - **The same piece has the same licence on every platform.** A port is LGPL when - its original is (Keyring.kt ↔ keyring.js ↔ keyderive.js), and stays on the AGPL - side when its original is part of a shell (DeviceKey.kt ↔ the device key in - `main.js`). A new protocol file is added to the list here, to the README, and + its original is (Keyring.kt ↔ keyring.js ↔ keyderive.js ↔ the CLI's + `client/keys.py`), and stays on the AGPL side when its original is part of a + shell (DeviceKey.kt ↔ the device key in `main.js`). A new protocol file is added to the list here, to the README, and to `LGPL_FILES` in the test, with its SPDX line. The Android application adds a §7 permission for Google Play services (`meshbay-android/LICENSE-EXCEPTION.txt`). @@ -1152,6 +1154,22 @@ which pytest also runs when `ANDROID_HOME` is set (`test_android_shell.py`). | Downloads | `save/SaveSinks.kt`, `SaveNames.kt` | `OPENABLE` is `downloads.js`'s, compared by `test_android_downloads.py` | | Casting | `cast/CastRelay.kt`, `CastControl.kt`, `CastChannels.kt`, `CastService.kt`, `shell/ShellWebView.kt` | `MeshBayCast` in logcat says what the receiver and the relay did | +### Command-line client (`packages/meshbay-cli/`) + +| Need | File | Note | +|---|---|---| +| Actions and their parsers | `meshbay_cli/__main__.py` (`ACTIONS`), `commands/*.py` | §8.6. `test_every_action_is_run_here` refuses an action with no run in `test_cli_actions.py` | +| Exit codes | `meshbay_cli/errors.py` | **a contract for scripts**; the man page explains each (`manpage.EXIT_CODES`, tested) | +| Sign-in, renewal, the store | `client/account.py`, `client/store.py`, `client/hub.py` | renewal under the store's lock; the store key in the OS keyring, else a 0600 file | +| Passphrase keys, bundle opening | `client/keys.py` | **LGPL, parity-tested** with `keyderive.js` (`test_bundle_kdf_parity.py`) | +| Connection, handshake, identity, device linking | `client/session.py`, `client/groups.py`, `client/pool.py` | §5.2, §3.3 | +| Remote paths, listing, local names | `client/paths.py`, `client/tree.py`, `client/names.py` | `names.UPLOAD_NAME` is copied from the node's `SAFE_UPLOAD_NAME` | +| Downloads and uploads | `client/transfers.py`, `client/upload_plan.py` | §5.5; `upload_plan` is shared by `put` and the interactive mode | +| Interactive mode | `meshbay_cli/tui/` | Textual; names are drawn as text, never markup | +| Man page | `meshbay_cli/manpage.py` | generated from the parser at build time; `test_manpage.py` | +| What a frozen build must load | `meshbay_cli/selftest.py` | `MESHBAY_CLI_SELFTEST=1 meshbay-cli`, run by the Windows build | +| End to end against a real hub and node | `tests/test_e2e_local.py` | opt-in, `MESHBAY_CLI_E2E=1`; `MESHBAY_CLI_E2E_BIN` runs a packaged build instead | + ### Test harnesses that drive the real thing | Need | File | @@ -41,10 +41,11 @@ feature, and they are off on the reference deployment, [meshbay.org](https://mes | `packages/meshbay-common/` | Shared cryptography and protocol types | | `packages/meshbay-hub/` | Hub server (FastAPI + PostgreSQL) and the web client it serves | | `packages/meshbay-node/` | Node daemon, its CLI and its local control UI | +| `packages/meshbay-cli/` | Command-line client: a member's groups and files from a terminal | | `packaging/` | `.deb`, `.rpm` and Windows packaging, systemd units, Caddy and firewall configuration | | `docs/` | Design specification, protocol, HTTP API, user and operator guides | | `examples/` | Small Python programs using the hub, a node and its control API | -| `man/` | Manual page for `meshbay-node` | +| `man/` | Manual page for `meshbay-node` (`meshbay-cli`'s is generated from its parser at build time) | | `site/` | Static website pages (not deployed) | | `poc/` | Early proof-of-concept scripts, kept for reference | @@ -57,7 +58,7 @@ Python 3.12 or later. All dependencies are declared in the packages' python3 -m venv .venv source .venv/bin/activate pip install -e "packages/meshbay-common[dev]" -e "packages/meshbay-hub[dev]" \ - -e "packages/meshbay-node[dev]" + -e "packages/meshbay-node[dev]" -e "packages/meshbay-cli[dev]" .venv/bin/pytest ``` @@ -68,8 +69,8 @@ MeshBay is free software. | Component | Licence | |---|---| | `packages/meshbay-common/` — the protocol and its cryptography, in Python | [LGPL-3.0-or-later](packages/meshbay-common/COPYING.LESSER) (with the [GPL-3.0](packages/meshbay-common/COPYING) it builds on) | -| The same layer in the clients: the files marked `SPDX-License-Identifier: LGPL-3.0-or-later` — in the interface, `keyderive.js`, `crypto.js`, `playlist-crypto.js`, `transport.js` and `transport-*.js`; on the desktop, `keyring.js`, `transcripts.js` and `argon2-wasm.js`; on Android, `keys/Kdf.kt`, `keys/Keyring.kt` and `keys/Transcripts.kt` | LGPL-3.0-or-later | -| Everything else — hub, node, the rest of the interface, the desktop and Android shells | [AGPL-3.0-or-later](LICENSE) | +| The same layer in the clients: the files marked `SPDX-License-Identifier: LGPL-3.0-or-later` — in the interface, `keyderive.js`, `crypto.js`, `playlist-crypto.js`, `transport.js` and `transport-*.js`; on the desktop, `keyring.js`, `transcripts.js` and `argon2-wasm.js`; on Android, `keys/Kdf.kt`, `keys/Keyring.kt` and `keys/Transcripts.kt`; in the command-line client, `client/keys.py` | LGPL-3.0-or-later | +| Everything else — hub, node, command-line client, the rest of the interface, the desktop and Android shells | [AGPL-3.0-or-later](LICENSE) | The line is the protocol. Whatever a program needs to speak to a hub and a node — key derivation, the identity bundle, sealing and opening, what is signed, the diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md index 2174879..6dd1b2e 100644 --- a/docs/MESHBAY_DESIGN.md +++ b/docs/MESHBAY_DESIGN.md @@ -16,7 +16,7 @@ > them — it names the invariant that holds today, not the incident that produced > it. §13 is the register of those labels. > -> Wire versions at the time of writing: **MNP 6.2** (oldest peer accepted 4.0), +> Wire versions at the time of writing: **MNP 6.3** (oldest peer accepted 4.0), > **MHP 0.1**, packages **0.19.0**. The normative source for the wire format is > `MESHBAY_NODE_PROTOCOL.md`; this document states the design the protocol > serves, not its byte layout. @@ -302,6 +302,7 @@ comes from the hash binding: | Bound | Value | |---|---| | Request TTL | 1 h, `[node] device_request_ttl_minutes` | +| Pending requests per account per node | 10; filing one more drops the oldest | | Devices per account per node | 5 | | Attempts per connection | 5, audited on exhaustion | | Filing a request | requires the account to have at least one pinned identity already | @@ -1105,15 +1106,22 @@ legitimate rotation. > complete a handshake with. It is unverified at that point and is never a > substitute for the ack. -**The pre-proof window is three messages, and that is a bound to defend.** Only -the two bundle fetches and `join_request` are served before the proof, because -each is something a caller needs *in order to* prove possession of the group key: -a returning browser has to recover its identity, and a new member has no key to -prove with. Nothing else qualifies. Device messages are **authenticated-only** — -a device request is countersigned later by a device already pinned, so requiring -the caller to finish its own handshake first costs nothing and keeps the surface -at three. The bundle fetches are additionally bounded (4 per connection) and -audited, because that window is the neighbourhood **C4** and **C5b** came from. +**The pre-proof window is four messages, and that is a bound to defend.** Only +the two bundle fetches, `join_request` and `device_add_request` are served before +the proof, because each is something a caller needs *in order to* prove possession +of the group key: a returning browser has to recover its identity, a new member +has no key to prove with, and a new device of a known member has none either — the +group key is wrapped for an identity the node pinned, and a machine that keeps no +bundle there (the desktop application, the command-line client, a browser with +browser access off) holds no such identity until another device approves it. +Nothing else qualifies. Filing a device request is **inert**: nothing is admitted +until a device already pinned countersigns it, the code is bound to the new keys, +the request is bounded per connection (5) and per account (10 pending), and it is +audited. Every other device message — looking up, approving, listing, revoking, +saying hello — is sent by a device that can finish its own handshake, and is +**authenticated-only**. The bundle fetches are additionally bounded (4 per +connection) and audited, because that window is the neighbourhood **C4** and +**C5b** came from. **Pinning is defence in depth, not the primary control.** A substituted node already fails the key proof. Pinning covers the case where an attacker *holds* the @@ -2610,16 +2618,16 @@ The rules that make this safe: ## 8. Clients -### 8.1 Two clients, deliberately +### 8.1 The clients, deliberately different -| | Hub-served web SPA | Desktop client | -|---|---|---| -| Distribution | served by the hub | installed, signed release | -| Code integrity | **T3 accepted** — the hub can inject | detectable *if* reproducible builds ship | -| Key storage | IndexedDB / sessionStorage, plus a bundle on each node | OS-protected local storage; keys never bundled | -| Transport | WebRTC | WebRTC **+ QUIC** (sidecar, for hub-less `group://`) | -| Downloads | to disk where the engine allows it | native, streamed, unlimited | -| Positioning | **convenience tier** — zero install | recommended for sensitive use | +| | Hub-served web SPA | Desktop client | Command-line client (§8.6) | +|---|---|---|---| +| Distribution | served by the hub | installed, signed release | installed: its own `.deb`/`.rpm`, inside Windows Full and Store | +| Code integrity | **T3 accepted** — the hub can inject | detectable *if* reproducible builds ship | as the desktop client: the hub serves it no code | +| Key storage | IndexedDB / sessionStorage, plus a bundle on each node | OS-protected local storage; keys never bundled | OS key storage, else a 0600 file; opens a node's bundle, never writes one | +| Transport | WebRTC | WebRTC **+ QUIC** (sidecar, for hub-less `group://`) | WebRTC | +| Downloads | to disk where the engine allows it | native, streamed, unlimited | native, resumable, unlimited | +| Positioning | **convenience tier** — zero install | recommended for sensitive use | a member's files from a terminal, scripts and headless machines | **The SPA is not deprecated and stays.** It is the zero-install path, and the objective is explicit: **a native client must not prevent web use.** It must be @@ -2995,6 +3003,67 @@ MSE string) fed to a source buffer, with the node holding one slot per viewer. --- +### 8.6 The command-line client + +`meshbay-cli` (`packages/meshbay-cli`) is a member's files from a terminal: +`login`, `logout`, `groups`, `ls`, `get`, `put`, `trust`, and with no argument +an interactive two-pane view (Textual) over the same library. It is not an +operator tool — a node is administered through `meshbay-node` and its loopback +API (§6.7) — and it does not stream, play or chat: those belong to an engine +with a player and a page. English only. + +**The same protocol as the other clients, end to end.** The hub signs it in, +issues the per-node token (E10) and relays the WebRTC offer and answer; from the +DataChannel on, everything goes to the node and is sealed under the group key +(§5.1, §5.2), and transfers take leases like any client's (§5.5). Nothing +data-related passes through the hub. + +**Identity on a node, in this order:** the identity this machine kept from an +earlier run; the one waiting for a device approval; the bundle the node keeps +for browsers, opened with the passphrase (§3.7) and then kept locally; failing +all three, a new key pair filed as a device request (§3.3) — the code is +printed, an already linked device approves it, and the client joins with its +own keys. It never writes a bundle. That is the desktop client's position: an +installed program, no hub-served code (T3 does not apply), so device linking +is admitted for it as for the application. + +**What it keeps.** `state.json` (hub, account, pinned node keys) and +`secrets.enc` (AES-GCM: the refresh token, the device hub key, the node +identities) in the user's data directory (`$XDG_DATA_HOME/meshbay-cli`, +`%LOCALAPPDATA%\meshbay-cli`). The key that opens `secrets.enc` is in the OS +key storage; where there is none, in a 0600 file, which `login` says once. The +passphrase is never stored: a remembered session renews with the refresh +token, else with the device hub key, and asks for the passphrase only when both +fail. Two runs renewing at once take a file lock, so a rotated refresh token is +never spent twice. + +**Node keys are pinned at first contact.** A node presenting another key is +refused before anything is sent (exit 6) until `meshbay-cli trust reset NODE` +— the deliberate way out, for a rotation the operator announced. + +**Text from the network is data.** Every name a node or a member chose is +printed with control and bidi characters escaped, and drawn as plain text, +never markup, in the interactive view. Local names are made writable on the +machine and kept inside the destination; on Windows a download carries the +Mark of the Web. + +**Remote paths** are `group[@owner]/root/folder/file`: `@owner` only when two +groups share a name, names compared without case, `%2F` and `%25` for a `/` or +a `%` in a name, and a link copied from the application accepted as it is. + +**Concurrency.** At most six WebRTC negotiations in flight, one connection per +group; a group whose node is away is reported without stopping the others +(exit 4). + +**Exit codes** are a contract for scripts: 1 usage, 2 sign-in, 3 not found, +4 partial, 5 node unreachable or waiting for device approval, 6 node key +changed, 7 version, 8 refused before anything was sent, 130 interrupted. + +**Packaging.** Its own `.deb`/`.rpm` (`meshbay-cli`, with the packages only it +uses, on top of `meshbay-common`'s venv) and a man page generated from its +argument parser; on Windows, `meshbay-cli.exe` beside `meshbay-node.exe` in +the Full installer and the Store package, not in Light. + ## 9. Group applications ### 9.1 The plug-in architecture @@ -4487,7 +4556,9 @@ phone's photo backup (§9.12), the operator CLI and loopback control API, the desktop client through its identity and download stages, account recovery, the Windows port through packaging, and the Android client — the shell, native keys, downloads and uploads, casting, -and notifications while it is closed, fetched or pushed (§11.3). +and notifications while it is closed, fetched or pushed (§11.3) — and the +command-line client: groups, listing, downloads and uploads with resume, +device linking, and an interactive two-pane view (§8.6). The packages install: a machine has been taken from the built artefacts to a running hub and node on **Ubuntu 26.04 (`.deb`), Fedora 44 (`.rpm`) and @@ -4518,6 +4589,7 @@ process runs it — `systemctl --user` on Linux, Task Scheduler on Windows. | — | **HEIC/HEIF in Photos.** The indexer does not classify `.heic`/`.heif` as images and no browser but Safari draws them, so a phone that shoots HEIC has its backed-up photos stored and listed in Files but absent from Photos. Needs `pillow-heif` in every package and a JPEG rendition for the viewer (§9.12) | | — | **Photo backup with the application closed**, and **"remove what this phone sent"** (§9.12). A closed application has no page and so no transport; it runs at the next opening | | — | **Android: phone behaviour and release** — the back button driving the page, recovery from a network handover, keeping a download alive with the screen off, lock-screen media controls, and an update channel (§11.3) | +| — | **Command-line client: cross-group search and chat** (§8.6). Not run on Windows yet; its transfer windows are not yet measured against a node on another machine | | — | **Federation between two hubs.** The protocol is written and switched off in the code (§7.6); what is not built is one run between two machines | ### 15.3 Open, and why each is where it is @@ -4592,7 +4664,7 @@ superseded document kept under `docs/`, a note somebody holds elsewhere. | `draft-v5 §6.2`–`§6.4` — node registration, signaling, client addresses | §7.2 | | `draft-v5 §7` — cryptography | §4.1–§4.4, §4.6 | | `draft-v5 §7.1` — the keypair bundle and C4 | §3.7 | -| `draft-v5 §8.1`, `§8.2`, `desktop-client-v1.md §2` — the two clients, the shell | §8.1, §8.2 | +| `draft-v5 §8.1`, `§8.2`, `desktop-client-v1.md §2` — the clients, the shell | §8.1, §8.2 | | `draft-v5 §9`, `draft-v6 §5` — open items | §15.3 | | `draft-v5 §10` — testing posture | §12 | | `draft-v6 §2.2`, `desktop-client-v1.md §4`, `§4.5`, `§4.6` — device linking | §3.3 | diff --git a/docs/PACKAGING-GUIDE.md b/docs/PACKAGING-GUIDE.md index 5fbf7a7..e79fcd6 100644 --- a/docs/PACKAGING-GUIDE.md +++ b/docs/PACKAGING-GUIDE.md @@ -1,16 +1,19 @@ # MeshBay — Installation Guide -Four packages, all installed under `/opt/`: +Five packages, all installed under `/opt/`: | Package | What it does | |---|---| | `meshbay-common` | Shared Python venv with all dependencies | | `meshbay-hub` | Identity authority and group registry (server) | | `meshbay-node` | Local file host, streaming, chat daemon | +| `meshbay-cli` | Command-line client: your groups' files from a terminal (`man meshbay-cli`) | | `meshbay-client` | Desktop app (Electron) | Pick what you need: a desktop user installs **common + node + client**. -A server running the hub installs **common + hub**. +A server running the hub installs **common + hub**. **common + cli** gives a +machine — a server, one with no desktop — a member's access to their groups' +files from a terminal. --- @@ -23,6 +26,7 @@ sudo dpkg -i meshbay-common_0.9.0_amd64.deb sudo dpkg -i meshbay-node_0.9.0_amd64.deb # desktop machine sudo dpkg -i meshbay-hub_0.9.0_amd64.deb # server only sudo dpkg -i meshbay-client_0.9.0_amd64.deb # desktop machine +sudo dpkg -i meshbay-cli_0.9.0_amd64.deb # optional: the command-line client ``` If dpkg complains about missing dependencies: @@ -34,7 +38,7 @@ sudo apt-get install -f ### Uninstall ```bash -sudo dpkg --remove meshbay-client meshbay-hub meshbay-node meshbay-common +sudo dpkg --remove meshbay-client meshbay-cli meshbay-hub meshbay-node meshbay-common sudo rm -rf /opt/meshbay-* ``` @@ -49,12 +53,13 @@ sudo rpm -ivh meshbay-common-0.9.0-1.fc44.x86_64.rpm sudo rpm -ivh meshbay-node-0.9.0-1.fc44.noarch.rpm # desktop machine sudo rpm -ivh meshbay-hub-0.9.0-1.fc44.noarch.rpm # server only sudo rpm -ivh meshbay-client-0.9.0-1.fc44.x86_64.rpm # desktop machine +sudo rpm -ivh meshbay-cli-0.9.0-1.fc44.x86_64.rpm # optional: the command-line client ``` ### Uninstall ```bash -sudo rpm -e meshbay-client meshbay-hub meshbay-node meshbay-common +sudo rpm -e meshbay-client meshbay-cli meshbay-hub meshbay-node meshbay-common ``` --- @@ -69,7 +74,9 @@ node (with `meshbay-common` inside it). There is no Windows hub. Run the installer. It is **per-user** and lands in `%LOCALAPPDATA%\Programs\MeshBay\` without needing admin rights. The node daemon ships beside the app at `resources\node-runtime\meshbay-node.exe`; -the client finds it automatically. +the client finds it automatically. The command-line client, `meshbay-cli.exe`, +is beside it, and works in any new terminal (the light installer does not +carry it). It asks one thing — **when the node runs**: diff --git a/docs/USERGUIDE.md b/docs/USERGUIDE.md index bab2074..1c21c80 100644 --- a/docs/USERGUIDE.md +++ b/docs/USERGUIDE.md @@ -224,6 +224,43 @@ points the node at another account. A folder you pick in the folder dialog is no twice. That window belongs to the application, so nothing displayed in the page — which shows content from other people's nodes — can answer it for you. +### The command-line client + +`meshbay-cli` reaches your groups' files from a terminal — a server, a script, a +machine with no desktop. It is a package of its own on Linux (`meshbay-cli`, +with a manual page: `man meshbay-cli`), and comes with the desktop +application's full installer on Windows (not with the light one). + +```bash +meshbay-cli login -u alice # asks for the passphrase +meshbay-cli groups # your groups, their node, your role +meshbay-cli ls -l Family/photos # a group, a folder of it +meshbay-cli get -R Family/photos/2024 # download, here; -o elsewhere +meshbay-cli put -R ~/scans Family/documents +meshbay-cli logout +``` + +A group is named as the GROUP column of `meshbay-cli groups` prints it; add +`@owner` only when two of your groups share a name. Quote wildcards +(`'Family/music/*.flac'`): they are matched against the group, not against +your disk. An interrupted download resumes where it stopped the next time you +run the same `get`. Run `meshbay-cli` with nothing after it for a two-pane +view: this computer on the left, your groups on the right, F5 to copy the +selection across. + +`login` remembers the session, so later runs need no passphrase; `logout` +forgets it and everything else stored on this computer. What is kept is +encrypted, under a key held by the system's key storage — or, on a machine +that has none (a server with no desktop session), in a file only your user +can read, which `login` tells you. + +The first time it opens a group, the client may print a linking code instead +of the files: the node holds no copy of your keys (Browser access is off), so +this computer asks to be added as one of your devices. Enter the code in the +application under *Your devices on this node*; the client goes on by itself. + +Managing a node is not the client's job: that is `meshbay-node` (§6). + --- ## 4. Joining a group @@ -1198,6 +1235,8 @@ Better to know now than to go looking for it: phones take JPEG unless set otherwise. - **Hubs do not talk to each other yet.** Everyone in a group needs an account on the same hub. +- **The command-line client lists, downloads and uploads.** Searching across + groups and chat are not in it yet. - **Casting reaches Chromecast devices** from the desktop and Android applications, for films, photos and music. Support for other TV protocols is designed but not built. diff --git a/packages/meshbay-cli/LICENSE b/packages/meshbay-cli/LICENSE new file mode 100644 index 0000000..be3f7b2 --- /dev/null +++ b/packages/meshbay-cli/LICENSE @@ -0,0 +1,661 @@ + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +<https://www.gnu.org/licenses/>. diff --git a/packages/meshbay-cli/pyproject.toml b/packages/meshbay-cli/pyproject.toml new file mode 100644 index 0000000..e978d27 --- /dev/null +++ b/packages/meshbay-cli/pyproject.toml @@ -0,0 +1,28 @@ +[build-system] +requires = ["hatchling>=1.27"] # PEP 639: SPDX `license` + `license-files` +build-backend = "hatchling.build" + +[project] +name = "meshbay-cli" +version = "0.19.0" +description = "MeshBay command-line client — a member's files, from a terminal" +requires-python = ">=3.12" +license = "AGPL-3.0-or-later" +license-files = ["LICENSE"] +dependencies = [ + "meshbay-common>=0.10.0", + "aiortc>=1.9", # WebRTC: the only transport a node serves to members + "httpx>=0.28", # hub API + "argon2-cffi>=21.3", # the passphrase's half of the bundle key (A) + "keyring>=25", # OS key storage for the local store's key + "textual>=8.2", # the interactive mode +] + +[project.optional-dependencies] +dev = ["pytest>=8", "pytest-asyncio>=0.24", "pytest-timeout>=2.3", "ruff>=0.6"] + +[project.scripts] +meshbay-cli = "meshbay_cli.__main__:main" + +[tool.hatch.build.targets.wheel] +packages = ["src/meshbay_cli"] diff --git a/packages/meshbay-cli/src/meshbay_cli/__init__.py b/packages/meshbay-cli/src/meshbay_cli/__init__.py new file mode 100644 index 0000000..911e0e3 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/__init__.py @@ -0,0 +1,3 @@ +"""MeshBay command-line client.""" + +__version__ = "0.19.0" diff --git a/packages/meshbay-cli/src/meshbay_cli/__main__.py b/packages/meshbay-cli/src/meshbay_cli/__main__.py new file mode 100644 index 0000000..2b0b349 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/__main__.py @@ -0,0 +1,134 @@ +""" +meshbay-cli [global options] <action> [action options] [arguments] + +With no argument at all, the interactive mode opens. +""" + +import argparse +import asyncio +import logging +import os +import sys + +from . import __version__ +from .commands import get, groups, login, logout, ls, put, trust +from .commands.context import Context +from .errors import INTERRUPTED, USAGE, CliError + +ACTIONS = {"login": login, "logout": logout, "groups": groups, "ls": ls, "get": get, + "put": put, "trust": trust} + + +class _Parser(argparse.ArgumentParser): + """argparse exits 2 on a usage error, which here means a refused sign-in.""" + + def error(self, message: str): + self.print_usage(sys.stderr) + self.exit(USAGE, f"{self.prog}: error: {message}\n") + + +def _global_options(p: argparse.ArgumentParser, default) -> None: + p.add_argument("--hub", default=default, help="the hub (default: the one signed in to, " + "else https://meshbay.org); also MESHBAY_HUB") + p.add_argument("-u", "--user", default=default, help="account name; also MESHBAY_USER") + p.add_argument("-p", "--password", default=default, + help="the passphrase; visible to other users in the process list and " + "kept in shell history — --password-file - or MESHBAY_PASSWORD " + "are not") + p.add_argument("--password-file", metavar="FILE", default=default, + help="read the passphrase from FILE's first line (- for stdin)") + p.add_argument("--json", action="store_true", default=default or False, + help="machine-readable output") + p.add_argument("-q", "--quiet", action="store_true", default=default or False, + help="errors only") + p.add_argument("-v", "--verbose", action="count", default=default or 0, + help="-v: connection steps, -vv: protocol messages") + + +def build_parser() -> argparse.ArgumentParser: + """Global options are accepted before the action and after it.""" + p = _Parser( + prog="meshbay-cli", + description="A member's MeshBay files, from a terminal. Run with no argument " + "for the interactive mode.") + _global_options(p, None) + p.add_argument("--version", action="version", version=f"meshbay-cli {__version__}") + # SUPPRESS: an option the action's own parser did not see must not + # overwrite the value given before the action. + late = _Parser(add_help=False) + _global_options(late, argparse.SUPPRESS) + sub = p.add_subparsers(dest="action", metavar="<action>") + for module in ACTIONS.values(): + module.add_parser(sub, late) + return p + + +def _logging(verbose: int, quiet: bool) -> None: + handler = logging.StreamHandler(sys.stderr) + handler.setFormatter(logging.Formatter("meshbay-cli: %(levelname)s: %(message)s")) + root = logging.getLogger() + root.handlers[:] = [handler] + root.setLevel(logging.ERROR if quiet else logging.WARNING) + level = {0: logging.WARNING, 1: logging.INFO}.get(verbose, logging.DEBUG) + logging.getLogger("meshbay_cli").setLevel(logging.ERROR if quiet else level) + # aiortc and aioice are chatty at INFO; their story is told from -vv on. + for name in ("aiortc", "aioice", "httpx", "httpcore"): + logging.getLogger(name).setLevel(logging.DEBUG if verbose >= 3 else logging.WARNING) + + +async def _run(args: argparse.Namespace) -> int: + ctx = Context(args) + try: + if args.action is None: + from .tui.app import run_interactive + return await run_interactive(ctx) + return await ACTIONS[args.action].run(ctx) + finally: + await ctx.aclose() + + +def loop_factory(): + """aiortc's ICE needs a selector loop: Windows' default proactor never + completes a DataChannel handshake.""" + return asyncio.SelectorEventLoop if sys.platform == "win32" else None + + +def main(argv: list[str] | None = None) -> int: + argv = sys.argv[1:] if argv is None else argv + if not argv and os.environ.get("MESHBAY_CLI_SELFTEST") == "1": + from .selftest import run + return run() + parser = build_parser() + interactive = not argv + try: + args = parser.parse_args(argv) + except SystemExit as e: # --help, --version, or a usage error + return int(e.code or 0) + if args.action is None and not interactive: + parser.print_usage(sys.stderr) + return USAGE + _logging(args.verbose, args.quiet) + _utf8_output() + try: + return asyncio.run(_run(args), loop_factory=loop_factory()) + except CliError as e: + print(f"meshbay-cli: {e.message}", file=sys.stderr) + return e.code + except KeyboardInterrupt: + print("meshbay-cli: interrupted", file=sys.stderr) + return INTERRUPTED + + +def _utf8_output() -> None: + """Names are printed as they are; a Windows console's code page cannot hold most.""" + for stream in (sys.stdout, sys.stderr): + reconfigure = getattr(stream, "reconfigure", None) + if reconfigure is not None: + try: + reconfigure(encoding="utf-8", errors="replace") + except (ValueError, OSError): + pass + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/packages/meshbay-cli/src/meshbay_cli/client/account.py b/packages/meshbay-cli/src/meshbay_cli/client/account.py new file mode 100644 index 0000000..d620f6d --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/account.py @@ -0,0 +1,344 @@ +""" +A signed-in account: its hub session, its bundle key `M`, its node identities. + +Two lives. **Remembered** (after `login`): everything is in the store, and a +run needs no passphrase — an expired access token is renewed with the refresh +token, a refused one falls back to the device's hub key. **Ephemeral** (no +store, or another account named on the command line): signed in with the +passphrase for this run only, nothing written, the session ended on the hub +when the run ends. + +The refresh token rotates and a spent one presented again revokes the whole +family, so every renewal happens under the store's lock, after re-reading what +another run may have just written. +""" + +import asyncio +import contextlib +import logging +import socket +import time +from collections.abc import AsyncIterator, Callable +from dataclasses import dataclass + +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from ..errors import AUTH, CliError +from .hub import Hub, HubError +from .keys import ( + Identity, + auth_key, + b64, + bundle_master_key, + passphrase_half, + unb64, +) +from .store import KEY_FILE, Store + +log = logging.getLogger(__name__) + +# Renew this long before the access token runs out, so a request never leaves +# with a token that expires on the way. +RENEW_MARGIN = 300 + +PassphraseSource = Callable[[], str] + + +class NoPassphrase(CliError): + """Raised by a passphrase source that has none to give.""" + + def __init__(self, message: str): + super().__init__(AUTH, message) + + +def no_passphrase() -> str: + """The source for an account that must never ask: one being signed out.""" + raise NoPassphrase("No passphrase is asked for to sign out.") + + +@dataclass +class Pending: + identity: Identity + code: str + expires_at: float + + +class Account: + def __init__(self, hub: Hub, username: str, passphrase: PassphraseSource, + store: Store | None = None): + self.hub = hub + self.username = username + self._passphrase = passphrase + self.store = store # None: ephemeral + # Called with (group, DeviceLinkPending, first=) when this machine waits + # for a device approval; returns whether to keep waiting. + self.on_device_code = None + self.state: dict = {} + self.secrets: dict = {} + + @property + def remembered(self) -> bool: + return self.store is not None + + @property + def user_id(self) -> str: + return self.state["user_id"] + + @property + def master(self) -> bytes | None: + m = self.secrets.get("master") + return unb64(m) if m else None + + @property + def legacy(self) -> bytes | None: + a = self.secrets.get("legacy") + return unb64(a) if a else None + + # ── Opening ────────────────────────────────────────────────────────────── + + @classmethod + def open(cls, hub: Hub, username: str | None, passphrase: PassphraseSource, + store: Store) -> "Account": + """The remembered account when it is the one asked for, else an ephemeral one.""" + stored = store.peek() + same_hub = stored.get("hub") == hub.url + same_user = username is None or username == stored.get("username") + if stored and same_hub and same_user and store.load(): + acc = cls(hub, store.state["username"], passphrase, store) + acc.state, acc.secrets = store.state, store.secrets + return acc + if not username: + raise CliError(AUTH, "Not signed in. Run `meshbay-cli login`, or pass --user.") + return cls(hub, username, passphrase) + + @contextlib.asynccontextmanager + async def _locked(self) -> AsyncIterator[None]: + """The store's lock, taken off the loop; re-reads what is stored.""" + if self.store is None: + yield + return + cm = self.store.locked() + await asyncio.to_thread(cm.__enter__) + try: + if not self.store.load(): + raise CliError(AUTH, "Signed out by another run; run `meshbay-cli login`.") + self.state, self.secrets = self.store.state, self.store.secrets + yield + finally: + cm.__exit__(None, None, None) + + def _save(self) -> None: + """Under the lock. `sign_in` replaces both dicts, so hand over the current ones.""" + self.store.state, self.store.secrets = self.state, self.secrets + self.store.save() + + async def update(self, mutate: Callable[[dict, dict], None]) -> None: + """Change state and secrets, and write them when remembered.""" + async with self._locked(): + mutate(self.state, self.secrets) + if self.store is not None: + self._save() + + # ── Signing in ─────────────────────────────────────────────────────────── + + async def sign_in(self, *, remember: bool) -> None: + """With the passphrase: tokens, `M`, and — remembered — the device's hub key.""" + passphrase = self._passphrase() + ak = await asyncio.to_thread(auth_key, self.username, passphrase) + previous = dict(self.state) + r = await self.hub.login(self.username, ak, previous.get("known_browser")) + token = r["access_token"] + me = await self.hub.me(token) + if not r.get("bundle_pepper"): + raise CliError(AUTH, "The hub did not provide the bundle pepper.") + a = await asyncio.to_thread(passphrase_half, self.username, passphrase) + master = bundle_master_key(a, unb64(r["bundle_pepper"]), me["user_id"]) + + same_account = previous.get("user_id") == me["user_id"] + state = { + "hub": self.hub.url, + "username": self.username, + "user_id": me["user_id"], + "known_browser": r.get("known_browser") or previous.get("known_browser"), + # A node's key does not depend on who signed in. + "pins": previous.get("pins", {}) if previous.get("hub") == self.hub.url else {}, + } + secrets = { + "access_token": token, + "access_exp": time.time() + int(r.get("expires_in", 0)), + "refresh_token": r["refresh_token"], + "master": b64(master), + "legacy": b64(a), + "identities": self.secrets.get("identities", {}) if same_account else {}, + } + if remember: + device = (self.secrets.get("device_sk") if same_account else None) + sk = (Ed25519PrivateKey.from_private_bytes(unb64(device)) if device + else Ed25519PrivateKey.generate()) + reg = await self.hub.register_device( + token, b64(sk.public_key().public_bytes_raw()), + f"meshbay-cli on {socket.gethostname()}", ak) + secrets["device_sk"] = b64(sk.private_bytes_raw()) + state["device_id"] = reg["id"] + self.state, self.secrets = state, secrets + + async def remember(self, store: Store) -> str | None: + """Write this account to `store`, replacing what it held. A notice, or None.""" + self.store = store + async with self._locked_fresh(): + store.state, store.secrets = self.state, self.secrets + store.save(new=True) + return KEY_FILE if store.state.get("key_store") == KEY_FILE else None + + @contextlib.asynccontextmanager + async def _locked_fresh(self) -> AsyncIterator[None]: + cm = self.store.locked() + await asyncio.to_thread(cm.__enter__) + try: + yield + finally: + cm.__exit__(None, None, None) + + # ── The access token ───────────────────────────────────────────────────── + + def _fresh(self) -> bool: + return bool(self.secrets.get("access_token")) and \ + self.secrets.get("access_exp", 0) - RENEW_MARGIN > time.time() + + async def token(self, *, rejected: str | None = None) -> str: + """A valid access token. `rejected`: the one the hub just refused.""" + if rejected is None and self._fresh(): + return self.secrets["access_token"] + if not self.remembered: + if not self.secrets: + await self.sign_in(remember=False) + else: + await self._renew() + return self.secrets["access_token"] + async with self._locked(): + # Another run may have renewed while this one waited for the lock; + # renewing again would present a refresh token it has just spent. + current = self.secrets.get("access_token") + if not self._fresh() or current == rejected: + await self._renew() + self._save() + return self.secrets["access_token"] + + async def _renew(self) -> None: + rt = self.secrets.get("refresh_token") + if rt: + try: + r = await self.hub.refresh(rt) + self._take(r) + return + except HubError as e: + log.debug("refresh refused: %s", e) + sk = self.secrets.get("device_sk") + if sk: + try: + r = await self.hub.device_auth( + self.username, Ed25519PrivateKey.from_private_bytes(unb64(sk))) + self._take(r) + return + except CliError as e: + log.debug("device sign-in refused: %s", e.message) + try: + await self.sign_in(remember=self.remembered) + except NoPassphrase as e: + if self.remembered: + raise CliError(AUTH, "The saved session has ended; run " + "`meshbay-cli login`, or pass --password.") from e + raise + + def _take(self, r: dict) -> None: + self.secrets["access_token"] = r["access_token"] + self.secrets["access_exp"] = time.time() + int(r.get("expires_in", 0)) + self.secrets["refresh_token"] = r["refresh_token"] + + async def call(self, fn, *args): + """A hub call with the access token, renewed once on a 401.""" + token = await self.token() + try: + return await fn(token, *args) + except HubError as e: + if e.status != 401: + raise + return await fn(await self.token(rejected=token), *args) + + # ── Identities and pins ────────────────────────────────────────────────── + + def identity(self, node_pk: str) -> Identity | None: + keys = self.secrets.get("identities", {}).get(node_pk) + return Identity.from_pkcs8(keys) if keys else None + + async def keep_identity(self, node_pk: str, identity: Identity) -> None: + def put(state: dict, secrets: dict) -> None: + secrets.setdefault("identities", {})[node_pk] = identity.to_pkcs8() + await self.update(put) + + async def forget_identity(self, node_pk: str) -> None: + def drop(state: dict, secrets: dict) -> None: + secrets.get("identities", {}).pop(node_pk, None) + await self.update(drop) + + # A device request this machine filed and that waits for an approval: the + # keys it named, and the code shown for them. Kept so a later run shows the + # same code and joins with the same keys once approved. + + def pending_identity(self, node_pk: str) -> "Pending | None": + p = self.secrets.get("pending", {}).get(node_pk) + if not p or p["expires_at"] < time.time(): + return None + return Pending(Identity.from_pkcs8(p["keys"]), p["code"], p["expires_at"]) + + async def keep_pending(self, node_pk: str, identity: Identity, code: str, + expires_at: float) -> None: + def put(state: dict, secrets: dict) -> None: + secrets.setdefault("pending", {})[node_pk] = { + "keys": identity.to_pkcs8(), "code": code, "expires_at": expires_at} + await self.update(put) + + async def adopt_pending(self, node_pk: str) -> None: + """The node admitted the pending keys: they are this machine's identity there.""" + def move(state: dict, secrets: dict) -> None: + p = secrets.get("pending", {}).pop(node_pk, None) + if p: + secrets.setdefault("identities", {})[node_pk] = p["keys"] + await self.update(move) + + def pin(self, node_id: str) -> str | None: + return self.state.get("pins", {}).get(node_id) + + async def set_pin(self, node_id: str, node_pk: str) -> None: + def put(state: dict, secrets: dict) -> None: + state.setdefault("pins", {})[node_id] = node_pk + await self.update(put) + + # ── Ending ─────────────────────────────────────────────────────────────── + + async def close(self) -> None: + """An ephemeral session ends on the hub with the run.""" + if not self.remembered and self.secrets.get("refresh_token"): + with contextlib.suppress(Exception): + await self.hub.logout(self.secrets["refresh_token"]) + + async def sign_out(self) -> list[str]: + """End the session, retire the device's hub key, erase the store. Warnings.""" + warnings = [] + device_id = self.state.get("device_id") + if device_id: + try: + await self.call(self.hub.delete_device, device_id) + except (HubError, CliError) as e: + warnings.append( + "The hub did not retire this machine's device key " + f"({getattr(e, 'detail', None) or getattr(e, 'message', e)}); remove " + f"\"meshbay-cli on {socket.gethostname()}\" from your account's devices.") + rt = self.secrets.get("refresh_token") + if rt: + with contextlib.suppress(Exception): + await self.hub.logout(rt) + if self.store is not None: + async with self._locked_fresh(): + self.store.erase() + return warnings diff --git a/packages/meshbay-cli/src/meshbay_cli/client/groups.py b/packages/meshbay-cli/src/meshbay_cli/client/groups.py new file mode 100644 index 0000000..ac738f3 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/groups.py @@ -0,0 +1,68 @@ +"""Opening a group: which of the nodes serving it to dial, and in which order.""" + +import asyncio +import logging +import time + +from ..errors import NODE, CliError +from .account import Account +from .paths import handle +from .session import DeviceLinkPending, GroupSession, NodeKeyChanged, connect + +# How often a machine waiting for its approval tries the node again. Each try is +# one offer: well inside what the hub admits per account and per node. +LINK_POLL = 10 + +log = logging.getLogger(__name__) + + +async def my_groups(account: Account) -> list[dict]: + return await account.call(account.hub.my_groups) + + +async def open_group(account: Account, group: dict) -> GroupSession: + """ + A session with a node serving `group`: the pinned one first, then the others + in the hub's order. A node whose key changed stops everything — that is an + alarm, not a reason to quietly try its neighbour. + """ + nodes = await account.call(account.hub.group_nodes, group["id"]) + if not nodes: + raise CliError(NODE, f"No node serving {handle(group)} is online.") + nodes = sorted(nodes, key=lambda n: account.pin(n["node_id"]) is None) + failures = [] + for node in nodes: + try: + return await _connect_linking(account, group, node) + except (NodeKeyChanged, DeviceLinkPending): + raise + except CliError as e: + if e.code != NODE or len(nodes) == 1: + raise + log.info("node %s: %s", node["node_id"], e.message) + failures.append(f"{node['node_id']}: {e.message}") + raise CliError(NODE, f"No node serving {handle(group)} could be reached:\n " + + "\n ".join(failures)) + + +async def _connect_linking(account: Account, group: dict, node: dict) -> GroupSession: + """ + `connect`, and — when this machine is waiting to be approved as a device on + that node — the wait: the code is shown once (`account.on_device_code`), and + the node is tried again until the approval lands or the request expires. The + code names this node, so no other node is tried meanwhile. + """ + shown = False + while True: + try: + return await connect(account, group["id"], node) + except DeviceLinkPending as pending: + hook = account.on_device_code + wait = hook(group, pending, first=not shown) if hook else False + shown = True + if not wait: + raise + if pending.expires_at <= time.time(): + raise CliError(NODE, "The device request expired before it was approved; " + "run the command again for a new code.") from None + await asyncio.sleep(LINK_POLL) diff --git a/packages/meshbay-cli/src/meshbay_cli/client/hub.py b/packages/meshbay-cli/src/meshbay_cli/client/hub.py new file mode 100644 index 0000000..c8ce0c1 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/hub.py @@ -0,0 +1,160 @@ +""" +The hub's API, as a member's client uses it. + +The hub signs in, lists groups and the nodes serving them, mints the token a +node is shown, and relays the WebRTC offer. Nothing else: no listing, no file +and no byte of content passes through it (docs/MESHBAY_DESIGN.md §1.3, §8.4). +""" + +import logging +import time + +import httpx +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from .. import __version__ +from ..errors import AUTH, NODE, NOT_FOUND, VERSION, CliError +from .keys import b64 + +log = logging.getLogger(__name__) + +DEFAULT_HUB = "https://meshbay.org" + + +class HubError(Exception): + def __init__(self, status: int, detail: str): + super().__init__(f"{status}: {detail}") + self.status = status + self.detail = detail + + +def _version_tuple(v: str) -> tuple[int, ...]: + out = [] + for part in v.split("."): + digits = "".join(c for c in part if c.isdigit()) + out.append(int(digits or 0)) + return tuple(out) + + +class Hub: + def __init__(self, url: str): + self.url = url.rstrip("/") + self._http = httpx.AsyncClient( + base_url=self.url, timeout=30, + headers={"User-Agent": f"meshbay-cli/{__version__}"}) + + async def aclose(self) -> None: + await self._http.aclose() + + async def _call(self, method: str, path: str, *, token: str | None = None, + json: dict | None = None) -> dict: + headers = {"Authorization": f"Bearer {token}"} if token else {} + try: + r = await self._http.request(method, path, json=json, headers=headers) + except httpx.HTTPError as e: + raise CliError(NODE, f"Cannot reach the hub at {self.url}: {e}") from e + if r.status_code >= 400: + try: + detail = r.json().get("detail", r.text) + except Exception: + detail = r.text + raise HubError(r.status_code, str(detail)) + return r.json() if r.content else {} + + # ── Version ────────────────────────────────────────────────────────────── + + async def check_version(self) -> str | None: + """Refuse a client below the hub's minimum. Returns a notice, or None.""" + try: + info = await self._call("GET", "/v1/hub/version") + except HubError as e: + raise CliError(NODE, f"{self.url} does not answer as a MeshBay hub " + f"({e.status})") from e + client = info.get("client") or {} + mine = _version_tuple(__version__) + minimum = client.get("minimum") + if minimum and mine < _version_tuple(minimum): + raise CliError(VERSION, f"This hub needs meshbay-cli {minimum} or later; " + f"this is {__version__}.") + recommended = client.get("recommended") + if recommended and mine < _version_tuple(recommended): + return f"meshbay-cli {recommended} is available; this is {__version__}." + return None + + # ── Signing in ─────────────────────────────────────────────────────────── + + async def login(self, username: str, auth_key: str, + known_browser: str | None) -> dict: + body = {"username": username, "auth_key": auth_key} + if known_browser: + body["known_browser"] = known_browser + try: + return await self._call("POST", "/v1/users/login", json=body) + except HubError as e: + raise _sign_in_refusal(e) from e + + async def register_device(self, token: str, pk: str, label: str, auth_key: str) -> dict: + return await self._call("POST", "/v1/users/devices", token=token, json={ + "pk_auth_ed25519": pk, "label": label[:64], "auth_key": auth_key}) + + async def delete_device(self, token: str, device_id: str) -> None: + await self._call("DELETE", f"/v1/users/devices/{device_id}", token=token) + + async def device_auth(self, username: str, sk: Ed25519PrivateKey) -> dict: + ts = int(time.time()) + sig = sk.sign(f"meshbay:user_auth:{username}:{ts}".encode()) + try: + return await self._call("POST", "/v1/users/auth", json={ + "username": username, "timestamp": ts, "signature": b64(sig)}) + except HubError as e: + raise _sign_in_refusal(e) from e + + async def refresh(self, refresh_token: str) -> dict: + return await self._call("POST", "/v1/users/token/refresh", + json={"refresh_token": refresh_token}) + + async def logout(self, refresh_token: str) -> None: + await self._call("POST", "/v1/users/logout", json={"refresh_token": refresh_token}) + + async def me(self, token: str) -> dict: + return await self._call("GET", "/v1/users/me", token=token) + + # ── Groups and nodes ───────────────────────────────────────────────────── + + async def my_groups(self, token: str) -> list[dict]: + return (await self._call("GET", "/v1/groups/mine", token=token))["groups"] + + async def group_nodes(self, token: str, group_id: str) -> list[dict]: + try: + return (await self._call("GET", f"/v1/groups/{group_id}/nodes", + token=token))["nodes"] + except HubError as e: + if e.status in (403, 404): + raise CliError(NOT_FOUND, f"The hub refused the group: {e.detail}") from e + raise + + async def mnp_token(self, token: str, node_pk: str, group_id: str) -> str: + """A token for one node and one group, never the hub session (E10).""" + r = await self._call("POST", "/v1/nodes/mnp-token", token=token, + json={"node_pk": node_pk, "group_id": group_id}) + return r["mnp_token"] + + async def offer(self, token: str, node_id: str, sdp: str) -> str: + try: + r = await self._call("POST", f"/v1/nodes/{node_id}/webrtc/offer", token=token, + json={"sdp": sdp, "ice_candidates": []}) + except HubError as e: + raise CliError(NODE, f"The node did not answer the offer: {e.detail}") from e + return r["sdp"] + + +def _sign_in_refusal(e: HubError) -> CliError: + if e.status == 401: + return CliError(AUTH, "Sign-in refused: wrong username or passphrase.") + if e.status == 403 and e.detail == "email_verification_required": + return CliError(AUTH, "This account's e-mail address is not verified yet.") + if e.status == 403: + return CliError(AUTH, f"Sign-in refused: {e.detail}.") + if e.status == 429: + return CliError(AUTH, f"Too many attempts: {e.detail}") + return CliError(AUTH, f"Sign-in failed ({e.status}): {e.detail}") diff --git a/packages/meshbay-cli/src/meshbay_cli/client/keys.py b/packages/meshbay-cli/src/meshbay_cli/client/keys.py new file mode 100644 index 0000000..faca01e --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/keys.py @@ -0,0 +1,137 @@ +# SPDX-License-Identifier: LGPL-3.0-or-later +""" +The keys a member derives from a passphrase, and the identities they open. + +Byte for byte what `static/keyderive.js` does (docs/MESHBAY_DESIGN.md §3.1, +§3.7); `test_bundle_kdf_parity.py` holds this module to it. A mismatch does not +look like an error — it looks like an account nobody can open. +""" + +import base64 +import hashlib +import json +from dataclasses import dataclass + +from argon2.low_level import Type, hash_secret_raw +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey +from cryptography.hazmat.primitives.ciphers.aead import AESGCM +from cryptography.hazmat.primitives.kdf.hkdf import HKDF +from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC + +PBKDF2_ITERATIONS = 600_000 +ARGON2_MEM_KIB = 131072 # 128 MB +ARGON2_TIME = 3 +ARGON2_LANES = 1 + +BUNDLE_MAGIC = b"MBK3" +LEGACY_MAGIC = b"MBK2" # TRANSITIONAL — read, never written + + +def b64(data: bytes) -> str: + return base64.b64encode(data).decode() + + +def unb64(text: str) -> bytes: + return base64.b64decode(text) + + +def hkdf(key: bytes, info: str) -> bytes: + return HKDF(algorithm=hashes.SHA256(), length=32, salt=None, + info=info.encode()).derive(key) + + +def auth_key(username: str, passphrase: str) -> str: + """What the hub checks at sign-in. The passphrase itself never leaves here.""" + salt = hashlib.sha256(f"meshbay:auth:v1:{username}".encode()).digest() + return b64(PBKDF2HMAC(algorithm=hashes.SHA512(), length=32, salt=salt, + iterations=PBKDF2_ITERATIONS).derive(passphrase.encode())) + + +def passphrase_half(username: str, passphrase: str) -> bytes: + """`A`: Argon2id of the passphrase, salted by the account name.""" + salt = hashlib.sha256(f"meshbay:bundle:v2:{username}".encode()).digest()[:16] + return hash_secret_raw(passphrase.encode(), salt, time_cost=ARGON2_TIME, + memory_cost=ARGON2_MEM_KIB, parallelism=ARGON2_LANES, + hash_len=32, type=Type.ID) + + +def bundle_master_key(a: bytes, pepper: bytes, user_id: str) -> bytes: + """`M`, from `A` and the pepper the hub hands out at sign-in.""" + return hkdf(a + pepper, f"meshbay:bundle-master:v3|{user_id}") + + +def node_bundle_key(master: bytes, node_pk: str) -> bytes: + return hkdf(master, f"meshbay:bundle:v3|node|{node_pk}") + + +class BundleError(Exception): + pass + + +def open_bundle(bundle_b64: str, *, master: bytes | None, legacy: bytes | None, + user_id: str, node_pk: str) -> dict: + """ + The `{skEd, skX}` sealed in a keypair bundle. + + `MBK3` opens under the node's key derived from `M`, with the account and the + node as associated data. `MBK2` opens under `A` alone; it is read so a member + is not locked out of a node still holding one, and never rewritten here — + the application or a browser replaces it. Anything older is refused by name. + """ + try: + raw = unb64(bundle_b64) + except Exception as e: + raise BundleError("the node's copy of your keys is unreadable") from e + try: + if raw[:4] == BUNDLE_MAGIC: + if master is None: + raise BundleError("no bundle key in this session") + aad = f"meshbay:bundle:v3|{user_id}|{node_pk}".encode() + plain = AESGCM(node_bundle_key(master, node_pk)).decrypt(raw[5:17], raw[17:], aad) + elif raw[:4] == LEGACY_MAGIC: + if legacy is None: + raise BundleError("this node holds an older copy of your keys; sign in " + "again with your passphrase to open it") + plain = AESGCM(legacy).decrypt(raw[4:16], raw[16:], None) + else: + raise BundleError("this node holds your keys in a format no longer read; " + "ask its operator to unpin you and send a fresh code") + except BundleError: + raise + except Exception as e: + raise BundleError("the node's copy of your keys does not open with this " + "passphrase") from e + return json.loads(plain) + + +@dataclass +class Identity: + """One account's keys on one node.""" + + sk_ed: Ed25519PrivateKey + sk_x: X25519PrivateKey + + @classmethod + def from_pkcs8(cls, keys: dict) -> "Identity": + sk_ed = serialization.load_der_private_key(unb64(keys["skEd"]), password=None) + sk_x = serialization.load_der_private_key(unb64(keys["skX"]), password=None) + if not isinstance(sk_ed, Ed25519PrivateKey) or not isinstance(sk_x, X25519PrivateKey): + raise BundleError("the node's copy of your keys holds the wrong kind of key") + return cls(sk_ed, sk_x) + + def to_pkcs8(self) -> dict: + def der(k) -> str: + return b64(k.private_bytes(serialization.Encoding.DER, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption())) + return {"skEd": der(self.sk_ed), "skX": der(self.sk_x)} + + @property + def pk_ed(self) -> str: + return b64(self.sk_ed.public_key().public_bytes_raw()) + + @property + def pk_x(self) -> str: + return b64(self.sk_x.public_key().public_bytes_raw()) diff --git a/packages/meshbay-cli/src/meshbay_cli/client/names.py b/packages/meshbay-cli/src/meshbay_cli/client/names.py new file mode 100644 index 0000000..f6d7d32 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/names.py @@ -0,0 +1,69 @@ +""" +Names crossing between a group and this machine, in both directions. + +**Uploading**, the node decides: `SAFE_UPLOAD_NAME` and the Explorer-active types +in `meshbay_node/roots.py`. This is a copy of that rule, checked before anything +is sent so a folder holding one refused name is refused whole instead of +arriving with holes in it — as the Files tab does with its own copy. +`test_names.py` holds this copy to the node's answers. + +**Downloading**, a name from the index is someone else's text. It is made +writable on this platform (`meshbay_common.paths.sanitize_for_download`, the +same rule the browser applies), and the resolved path must stay under the +destination: checked after resolution, never by comparing strings. +""" + +import re +from pathlib import Path + +from meshbay_common.paths import fold, sanitize_for_download + +UPLOAD_NAME = re.compile( + r"^[^\W_]" + r"[\w .\-()\[\]'’,&+#@]{0,127}" + r"(?<![ .])$", + re.UNICODE) + +SHELL_ACTIVE_NAMES = frozenset({"desktop.ini"}) +SHELL_ACTIVE_SUFFIXES = (".lnk", ".url", ".scf", ".library-ms", ".searchconnector-ms") + + +def upload_name_problem(name: str, *, is_dir: bool = False) -> str | None: + if not UPLOAD_NAME.match(name): + return ("starts with something other than a letter or digit, ends with a space " + "or a dot, holds a character the node does not accept, or is longer " + "than 128 characters") + lower = name.lower() + if not is_dir and (lower in SHELL_ACTIVE_NAMES or lower.endswith(SHELL_ACTIVE_SUFFIXES)): + return "a file type Windows Explorer acts on by itself; no node accepts it" + return None + + +def same_name(a: str, b: str) -> bool: + """As the node and the Files tab compare names in a folder: without case.""" + return fold(a) == fold(b) + + +class UnsafeName(Exception): + pass + + +def local_path(dest: Path, parts: list[str]) -> tuple[Path, bool]: + """ + Where a remote file or folder lands under `dest`, and whether a name had to + be changed to be written here. + """ + renamed = False + out = dest + for part in parts: + safe = sanitize_for_download(part) + if safe in (".", ".."): + safe = "unnamed" + renamed |= safe != part + out = out / safe + base = dest.resolve() + try: + out.resolve().relative_to(base) + except ValueError: + raise UnsafeName(f"{'/'.join(parts)!r} would land outside {dest}") from None + return out, renamed diff --git a/packages/meshbay-cli/src/meshbay_cli/client/paths.py b/packages/meshbay-cli/src/meshbay_cli/client/paths.py new file mode 100644 index 0000000..1086085 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/paths.py @@ -0,0 +1,95 @@ +""" +Remote paths: `<group>[@<owner>]/<root>/<folder>/.../<file>`. + +The same handle the application shows under every group and puts in its links +(docs/MESHBAY_DESIGN.md §8.4, `static/group-link.js`), without the `#/`. + +A group name may contain `/` and `@` (the hub refuses only control characters +and bidirectional overrides); a username contains neither. So the owner is +whatever follows the **last** `@` of the first segment, and a `/` inside a name +is written `%2F`. Only `%2F` and `%25` are decoded in a typed path — a file +called `100%.txt` is typed as it is. A link copied from the application +(anything containing `#/`) is decoded as the application encodes it, every +segment in full. +""" + +import unicodedata +from dataclasses import dataclass +from urllib.parse import unquote + +from ..errors import NOT_FOUND, USAGE, CliError + + +@dataclass +class RemotePath: + group: str # the first segment, decoded: a name, maybe with @owner + parts: list[str] # root, folders, file — decoded + from_link: bool = False + + @property + def inner(self) -> str: + return "/".join(self.parts) + + +def _typed(segment: str) -> str: + return segment.replace("%2F", "/").replace("%2f", "/").replace("%25", "%") + + +def parse(text: str) -> RemotePath | None: + """None for "all my groups" (empty, or `/`).""" + from_link = "#/" in text + if from_link: + text = text.split("#", 1)[1] + segments = [s for s in text.split("/") if s] + if not segments: + return None + decode = unquote if from_link else _typed + decoded = [unicodedata.normalize("NFC", decode(s)) for s in segments] + if any(s in (".", "..") for s in decoded[1:]): + raise CliError(USAGE, "A remote path cannot contain . or .. segments.") + return RemotePath(decoded[0], decoded[1:], from_link) + + +def encode_name(name: str) -> str: + """A group or file name as it is typed in a path.""" + return name.replace("%", "%25").replace("/", "%2F") + + +def handle(group: dict) -> str: + """`name@owner`, ready to be typed back.""" + return f"{encode_name(group['name'])}@{group.get('owner_username') or ''}" + + +def _by_owner(named: list[dict], owner: str) -> dict | None: + exact = [g for g in named if g.get("owner_username") == owner] + if exact: + return exact[0] + loose = [g for g in named if (g.get("owner_username") or "").lower() == owner.lower()] + return loose[0] if len(loose) == 1 else None + + +def resolve_group(groups: list[dict], spec: str) -> dict: + """ + The group `spec` names among `/v1/groups/mine` rows. + + Names compare without case, as the hub keeps them unique per owner. With an + `@`, `name@owner` is tried first; when it names nothing, the whole segment is + tried as a name, since a group name may contain an `@` of its own. + """ + def named(name: str) -> list[dict]: + n = name.lower() + return [g for g in groups if (g.get("name") or "").lower() == n] + + at = spec.rfind("@") + if 0 < at < len(spec) - 1: + found = _by_owner(named(spec[:at]), spec[at + 1:]) + if found is not None: + return found + candidates = named(spec) + if len(candidates) == 1: + return candidates[0] + if len(candidates) > 1: + options = ", ".join(sorted(handle(g) for g in candidates)) + raise CliError(USAGE, f"Several of your groups are called {spec!r}; " + f"name the owner: {options}") + raise CliError(NOT_FOUND, f"You are not a member of a group called {spec!r}.") diff --git a/packages/meshbay-cli/src/meshbay_cli/client/pool.py b/packages/meshbay-cli/src/meshbay_cli/client/pool.py new file mode 100644 index 0000000..00e69fb --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/pool.py @@ -0,0 +1,48 @@ +""" +One connection per group for the whole run, and a ceiling on how many are being +negotiated at once — the application's `ConnectionPool` rules +(docs/MESHBAY_DESIGN.md §9.11). + +The ceiling is on concurrency, never a batch: a group whose node does not +answer holds one place for its own deadline and delays nobody else. Six stays +far below what the hub admits per account (32 pending offers), even with the +person's application and a browser open at the same time. +""" + +import asyncio +import contextlib + +from .account import Account +from .groups import open_group +from .session import GroupSession + +MAX_IN_FLIGHT = 6 + + +class Pool: + def __init__(self, account: Account, max_in_flight: int = MAX_IN_FLIGHT): + self.account = account + self._gate = asyncio.Semaphore(max_in_flight) + self._sessions: dict[str, asyncio.Task] = {} + + async def session(self, group: dict) -> GroupSession: + """The group's session, negotiated once and shared by every caller.""" + task = self._sessions.get(group["id"]) + if task is None: + task = asyncio.ensure_future(self._open(group)) + self._sessions[group["id"]] = task + return await asyncio.shield(task) + + async def _open(self, group: dict) -> GroupSession: + async with self._gate: + return await open_group(self.account, group) + + async def close(self) -> None: + for task in self._sessions.values(): + if task.done() and not task.cancelled() and task.exception() is None: + await task.result().close() + else: + task.cancel() + with contextlib.suppress(BaseException): + await task + self._sessions.clear() diff --git a/packages/meshbay-cli/src/meshbay_cli/client/session.py b/packages/meshbay-cli/src/meshbay_cli/client/session.py new file mode 100644 index 0000000..1ea398a --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/session.py @@ -0,0 +1,443 @@ +""" +A member's connection to one node, for one group. + +The hub relays the WebRTC offer and the answer and nothing after; everything +from the handshake on goes straight to the node, sealed under the group key +(docs/MESHBAY_DESIGN.md §5.1, §5.2). The steps, in order: + +1. a token for this node and this group only (`/v1/nodes/mnp-token`, E10); +2. the DataChannel, and the channel binding from both DTLS fingerprints; +3. the handshake: the node signs its challenge with the key it announces, and + that key must be the one pinned for this node, if any (§5.2, TOFU); +4. this account's identity on the node — kept from an earlier run, or opened + from the bundle the node keeps for browsers; +5. the group key, wrapped by the node for that identity (`join_request`); +6. the mutual proof, the ack verified and unsealed — a payload that does not + open ends the session, never reads as an empty group (§4.4); +7. `device_hello`, so the node knows which of the account's devices this is. + +Every request carries a `req_id` and its reply is matched on it (§5.3): matching +by arrival order is a guess that fails silently, and against the unrelated +request. +""" + +import asyncio +import contextlib +import itertools +import logging +import os +import struct +import time +from datetime import UTC, datetime + +import msgpack +from aiortc import RTCConfiguration, RTCIceServer, RTCPeerConnection, RTCSessionDescription +from cryptography.exceptions import InvalidSignature +from cryptography.hazmat.primitives.asymmetric.ed25519 import ( + Ed25519PrivateKey, + Ed25519PublicKey, +) +from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey +from meshbay_common import MNP_VERSION +from meshbay_common.crypto import unwrap_gek_aes +from meshbay_common.device import ( + device_code_hash, + device_hello_transcript, + device_request_transcript, +) +from meshbay_common.groupbox import PURPOSE_ACK, PURPOSE_INDEX, unseal +from meshbay_common.handshake import ( + MNP_MIN_SUPPORTED, + HandshakeError, + challenge_transcript, + check_version, + handshake_transcript, + make_proof, + verify_proof, + webrtc_binding, +) +from meshbay_common.join import join_transcript + +from ..errors import NODE, NODE_KEY, VERSION, CliError +from .account import Account +from .keys import BundleError, Identity, b64, open_bundle, unb64 + +log = logging.getLogger(__name__) + +# The browser's list (transport.js), two providers deep: one slow server must +# not add its whole timeout to every connection. +STUN_SERVERS = [ + "stun:stun.l.google.com:19302", + "stun:stun1.l.google.com:19302", + "stun:stun.cloudflare.com:3478", +] + +OPEN_TIMEOUT = 30 # ICE + DTLS + SCTP, measured at 0.6–7 s +REPLY_TIMEOUT = 60 + + +class NodeRefusal(CliError): + def __init__(self, detail: str, code: str = ""): + super().__init__(NODE, f"The node refused: {detail}") + self.detail = detail + self.refusal_code = code + + +class NodeKeyChanged(CliError): + def __init__(self, node_id: str, pinned: str, now: str): + super().__init__( + NODE_KEY, + f"The node {node_id} presents a different key than the one pinned at " + f"first contact.\n pinned: {pinned}\n now: {now}\n" + "If its operator announced a new key, run `meshbay-cli trust reset " + f"{node_id}`; otherwise do not connect.") + + +class Channel: + """MNP frames over a DataChannel: a 4-byte big-endian length, then msgpack.""" + + def __init__(self, dc): + self.dc = dc + self._buf = bytearray() + self._ids = itertools.count(1) + self._pending: dict[str, asyncio.Future] = {} + self._push_handlers: dict[str, list] = {} + self.closed = asyncio.Event() + dc.on("message", self._on_message) + dc.on("close", self._on_close) + + def on_push(self, msg_type: str, handler) -> None: + """Call `handler(msg)` for every unsolicited message of this type.""" + self._push_handlers.setdefault(msg_type, []).append(handler) + + def off_push(self, msg_type: str, handler) -> None: + with contextlib.suppress(KeyError, ValueError): + self._push_handlers[msg_type].remove(handler) + + def _on_close(self) -> None: + self.closed.set() + for fut in self._pending.values(): + if not fut.done(): + fut.set_exception(CliError(NODE, "The connection to the node closed.")) + + def _on_message(self, data) -> None: + self._buf.extend(data if isinstance(data, bytes) else data.encode()) + while len(self._buf) >= 4: + size = struct.unpack(">I", self._buf[:4])[0] + if len(self._buf) < 4 + size: + break + msg = msgpack.unpackb(bytes(self._buf[4:4 + size]), raw=False) + del self._buf[:4 + size] + self._route(msg) + + def _route(self, msg: dict) -> None: + # Type and id only: payloads carry sealed content and key material. + log.debug("← %s req_id=%s", msg.get("type"), msg.get("req_id")) + fut = self._pending.pop(str(msg.get("req_id", "")), None) + if fut is not None and not fut.done(): + fut.set_result(msg) + return + handlers = self._push_handlers.get(str(msg.get("type")), []) + if not handlers and msg.get("type") == "error": + # A refusal that names no request answers none of ours: logged, + # never handed to whichever request happens to be waiting. + log.debug("unattributed error from the node: %s", msg.get("detail")) + for handler in list(handlers): + try: + handler(msg) + except Exception: + log.exception("push handler for %s failed", msg.get("type")) + + def send(self, msg: dict) -> None: + log.debug("→ %s req_id=%s", msg.get("type"), msg.get("req_id")) + data = msgpack.packb(msg, use_bin_type=True) + self.dc.send(struct.pack(">I", len(data)) + data) + + async def request(self, msg: dict, *expect: str, timeout: float = REPLY_TIMEOUT) -> dict: + req_id = str(next(self._ids)) + fut = asyncio.get_running_loop().create_future() + self._pending[req_id] = fut + self.send({**msg, "req_id": req_id}) + try: + reply = await asyncio.wait_for(fut, timeout) + except TimeoutError: + raise CliError(NODE, f"The node did not answer {msg.get('type')} " + f"within {timeout:.0f} s.") from None + finally: + self._pending.pop(req_id, None) + if reply.get("type") == "error": + raise NodeRefusal(str(reply.get("detail") or "no reason given"), + str(reply.get("code") or "")) + if expect and reply.get("type") not in expect: + raise CliError(NODE, f"The node answered {msg.get('type')} with " + f"{reply.get('type')}.") + return reply + + +def _dtls_fingerprint(sdp: str) -> bytes: + for line in sdp.splitlines(): + if line.startswith("a=fingerprint:sha-256 "): + return bytes.fromhex(line.split(" ", 1)[1].strip().replace(":", "")) + raise CliError(NODE, "No DTLS fingerprint in the session description.") + + +class GroupSession: + """An authenticated connection to one node, for one group.""" + + def __init__(self, pc: RTCPeerConnection, ch: Channel, group_id: str, + node_id: str, node_pk: str, gek: bytes, ack: dict): + self.pc = pc + self.ch = ch + self.group_id = group_id + self.node_id = node_id + self.node_pk = node_pk + self.gek = gek + self.ack = ack + + async def index(self) -> dict: + """`{version, entries, dirs, roots}`, opened under the group key.""" + msg = await self.ch.request({"type": "index_sync", "v": MNP_VERSION}, "index_sync") + try: + return unseal(self.gek, PURPOSE_INDEX, "index_sync", self.group_id, msg) + except Exception as e: + raise CliError(NODE, "The group's index does not open under the group key; " + "the session is ended.") from e + + async def close(self) -> None: + with contextlib.suppress(Exception): + await self.pc.close() + + +async def connect(account: Account, group_id: str, node: dict) -> GroupSession: + """Open a session with `node` (a `/v1/groups/{id}/nodes` row) for `group_id`.""" + node_id, listed_pk = node["node_id"], node["pk_node"] + pinned = account.pin(node_id) + if pinned and pinned != listed_pk: + raise NodeKeyChanged(node_id, pinned, listed_pk) + + mnp_token = await account.call(account.hub.mnp_token, listed_pk, group_id) + + pc = RTCPeerConnection(RTCConfiguration( + iceServers=[RTCIceServer(urls=u) for u in STUN_SERVERS])) + try: + dc = pc.createDataChannel("mnp", ordered=True) + ch = Channel(dc) + opened = asyncio.Event() + dc.on("open", opened.set) + await pc.setLocalDescription(await pc.createOffer()) + offer_sdp = pc.localDescription.sdp + answer_sdp = await account.call(account.hub.offer, node_id, offer_sdp) + await pc.setRemoteDescription(RTCSessionDescription(sdp=answer_sdp, type="answer")) + try: + await asyncio.wait_for(opened.wait(), OPEN_TIMEOUT) + except TimeoutError: + raise CliError(NODE, f"No connection to the node within {OPEN_TIMEOUT} s " + "(it is online at the hub, but not reachable from " + "here).") from None + binding = webrtc_binding(_dtls_fingerprint(offer_sdp), _dtls_fingerprint(answer_sdp)) + return await _handshake(account, pc, ch, group_id, node_id, listed_pk, + mnp_token, binding) + except BaseException: + with contextlib.suppress(Exception): + await pc.close() + raise + + +async def _handshake(account: Account, pc, ch: Channel, group_id: str, node_id: str, + listed_pk: str, mnp_token: str, binding: bytes) -> GroupSession: + nonce_c = os.urandom(32) + challenge = await ch.request( + {"type": "handshake", "v": MNP_VERSION, "v_min": MNP_MIN_SUPPORTED, + "token": mnp_token, "group_id": group_id, "nonce": b64(nonce_c)}, + "handshake_challenge") + try: + check_version(challenge["v"], challenge.get("v_min", "")) + except HandshakeError as e: + raise CliError(VERSION, f"Protocol mismatch with the node: {e}") from e + nonce_s = unb64(challenge["nonce"]) + node_pk = challenge["node_pk"] + if node_pk != listed_pk: + raise CliError(NODE, "The node answering is not the one the hub listed.") + try: + Ed25519PublicKey.from_public_bytes(unb64(node_pk)).verify( + unb64(challenge.get("sig", "")), + challenge_transcript(group_id, nonce_c, nonce_s, binding)) + except (InvalidSignature, ValueError) as e: + raise CliError(NODE, "The node's challenge is not signed by the key it " + "announces.") from e + + identity, source = await _identity(account, ch, node_pk) + + ts = int(time.time()) + joined = await ch.request({ + "type": "join_request", "v": MNP_VERSION, "group_id": group_id, + "pk_ed25519": identity.pk_ed, "pk_x25519": identity.pk_x, "code": "", "ts": ts, + "sig": b64(identity.sk_ed.sign(join_transcript( + node_pk_b64=node_pk, group_id=group_id, user_id=account.user_id, + pk_ed25519_b64=identity.pk_ed, pk_x25519_b64=identity.pk_x, + nonce_node=nonce_s, ts=ts)))}, "join_result") + if not joined.get("ok") or not joined.get("gek"): + reason = str(joined.get("reason") or "") + if reason == "unknown_device" and source != "bundle": + await _link(account, ch, node_pk, nonce_s, identity, source) + raise NodeRefusal(JOIN_REFUSALS.get(reason, reason or "the group key was not given"), + reason) + if source == "pending": + await account.adopt_pending(node_pk) + gek = unwrap_gek_aes(joined, identity.sk_x.private_bytes_raw(), + identity.sk_x.public_key().public_bytes_raw()) + + ack = await ch.request({"type": "handshake_response", "v": MNP_VERSION, + "proof": b64(make_proof(gek, "client", group_id, nonce_c, + nonce_s, binding))}, + "handshake_ack") + if not verify_proof(gek, unb64(ack["proof"]), "node", group_id, nonce_c, nonce_s, + binding): + raise CliError(NODE, "The node does not hold the group key.") + if ack.get("node_pk") != node_pk: + raise CliError(NODE, "The node changed identity during the handshake.") + try: + Ed25519PublicKey.from_public_bytes(unb64(node_pk)).verify( + unb64(ack["sig"]), handshake_transcript("node", group_id, nonce_c, nonce_s, + binding)) + except (InvalidSignature, ValueError, KeyError) as e: + raise CliError(NODE, "The node's acknowledgement is not signed by its key.") from e + try: + config = unseal(gek, PURPOSE_ACK, "handshake_ack", group_id, ack) + except Exception as e: + raise CliError(NODE, "The node's configuration does not open under the group " + "key; the session is ended.") from e + + if account.pin(node_id) is None: + await account.set_pin(node_id, node_pk) + + await _device_hello(account, ch, identity, node_pk, group_id, nonce_s) + return GroupSession(pc, ch, group_id, node_id, node_pk, gek, config) + + +async def _identity(account: Account, ch: Channel, node_pk: str) -> tuple[Identity, str]: + """ + This account's keys on this node, and where they came from: kept from an + earlier run, waiting for a device approval, opened from the copy the node + keeps for browsers — or, failing all three, new, for device linking. + """ + kept = account.identity(node_pk) + if kept is not None: + return kept, "kept" + pending = account.pending_identity(node_pk) + if pending is not None: + return pending.identity, "pending" + found = await ch.request({"type": "keypair_bundle_fetch", "v": MNP_VERSION}, + "keypair_bundle_resp") + if not found.get("found"): + return Identity(Ed25519PrivateKey.generate(), X25519PrivateKey.generate()), "new" + try: + keys = open_bundle(found["bundle_enc"], master=account.master, + legacy=account.legacy, user_id=account.user_id, node_pk=node_pk) + identity = Identity.from_pkcs8(keys) + except BundleError as e: + raise CliError(NODE, str(e)) from e + await account.keep_identity(node_pk, identity) + return identity, "bundle" + + +# What a refused join means, for the person reading it. +JOIN_REFUSALS = { + "code_required": "this node has not admitted you yet. Accept the invitation in the " + "MeshBay application and enter its code there; the command line " + "does not join groups", + "unknown_device": "the key this node holds for you is not one of your devices there " + "any more; ask its operator, or approve this machine from a device " + "already linked", + "key_changed": "your identity on this node no longer matches the one it pinned", + "not_authorized_for_group": "this node does not count you a member of the group", +} + +# The alphabet of every MeshBay code (Crockford base32, no I L O U), so the one +# this machine shows is typed like an invitation code, and needs no normalising. +CODE_ALPHABET = "0123456789ABCDEFGHJKMNPQRSTVWXYZ" + + +class DeviceLinkPending(CliError): + """This machine asked to be a device of the account on a node; it waits for an + approval from one already linked there.""" + + def __init__(self, link_code: str, expires_at: float): + super().__init__(NODE, f"Waiting for this machine to be approved, code {link_code}.") + # Not `code`: on a CliError that is the exit status. + self.link_code = link_code + self.expires_at = expires_at + + +async def _link(account: Account, ch: Channel, node_pk: str, nonce_s: bytes, + identity: Identity, source: str) -> None: + """ + File a device request bound to a code this machine shows (design §3.3), or + show again the one already filed. Always raises `DeviceLinkPending`. + + The code is hashed together with this machine's keys, so the device that + approves cannot be handed a substituted key; the code itself never reaches + the node. + """ + if not account.remembered: + raise CliError(NODE, "This machine is not yet a device of yours on this node. " + "Run `meshbay-cli login` first: linking it only makes sense " + "on a machine that keeps the identity it links.") + if source == "pending": + pending = account.pending_identity(node_pk) + raise DeviceLinkPending(pending.code, pending.expires_at) + if source == "kept": + # Revoked or unpinned on the node since this machine kept it. + await account.forget_identity(node_pk) + identity = Identity(Ed25519PrivateKey.generate(), X25519PrivateKey.generate()) + raw = "".join(CODE_ALPHABET[b % 32] for b in os.urandom(8)) + code = f"{raw[:4]}-{raw[4:]}" + code_hash = device_code_hash(raw, identity.pk_ed, identity.pk_x) + ts = int(time.time()) + sig = identity.sk_ed.sign(device_request_transcript( + node_pk_b64=node_pk, user_id=account.user_id, pk_ed25519_b64=identity.pk_ed, + pk_x25519_b64=identity.pk_x, code_hash=code_hash, nonce_node=nonce_s, ts=ts)) + try: + ack = await ch.request({"type": "device_add_request", "v": MNP_VERSION, + "pk_ed25519": identity.pk_ed, "pk_x25519": identity.pk_x, + "code_hash": code_hash, "ts": ts, "sig": b64(sig)}, + "device_add_request_ack") + except NodeRefusal as e: + if e.detail != "Handshake required": + raise + # The node answers a device request only on a session that has proved + # the group key, which a machine with no identity there cannot do. + raise CliError(NODE, "This machine is not yet a device of yours on this node, and " + "the node does not accept a device request from a machine " + "that holds no key there. Turn on Browser access in the " + "MeshBay application's settings, then run the command " + "again.") from e + expires = _parse_time(ack.get("expires_at")) or time.time() + 3600 + await account.keep_pending(node_pk, identity, code, expires) + raise DeviceLinkPending(code, expires) + + +def _parse_time(text) -> float | None: + try: + moment = datetime.fromisoformat(str(text)) + except ValueError: + return None + if moment.tzinfo is None: + moment = moment.replace(tzinfo=UTC) + return moment.timestamp() + + +async def _device_hello(account: Account, ch: Channel, identity: Identity, node_pk: str, + group_id: str, nonce_s: bytes) -> None: + ts = int(time.time()) + sig = identity.sk_ed.sign(device_hello_transcript( + node_pk_b64=node_pk, group_id=group_id, user_id=account.user_id, + pk_ed25519_b64=identity.pk_ed, nonce_node=nonce_s, ts=ts)) + try: + await ch.request({"type": "device_hello", "v": MNP_VERSION, + "pk_ed25519": identity.pk_ed, "ts": ts, "sig": b64(sig)}, + "device_hello_ack") + except NodeRefusal as e: + # Listing does not depend on it; what is lost is the attribution of an + # upload to this device rather than to the account's oldest one. + log.warning("The node did not accept this device's hello: %s", e.detail) diff --git a/packages/meshbay-cli/src/meshbay_cli/client/store.py b/packages/meshbay-cli/src/meshbay_cli/client/store.py new file mode 100644 index 0000000..c48fe89 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/store.py @@ -0,0 +1,251 @@ +""" +What the CLI remembers between runs. + +Two files in the data directory. `state.json` holds what is not secret: the hub, +the account, the device's id, each node's pinned key. `secrets.enc` holds the +rest — tokens, the device's hub key, the bundle key `M`, the identity on each +node — under AES-256-GCM with a 32-byte **store key**. + +The store key lives in the OS key storage (Windows Credential Manager, Secret +Service on a Linux desktop). One key there and a file beside it, rather than +every secret there: Credential Manager caps a credential at 2560 bytes, and the +identities grow with the number of nodes. Where there is no key storage — an +SSH session on a server — the key is kept in a file only this user can read, +and `login` says so once (`file_key_notice`). + +Every write goes through `locked()`: the hub rotates refresh tokens and revokes +the whole session when a spent one comes back, so two runs renewing at once +must take turns. +""" + +import contextlib +import hashlib +import json +import logging +import os +import secrets +import sys +import time +from collections.abc import Iterator +from pathlib import Path + +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + +from ..errors import AUTH, CliError + +log = logging.getLogger(__name__) + +KEYRING_SERVICE = "meshbay-cli" +SECRETS_MAGIC = b"MBC1" +SECRETS_AAD = b"meshbay-cli:secrets:v1" + +KEY_OS = "os" +KEY_FILE = "file" + +FILE_KEY_NOTICE = ( + "No OS key storage here: the session is kept in a file only your user can " + "read (like an SSH key without a passphrase). `meshbay-cli logout` removes it." +) + + +def data_dir() -> Path: + """Never the node's directories (`~/.config/meshbay`, `%LOCALAPPDATA%\\meshbay`).""" + override = os.environ.get("MESHBAY_CLI_HOME") + if override: + return Path(override) + if sys.platform == "win32": + return Path(os.environ.get("LOCALAPPDATA") or Path.home() / "AppData" / "Local") \ + / "meshbay-cli" + base = os.environ.get("XDG_DATA_HOME") or Path.home() / ".local" / "share" + return Path(base) / "meshbay-cli" + + +def _write_private(path: Path, data: bytes) -> None: + """Atomically, readable by this user only.""" + tmp = path.with_name(path.name + ".tmp") + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + try: + os.write(fd, data) + os.fsync(fd) + finally: + os.close(fd) + os.replace(tmp, path) + + +class _Keyring: + """ + The OS key storage. One entry per store directory, so two stores (another + `MESHBAY_CLI_HOME`) never overwrite each other's key. + """ + + def __init__(self, root: Path): + digest = hashlib.sha256(str(root.resolve()).encode()).hexdigest()[:16] + self.entry = f"store-key:{digest}" + + def usable(self) -> bool: + try: + import keyring + from keyring.backends import fail + backend = keyring.get_keyring() + if isinstance(backend, fail.Keyring): + return False + # A Secret Service that is installed but not running answers the + # first call with an exception rather than at selection time. + keyring.get_password(KEYRING_SERVICE, self.entry) + return True + except Exception as e: + log.debug("OS key storage unavailable: %s", e) + return False + + def get(self) -> bytes | None: + import keyring + value = keyring.get_password(KEYRING_SERVICE, self.entry) + return bytes.fromhex(value) if value else None + + def set(self, key: bytes) -> None: + import keyring + keyring.set_password(KEYRING_SERVICE, self.entry, key.hex()) + + def delete(self) -> None: + import keyring + with contextlib.suppress(Exception): + keyring.delete_password(KEYRING_SERVICE, self.entry) + + +class Store: + def __init__(self, root: Path | None = None): + self.root = root or data_dir() + self._keyring = _Keyring(self.root) + self.state: dict = {} + self.secrets: dict = {} + + @property + def _state_path(self) -> Path: + return self.root / "state.json" + + @property + def _secrets_path(self) -> Path: + return self.root / "secrets.enc" + + @property + def _key_path(self) -> Path: + return self.root / "store.key" + + def exists(self) -> bool: + return self._state_path.exists() and self._secrets_path.exists() + + # ── Locking ────────────────────────────────────────────────────────────── + + @contextlib.contextmanager + def locked(self) -> Iterator[None]: + """An exclusive lock on the store, across processes. Blocks.""" + self.root.mkdir(parents=True, exist_ok=True) + fd = os.open(self.root / "store.lock", os.O_RDWR | os.O_CREAT, 0o600) + try: + if sys.platform == "win32": + import msvcrt + while True: + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + break + except OSError: + time.sleep(0.1) + else: + import fcntl + fcntl.flock(fd, fcntl.LOCK_EX) + yield + finally: + if sys.platform == "win32": + import msvcrt + with contextlib.suppress(OSError): + os.lseek(fd, 0, os.SEEK_SET) + msvcrt.locking(fd, msvcrt.LK_UNLCK, 1) + os.close(fd) + + # ── The store key ──────────────────────────────────────────────────────── + + def _load_key(self) -> bytes: + kind = self.state.get("key_store") + if kind == KEY_FILE: + try: + return bytes.fromhex(self._key_path.read_text().strip()) + except OSError as e: + raise CliError(AUTH, f"The session's key file is missing ({e.strerror}); " + "run `meshbay-cli login`.") from e + if not self._keyring.usable(): + raise CliError(AUTH, "The session was saved in the OS key storage, which this " + "session cannot reach. Run `meshbay-cli login` here, or " + "pass --password.") + key = self._keyring.get() + if key is None: + raise CliError(AUTH, "The session's key is gone from the OS key storage; " + "run `meshbay-cli login`.") + return key + + def _new_key(self) -> tuple[bytes, str]: + key = secrets.token_bytes(32) + if self._keyring.usable(): + try: + self._keyring.set(key) + return key, KEY_OS + except Exception as e: + log.debug("OS key storage refused the key: %s", e) + _write_private(self._key_path, key.hex().encode()) + return key, KEY_FILE + + # ── Reading and writing ────────────────────────────────────────────────── + + def peek(self) -> dict: + """state.json alone — the hub and the account, which are not secret.""" + try: + return json.loads(self._state_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + + def load(self) -> bool: + """Read both files. False when nothing is stored.""" + if not self.exists(): + self.state, self.secrets = {}, {} + return False + self.state = json.loads(self._state_path.read_text(encoding="utf-8")) + raw = self._secrets_path.read_bytes() + if raw[:4] != SECRETS_MAGIC: + raise CliError(AUTH, "The saved session is unreadable; run `meshbay-cli login`.") + try: + plain = AESGCM(self._load_key()).decrypt(raw[4:16], raw[16:], SECRETS_AAD) + except CliError: + raise + except Exception as e: + raise CliError(AUTH, "The saved session does not open with its key; run " + "`meshbay-cli login`.") from e + self.secrets = json.loads(plain) + return True + + def save(self, *, new: bool = False) -> None: + """Write both files. `new` makes a fresh store key (a new sign-in).""" + self.root.mkdir(parents=True, exist_ok=True) + if new or "key_store" not in self.state: + self._forget_key() + key, kind = self._new_key() + self.state["key_store"] = kind + else: + key = self._load_key() + nonce = secrets.token_bytes(12) + ct = AESGCM(key).encrypt(nonce, json.dumps(self.secrets).encode(), SECRETS_AAD) + _write_private(self._secrets_path, SECRETS_MAGIC + nonce + ct) + _write_private(self._state_path, json.dumps(self.state, indent=1).encode()) + + def _forget_key(self) -> None: + kind = self.state.get("key_store") + if kind == KEY_OS: + self._keyring.delete() + with contextlib.suppress(FileNotFoundError): + self._key_path.unlink() + + def erase(self) -> None: + """Everything, key included.""" + self._forget_key() + for p in (self._secrets_path, self._state_path): + with contextlib.suppress(FileNotFoundError): + p.unlink() + self.state, self.secrets = {}, {} diff --git a/packages/meshbay-cli/src/meshbay_cli/client/transfers.py b/packages/meshbay-cli/src/meshbay_cli/client/transfers.py new file mode 100644 index 0000000..3f23c85 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/transfers.py @@ -0,0 +1,382 @@ +""" +Downloads and uploads: one transfer lease each, then the chunks. + +**The unit is the lease** (docs/MESHBAY_DESIGN.md §5.5): the node's record that +this connection is transferring something, opened before the first chunk and +closed after the last, on every path out. A lease may be granted at once or +queued behind the member's other transfers; the grant then arrives as a push. + +Downloads are 1 MiB chunks, each encrypted under its own key derived from the +group key and the file's hash, and each independently decryptable — which is +what makes resuming a download cost nothing in the protocol. Uploads are 48 KB +chunks sealed under the group key, name and folder inside the seal; the node +keeps its own position, and a probe chunk asks for it, which is what resumes +an upload. +""" + +import asyncio +import contextlib +import json +import logging +import math +import os +import secrets +import sys +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +from meshbay_common import MNP_VERSION +from meshbay_common.groupbox import PURPOSE_UPLOAD, unseal +from meshbay_common.paths import long_path +from meshbay_common.protocol import UPLOAD_PROBE_INDEX, file_upload_wire +from meshbay_common.webcrypto import chunk_key_aes, decrypt_chunk_aes + +from ..errors import NODE, CliError +from .session import GroupSession, NodeRefusal + +log = logging.getLogger(__name__) + +DOWNLOAD_CHUNK = 1024 * 1024 # the node serves files in 1 MiB chunks +UPLOAD_CHUNK = 48 * 1024 # what fits one DataChannel message after sealing +DOWNLOAD_WINDOW = 8 # chunk requests in flight per file +UPLOAD_WINDOW = 32 # chunks sent ahead of the node's acks +CHUNK_TIMEOUT = 60 +# A queued lease waits behind the member's own transfers and other members'; +# a queue that has said nothing for this long is not moving. +QUEUE_SILENCE = 600 + +PART_SUFFIX = ".meshbay-part" + +Progress = Callable[[int], None] # bytes just completed + + +class TransferError(CliError): + """One file failed; the run goes on unless `fatal`.""" + + def __init__(self, message: str, *, fatal: bool = False, code: str = ""): + super().__init__(NODE, message) + self.fatal = fatal + self.refusal_code = code + + +# ── Leases ─────────────────────────────────────────────────────────────────── + +class Lease: + def __init__(self, session: GroupSession, kind: str): + self.session = session + self.kind = kind + self.tr = secrets.token_hex(16) + self.closed_by_node = "" + self._state: asyncio.Queue = asyncio.Queue() + + def _on_state(self, msg: dict) -> None: + if msg.get("tr") != self.tr: + return + if msg.get("state") == "closed": + self.closed_by_node = str(msg.get("reason") or "closed by the node") + self._state.put_nowait(msg) + + async def open(self, size: int, chunks: int, + on_queued: Callable[[int], None] | None = None) -> None: + ch = self.session.ch + ch.on_push("transfer_state", self._on_state) + reply = await ch.request({"type": "transfer_open", "v": MNP_VERSION, "tr": self.tr, + "kind": self.kind, "bytes": size, "chunks": chunks}, + "transfer_state") + state = reply.get("state") + while state == "queued": + if on_queued: + on_queued(int(reply.get("ahead") or 0)) + try: + reply = await asyncio.wait_for(self._state.get(), QUEUE_SILENCE) + except TimeoutError: + raise TransferError("The node's transfer queue did not move for " + f"{QUEUE_SILENCE // 60} minutes.") from None + state = reply.get("state") + if state != "granted": + raise TransferError(f"The node did not grant the transfer " + f"({reply.get('reason') or state}).") + + def check(self) -> None: + if self.closed_by_node: + raise TransferError(f"The node ended the transfer ({self.closed_by_node}).") + + async def close(self, reason: str) -> None: + ch = self.session.ch + ch.off_push("transfer_state", self._on_state) + if self.closed_by_node or ch.closed.is_set(): + return + with contextlib.suppress(CliError): + await ch.request({"type": "transfer_close", "v": MNP_VERSION, "tr": self.tr, + "reason": reason}, "transfer_state", timeout=10) + + +# ── Download ───────────────────────────────────────────────────────────────── + +@dataclass +class Resume: + """What a `.meshbay-part` beside the target already holds, if it is this file.""" + + part: Path + sidecar: Path + chunks_done: int + + @classmethod + def find(cls, target: Path, entry: dict) -> "Resume": + part = target.with_name(target.name + PART_SUFFIX) + sidecar = target.with_name(target.name + PART_SUFFIX + ".json") + done = 0 + try: + meta = json.loads(sidecar.read_text(encoding="utf-8")) + if meta.get("id") == entry["id"] and meta.get("size") == entry["size"]: + # Every chunk but the last is whole, so the complete ones are + # what the part's length says. + done = os.path.getsize(long_path(part)) // DOWNLOAD_CHUNK + except (OSError, ValueError): + done = 0 + return cls(part, sidecar, done) + + +async def download(session: GroupSession, entry: dict, target: Path | None, *, + overwrite: bool = False, progress: Progress | None = None, + on_queued: Callable[[int], None] | None = None) -> int: + """ + Fetch `entry` into `target` (None: stdout). Returns the bytes written. + + Written to `<name>.meshbay-part` and renamed at the end, so an interrupted run + never leaves a truncated file under the real name; the part is resumed from + its last whole chunk when the sidecar says it is this file. + """ + size = int(entry["size"]) + chunks = max(1, math.ceil(size / DOWNLOAD_CHUNK)) + file_hash = bytes.fromhex(entry["id"]) + + resume = None + if target is not None: + resume = Resume.find(target, entry) + start = min(resume.chunks_done, chunks) + await asyncio.to_thread(_prepare_part, resume, entry, start) + out = open(long_path(resume.part), "r+b") + out.seek(start * DOWNLOAD_CHUNK) + else: + start = 0 + out = sys.stdout.buffer + if progress and start: + progress(min(size, start * DOWNLOAD_CHUNK)) + + lease = Lease(session, "download") + reason = "error" + try: + await lease.open(size, chunks, on_queued) + written = await _fetch_chunks(session, lease, entry["id"], file_hash, start, chunks, + out, progress, streaming=target is None) + if target is not None: + await asyncio.to_thread(out.flush) + await asyncio.to_thread(os.fsync, out.fileno()) + total = start * DOWNLOAD_CHUNK + written + if total != size: + raise TransferError(f"The node sent {total} bytes of a {size}-byte file.") + reason = "done" + except asyncio.CancelledError: + reason = "cancelled" + raise + finally: + if target is not None: + out.close() + await lease.close(reason) + + if target is not None: + await asyncio.to_thread(_publish, resume, target, overwrite) + return size if target is not None else written + + +def _prepare_part(resume: Resume, entry: dict, start: int) -> None: + part = long_path(resume.part) + if start == 0: + with open(part, "wb"): + pass + else: + os.truncate(part, start * DOWNLOAD_CHUNK) + resume.sidecar.write_text(json.dumps({"id": entry["id"], "size": entry["size"]}), + encoding="utf-8") + + +def _publish(resume: Resume, target: Path, overwrite: bool) -> None: + part, dest = long_path(resume.part), long_path(target) + if overwrite: + os.replace(part, dest) + else: + # A file that appeared meanwhile is not replaced: a hard link refuses + # an existing target, where a rename would silently take its place. + try: + os.link(part, dest) + os.unlink(part) + except FileExistsError: + raise TransferError(f"{target} appeared while downloading; the download " + f"is kept as {resume.part.name}.") from None + except OSError: + if os.path.exists(dest): + raise TransferError(f"{target} already exists.") from None + os.replace(part, dest) + with contextlib.suppress(OSError): + resume.sidecar.unlink() + if sys.platform == "win32": + mark_of_the_web(target) + + +async def _fetch_chunks(session: GroupSession, lease: Lease, file_id: str, file_hash: bytes, + start: int, chunks: int, out, progress: Progress | None, + *, streaming: bool) -> int: + """Chunk requests `DOWNLOAD_WINDOW` ahead, written in order as they land.""" + ch, gek = session.ch, session.gek + inflight: dict[int, asyncio.Task] = {} + nxt, written = start, 0 + + def ask(i: int) -> asyncio.Task: + return asyncio.ensure_future(ch.request( + {"type": "file_req", "v": MNP_VERSION, "file_id": file_id, "chunk_index": i, + "tr": lease.tr}, "file_chunk", timeout=CHUNK_TIMEOUT)) + + try: + for i in range(start, chunks): + while nxt < chunks and nxt - i < DOWNLOAD_WINDOW: + inflight[nxt] = ask(nxt) + nxt += 1 + try: + msg = await inflight.pop(i) + except NodeRefusal as e: + raise TransferError(f"chunk {i}: {e.detail}", code=e.refusal_code) from e + except CliError as e: + raise TransferError(f"chunk {i}: {e.message}") from e + lease.check() + data = decrypt_chunk_aes(chunk_key_aes(gek, file_hash, i), msg["nonce"], msg["ct"]) + if streaming: + out.write(data) + out.flush() + else: + await asyncio.to_thread(out.write, data) + written += len(data) + if progress: + progress(len(data)) + finally: + for task in inflight.values(): + task.cancel() + for task in inflight.values(): + with contextlib.suppress(BaseException): + await task + return written + + +def mark_of_the_web(path: Path) -> None: + """ZoneId 3, as a browser marks a download, so SmartScreen and Protected View + apply when it is opened. FAT and exFAT have no alternate streams: nothing to do.""" + with contextlib.suppress(OSError): + with open(long_path(path) + ":Zone.Identifier", "w", encoding="ascii") as f: + f.write("[ZoneTransfer]\r\nZoneId=3\r\n") + + +# ── Upload ─────────────────────────────────────────────────────────────────── + +async def create_dir(session: GroupSession, parent: str, name: str) -> str: + """A folder `name` inside `parent` (root/folders). Returns its path.""" + try: + ack = await session.ch.request({"type": "dir_create", "v": MNP_VERSION, + "dir": parent, "name": name}, "dir_create_ack") + except NodeRefusal as e: + raise TransferError(f"{parent}/{name}: {e.detail}", code=e.refusal_code, + fatal=e.refusal_code in FATAL_UPLOAD_CODES) from e + return str(ack.get("dir") or f"{parent}/{name}") + + +# Refusals that will not get better for the next file: the run stops. +FATAL_UPLOAD_CODES = frozenset({"disk_full", "root_unavailable", "root_read_only", + "no_writable_root", "no_roots", "no_group_key"}) + + +async def upload(session: GroupSession, read: Callable[[int, int], bytes], size: int, + filename: str, folder: str, *, progress: Progress | None = None, + on_queued: Callable[[int], None] | None = None) -> str: + """ + Send `size` bytes, read by `read(offset, length)`, as `filename` in `folder`. + Returns the name the node stored it under — it never replaces a file, and + takes a free name when this one is taken by the time the upload ends. + """ + gek, gid, ch = session.gek, session.group_id, session.ch + chunks = max(1, math.ceil(size / UPLOAD_CHUNK)) + upload_id = secrets.token_hex(16) + + def chunk(i: int, data: bytes) -> dict: + return {**file_upload_wire(gek, gid, upload_id=upload_id, chunk_index=i, + total_chunks=chunks, filename=filename, data=data, + dir=folder), "tr": lease.tr} + + def refused(e: NodeRefusal) -> TransferError: + return TransferError(f"{filename}: {e.detail}", code=e.refusal_code, + fatal=e.refusal_code in FATAL_UPLOAD_CODES) + + lease = Lease(session, "upload") + reason = "error" + try: + await lease.open(size, chunks, on_queued) + # Where am I? Only the node knows how much of this file it already holds. + try: + probe = await ch.request(chunk(UPLOAD_PROBE_INDEX, b""), "file_upload_ack", + timeout=10) + start = int(_open_ack(gek, gid, probe).get("resume_from") or 0) + except NodeRefusal as e: + raise refused(e) from e + except CliError: + start = 0 + if not 0 < start < chunks: + start = 0 + if progress and start: + progress(min(size, start * UPLOAD_CHUNK)) + + pending: list[tuple[int, asyncio.Task]] = [] + last: dict = {} + try: + for i in range(start, chunks): + while len(pending) >= UPLOAD_WINDOW: + last = await _ack(pending.pop(0), refused, progress, size) + lease.check() + data = await asyncio.to_thread(read, i * UPLOAD_CHUNK, UPLOAD_CHUNK) + pending.append((i, asyncio.ensure_future( + ch.request(chunk(i, data), "file_upload_ack", timeout=CHUNK_TIMEOUT)))) + while pending: + last = await _ack(pending.pop(0), refused, progress, size) + finally: + for _, task in pending: + task.cancel() + for _, task in pending: + with contextlib.suppress(BaseException): + await task + reason = "done" + except asyncio.CancelledError: + reason = "cancelled" + raise + finally: + await lease.close(reason) + return str(_open_ack(gek, gid, last).get("stored_as") or filename) + + +async def _ack(item: tuple[int, asyncio.Task], refused, progress: Progress | None, + size: int) -> dict: + i, task = item + try: + msg = await task + except NodeRefusal as e: + raise refused(e) from e + except CliError as e: + raise TransferError(f"chunk {i}: {e.message}") from e + if progress: + progress(min(UPLOAD_CHUNK, size - i * UPLOAD_CHUNK)) + return msg + + +def _open_ack(gek: bytes, group_id: str, msg: dict) -> dict: + try: + return unseal(gek, PURPOSE_UPLOAD, "file_upload_ack", group_id, msg) + except Exception as e: + raise TransferError("The node's acknowledgement does not open under the group " + "key.", fatal=True) from e diff --git a/packages/meshbay-cli/src/meshbay_cli/client/tree.py b/packages/meshbay-cli/src/meshbay_cli/client/tree.py new file mode 100644 index 0000000..6e537ef --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/tree.py @@ -0,0 +1,119 @@ +""" +A group's index as the Files tab draws it: roots, then folders, then files. + +Index entries carry `path` = the root's name and the folders, and `name`; the +index also lists every directory (`dirs`, roots included), because an empty +folder is not an entry and would otherwise not exist; and each root's +`available` and `writable` (docs/MESHBAY_DESIGN.md §6.2, §9.5). +""" + +from dataclasses import dataclass, field +from fnmatch import fnmatchcase + +from ..errors import NOT_FOUND, CliError + +GLOB_CHARS = frozenset("*?[") + + +def is_pattern(segment: str) -> bool: + return any(c in GLOB_CHARS for c in segment) + + +@dataclass +class Item: + name: str + path: str # inside the group: root/folders/name + is_dir: bool + entry: dict | None = None # the index entry, for a file + root: dict | None = None # the root's state, for a root + available: bool = True + + @property + def size(self) -> int: + return int(self.entry.get("size", 0)) if self.entry else 0 + + @property + def added_at(self) -> int: + return int(self.entry.get("added_at", 0)) if self.entry else 0 + + +@dataclass +class Tree: + roots: dict[str, dict] + dirs: set[str] + files: dict[str, list[dict]] = field(default_factory=dict) # parent path → entries + + @classmethod + def from_index(cls, index: dict) -> "Tree": + roots = {r["name"]: r for r in index.get("roots", [])} + dirs = set(index.get("dirs", [])) + files: dict[str, list[dict]] = {} + for e in index.get("entries", []): + files.setdefault(e["path"], []).append(e) + # A file's folders exist even if the node's directory walk missed one. + parts = e["path"].split("/") + for i in range(1, len(parts) + 1): + dirs.add("/".join(parts[:i])) + dirs |= set(roots) + return cls(roots, dirs, files) + + def root_available(self, path: str) -> bool: + root = self.roots.get(path.split("/", 1)[0]) + return bool(root.get("available", True)) if root else True + + def is_dir(self, path: str) -> bool: + return path in self.dirs + + def file(self, path: str) -> dict | None: + parent, _, name = path.rpartition("/") + for e in self.files.get(parent, []): + if e["name"] == name: + return e + return None + + def children(self, path: str, *, include_unavailable: bool = False) -> list[Item]: + """What a folder holds — at the top, the roots. Unavailable roots' content + is left out unless asked for, as the Files tab does.""" + if not path: + return [Item(name, name, True, root=r, available=bool(r.get("available", True))) + for name, r in self.roots.items()] + if not self.is_dir(path): + raise CliError(NOT_FOUND, f"No folder {path!r} in this group.") + available = self.root_available(path) + if not available and not include_unavailable: + return [] + prefix = path + "/" + items = [Item(d[len(prefix):], d, True, available=available) + for d in self.dirs + if d.startswith(prefix) and "/" not in d[len(prefix):]] + items += [Item(e["name"], f"{path}/{e['name']}", False, entry=e, available=available) + for e in self.files.get(path, [])] + return items + + def walk(self, path: str, *, include_unavailable: bool = False): + """(folder, children) for `path` and every folder below it, depth first.""" + kids = self.children(path, include_unavailable=include_unavailable) + yield path, kids + for item in sorted((k for k in kids if k.is_dir), key=lambda k: k.name.lower()): + yield from self.walk(item.path, include_unavailable=include_unavailable) + + def expand(self, parts: list[str]) -> list[str]: + """ + The paths `parts` names, wildcards (`*`, `?`, `[...]`, one segment each) + matched against the index rather than by a shell, so a pattern means the + same in bash and in PowerShell. Matching is case-sensitive, as paths are. + """ + paths = [""] + for seg in parts: + found = [] + for base in paths: + if base and not self.is_dir(base): + continue + if is_pattern(seg): + found += [i.path for i in self.children(base) if fnmatchcase(i.name, seg)] + else: + p = f"{base}/{seg}" if base else seg + if self.is_dir(p) or self.file(p) is not None: + found.append(p) + paths = found + return sorted(set(paths)) diff --git a/packages/meshbay-cli/src/meshbay_cli/client/upload_plan.py b/packages/meshbay-cli/src/meshbay_cli/client/upload_plan.py new file mode 100644 index 0000000..070e686 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/client/upload_plan.py @@ -0,0 +1,92 @@ +""" +What an upload will do, decided before anything is sent — shared by `put` and +the interactive mode, so both refuse the same names and the same collisions. + +A name already in the destination folder, compared without case as the node and +the Files tab compare it, or a name the node does not accept (`names.py`), is +found here; what to do about it (refuse the run, skip it, ask) is the caller's. +""" + +import os +import stat +from dataclasses import dataclass +from pathlib import Path + +from meshbay_common.paths import fold + +from ..errors import USAGE, CliError +from ..output import safe +from .names import same_name, upload_name_problem +from .tree import Tree + + +@dataclass +class UploadFile: + local: Path + rel: list[str] # remote parts below the destination folder + size: int + + +def plan_upload(sources: list[str], recursive: bool): + """Folders to create (parents first) and files to send, as remote parts.""" + dirs: list[list[str]] = [] + files: list[UploadFile] = [] + problems: list[str] = [] + seen: dict[str, str] = {} + for src in sources: + path = Path(src).expanduser() + try: + st = path.stat() + except OSError as e: + problems.append(f"{src}: {e.strerror}") + continue + top = path.resolve().name if src.rstrip("/\\") in (".", "..") else path.name + key = fold(top) + if key in seen: + problems.append(f"{src} and {seen[key]} would have the same name there") + continue + seen[key] = src + if stat.S_ISREG(st.st_mode): + files.append(UploadFile(path, [top], st.st_size)) + elif stat.S_ISDIR(st.st_mode): + if not recursive: + raise CliError(USAGE, f"{safe(src)} is a folder; use -R.") + for here, subdirs, names in os.walk(path): + rel = [top] + list(Path(here).relative_to(path).parts) + dirs.append(rel) + for d in list(subdirs): + if (Path(here) / d).is_symlink(): + subdirs.remove(d) + problems.append(f"{Path(here) / d} is a link to a folder; not " + "followed") + for n in names: + p = Path(here) / n + try: + st_n = p.stat() + except OSError as e: + problems.append(f"{p}: {e.strerror}") + continue + if stat.S_ISREG(st_n.st_mode): + files.append(UploadFile(p, rel + [n], st_n.st_size)) + subdirs.sort() + else: + problems.append(f"{src} is neither a file nor a folder") + return dirs, files, problems + + +def taken_names(tree: Tree, folder: str, names: list[str]) -> set[str]: + """The names among `names` already in `folder`, compared without case.""" + there = [i.name for i in tree.children(folder, include_unavailable=True)] + return {n for n in names if any(same_name(n, t) for t in there)} + + +def refused_names(file_names: list[str], dirs: list[list[str]]) -> list[str]: + """Names the node would refuse — the client's copy of its rule.""" + out = [] + for name in file_names: + if (why := upload_name_problem(name)): + out.append(f"{name}: {why}") + for d in dirs: + if (why := upload_name_problem(d[-1], is_dir=True)): + out.append(f"{'/'.join(d)}/: {why}") + return out diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/__init__.py b/packages/meshbay-cli/src/meshbay_cli/commands/__init__.py new file mode 100644 index 0000000..313c146 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/__init__.py @@ -0,0 +1,4 @@ +""" +One module per action. Each exposes `add_parser(subparsers, global_options)` and +`async run(ctx)`. +""" diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/context.py b/packages/meshbay-cli/src/meshbay_cli/commands/context.py new file mode 100644 index 0000000..c11533b --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/context.py @@ -0,0 +1,82 @@ +"""What every action needs: the hub, the account name, the passphrase, the store.""" + +import argparse +import getpass +import os +import sys +import time +from pathlib import Path + +from ..client.account import Account, NoPassphrase +from ..client.hub import DEFAULT_HUB, Hub +from ..client.paths import handle +from ..client.store import Store +from ..errors import USAGE, CliError +from ..output import safe + + +class Context: + def __init__(self, args: argparse.Namespace): + self.args = args + self.store = Store() + stored = self.store.peek() + explicit_hub = args.hub or os.environ.get("MESHBAY_HUB") + self.hub_url = (explicit_hub or stored.get("hub") or DEFAULT_HUB).rstrip("/") + user = args.user or os.environ.get("MESHBAY_USER") + self.username = user.strip() if user else None + self.stored_username = stored.get("username") if stored.get("hub") == self.hub_url \ + else None + self.hub = Hub(self.hub_url) + self._passphrase: str | None = None + + def passphrase(self) -> str: + """--password, --password-file, MESHBAY_PASSWORD, then a prompt on a terminal.""" + if self._passphrase is not None: + return self._passphrase + a = self.args + value = a.password + if value is None and a.password_file: + try: + text = sys.stdin.read() if a.password_file == "-" \ + else Path(a.password_file).read_text(encoding="utf-8") + except OSError as e: + raise CliError(USAGE, f"Cannot read {a.password_file}: {e.strerror}") from e + value = text.splitlines()[0] if text else "" + if value is None: + value = os.environ.get("MESHBAY_PASSWORD") + if value is None: + if not sys.stdin.isatty(): + raise NoPassphrase("A passphrase is needed and there is no terminal to ask " + "on; pass --password-file or set MESHBAY_PASSWORD.") + value = getpass.getpass(f"Passphrase for {self.username or self.stored_username}" + f" on {self.hub_url}: ") + if not value: + raise NoPassphrase("No passphrase given.") + self._passphrase = value + return value + + def account(self) -> Account: + account = Account.open(self.hub, self.username, self.passphrase, self.store) + account.on_device_code = self._device_code + return account + + def _device_code(self, group: dict, pending, *, first: bool) -> bool: + """Show the code once. Wait only where somebody is there to approve it.""" + waiting = sys.stderr.isatty() and not self.args.quiet + if first: + minutes = max(1, int((pending.expires_at - time.time()) // 60)) + print(f"This machine is not yet a device of yours on the node hosting " + f"{safe(handle(group))}.\n" + "Approve it from the MeshBay application or a browser where you use " + "this group:\n" + " the group's Settings → \"Your devices on this node\" → enter " + f"{pending.link_code} → Approve\n" + + (f"Waiting for the approval (the code expires in {minutes} min; " + "Ctrl-C to stop)..." if waiting else + f"Run the command again once it is approved (the code expires in " + f"{minutes} min)."), + file=sys.stderr, flush=True) + return waiting + + async def aclose(self) -> None: + await self.hub.aclose() diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/get.py b/packages/meshbay-cli/src/meshbay_cli/commands/get.py new file mode 100644 index 0000000..eec73c4 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/get.py @@ -0,0 +1,235 @@ +""" +`get`: download files and folders from groups. + +The whole run is planned before the first byte: paths and patterns resolved +against each group's index, local names made writable here, existing files +found. A path that names nothing stops the run there (exit 3); a file that +cannot be written is reported and the others go on (exit 4). +""" + +import asyncio +import os +from dataclasses import dataclass +from pathlib import Path + +from meshbay_common.paths import fold + +from ..client.groups import my_groups +from ..client.names import UnsafeName, local_path +from ..client.paths import handle, parse, resolve_group +from ..client.pool import Pool +from ..client.session import GroupSession +from ..client.transfers import PART_SUFFIX, TransferError, download +from ..client.tree import Tree +from ..errors import NODE, NOT_FOUND, PARTIAL, USAGE, CliError +from ..output import Progress, err, human_size, safe, sentence +from .context import Context + +HELP = "download files, or folders with -R" +PER_GROUP = 2 # the node's default lease cap per member and group (§5.5) + + +def add_parser(sub, parent) -> None: + p = sub.add_parser("get", parents=[parent], help=HELP, description=sentence(HELP) + + ". Wildcards (* ? [...]) are matched against the group, " + "so quote them.") + p.add_argument("paths", nargs="+", metavar="PATH", help="group[@owner]/root/...") + p.add_argument("-R", dest="recursive", action="store_true", help="folders, recursively") + p.add_argument("-o", dest="dest", metavar="DEST", + help="a folder (created when it ends with /), a file name for a " + "single file, or - for stdout; default: the current folder") + exist = p.add_mutually_exclusive_group() + exist.add_argument("-f", dest="force", action="store_true", + help="overwrite existing files") + exist.add_argument("-n", dest="skip", action="store_true", + help="skip existing files silently") + + +@dataclass +class Job: + group: dict + session: GroupSession + entry: dict + remote: str # as shown: handle/root/folders/name + target: Path | None # None: stdout + + +async def run(ctx: Context) -> int: + await ctx.hub.check_version() + account = ctx.account() + pool = Pool(account) + try: + return await _run(ctx, account, pool) + finally: + await pool.close() + await account.close() + + +async def _run(ctx: Context, account, pool: Pool) -> int: + a = ctx.args + groups = await my_groups(account) + trees: dict[str, Tree] = {} + + async def tree_of(group: dict) -> tuple[GroupSession, Tree]: + session = await pool.session(group) + if group["id"] not in trees: + trees[group["id"]] = Tree.from_index(await session.index()) + return session, trees[group["id"]] + + # ── What the paths name ────────────────────────────────────────────────── + named: list[tuple[dict, GroupSession, Tree, str]] = [] # group, session, tree, path + missing: list[str] = [] + unreachable: list[str] = [] + for text in a.paths: + remote = parse(text) + if remote is None: + raise CliError(USAGE, "Name a group, and what to fetch in it.") + group = resolve_group(groups, remote.group) + try: + session, tree = await tree_of(group) + except CliError as e: + # One group whose node is away does not cost the others their files. + if e.code != NODE: + raise + unreachable.append(f"{text}: {e.message}") + continue + found = tree.expand(remote.parts) + if not found and (remote.inner and (tree.is_dir(remote.inner) + or tree.file(remote.inner))): + found = [remote.inner] # a literal name with brackets in it + if not found: + missing.append(text) + named += [(group, session, tree, p) for p in found] + if missing: + raise CliError(NOT_FOUND, "Nothing in your groups matches: " + + ", ".join(safe(m) for m in missing)) + + to_stdout = a.dest == "-" + files = [(g, s, t, p) for g, s, t, p in named if p and not t.is_dir(p)] + folders = [(g, s, t, p) for g, s, t, p in named if not p or t.is_dir(p)] + if unreachable and not named: + raise CliError(NODE, "\n".join(safe(u) for u in unreachable)) + if folders and not a.recursive: + raise CliError(USAGE, "These are folders; use -R: " + + ", ".join(safe(_shown(g, p)) for g, _, _, p in folders)) + if to_stdout and (folders or len(files) != 1): + raise CliError(USAGE, "-o - writes exactly one file to stdout.") + + # ── Where it goes ──────────────────────────────────────────────────────── + single_file = not folders and len(files) == 1 and not to_stdout + dest, file_name = _destination(a.dest, single_file) + + problems: list[str] = list(unreachable) + jobs: list[Job] = [] + dirs: list[Path] = [] + planned: set[str] = set() + renamed: list[tuple[str, Path]] = [] + + def plan_file(group, session, tree, path, parts): + entry = tree.file(path) + shown = _shown(group, path) + if not tree.root_available(path): + problems.append(f"{shown}: its folder on the node is unavailable") + return + if to_stdout: + jobs.append(Job(group, session, entry, shown, None)) + return + try: + target, changed = local_path(dest, parts) + except UnsafeName as e: + problems.append(f"{shown}: {e}") + return + key = fold(os.path.normcase(str(target))) + if key in planned: + problems.append(f"{shown}: another file of this run has the same local name " + f"{target}") + return + planned.add(key) + if changed: + renamed.append((shown, target)) + if target.exists() and not target.with_name(target.name + PART_SUFFIX).exists(): + if a.skip: + return + if not a.force: + problems.append(f"{target} already exists (-f to overwrite, -n to skip)") + return + jobs.append(Job(group, session, entry, shown, target)) + + for group, session, tree, path in files: + name = path.rsplit("/", 1)[-1] + plan_file(group, session, tree, path, [file_name or name]) + for group, session, tree, path in folders: + top = path.rsplit("/", 1)[-1] if path else group["name"] + for folder, items in tree.walk(path): + rel = [top] + (folder[len(path):].strip("/").split("/") + if folder != path else []) + rel = [r for r in rel if r] + try: + dirs.append(local_path(dest, rel)[0]) + except UnsafeName as e: + problems.append(f"{_shown(group, folder)}: {e}") + continue + for item in items: + if not item.is_dir: + plan_file(group, session, tree, item.path, rel + [item.name]) + + # ── Fetch ──────────────────────────────────────────────────────────────── + progress = Progress("↓", quiet=a.quiet or to_stdout and not a.json, as_json=a.json) + for shown, target in renamed: + progress.note(f"meshbay-cli: {safe(shown)} is saved as {target.name} " + "(its name cannot be written here as it is)") + for problem in problems: + progress.note(f"meshbay-cli: {safe(problem)}") + for d in dirs: + d.mkdir(parents=True, exist_ok=True) + progress.plan(len(jobs), sum(int(j.entry["size"]) for j in jobs)) + + failed = len(problems) + gates: dict[str, asyncio.Semaphore] = {} + + async def one(job: Job) -> None: + nonlocal failed + gate = gates.setdefault(job.group["id"], asyncio.Semaphore(PER_GROUP)) + async with gate: + progress.start(safe(job.entry["name"])) + try: + await download(job.session, job.entry, job.target, overwrite=a.force, + progress=progress.advance, + on_queued=lambda n: progress.queued( + safe(job.entry["name"]), n)) + progress.finish(safe(job.entry["name"]), path=job.remote, + to=str(job.target or "-")) + except TransferError as e: + failed += 1 + progress.note(f"meshbay-cli: {safe(job.remote)}: {e.message}") + + await asyncio.gather(*(one(j) for j in jobs)) + progress.clear() + + if not a.quiet and not to_stdout and not a.json: + done = len(jobs) - (failed - len(problems)) + err(f"Downloaded {done} file{'s' if done != 1 else ''} " + f"({human_size(progress.bytes_done)})" + + (f"; {failed} failed." if failed else ".")) + return PARTIAL if failed else 0 + + +def _shown(group: dict, path: str) -> str: + return f"{handle(group)}/{path}".rstrip("/") + + +def _destination(dest: str | None, single_file: bool) -> tuple[Path, str | None]: + """The folder files land in, and a file name when -o names one.""" + if dest is None or dest == "-": + return Path.cwd(), None + path = Path(dest).expanduser() + if dest.endswith(("/", os.sep)): + path.mkdir(parents=True, exist_ok=True) + return path, None + if path.is_dir(): + return path, None + if single_file: + if not path.parent.is_dir(): + raise CliError(USAGE, f"{path.parent} is not a folder.") + return path.parent, path.name + raise CliError(USAGE, f"{dest} is not a folder (end it with / to create it).") diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/groups.py b/packages/meshbay-cli/src/meshbay_cli/commands/groups.py new file mode 100644 index 0000000..7595e2b --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/groups.py @@ -0,0 +1,69 @@ +""" +`groups`: the groups this account belongs to, one line each. + +The hub answers alone (`/v1/groups/mine`); no node is dialled. Whether a node +is up is what the hub sees of its socket, not a promise this machine can reach +it. "owner" is the account that created the group on the hub — a node's +operator is another notion. +""" + +import json +import sys +from datetime import datetime + +from ..client.groups import my_groups +from ..client.paths import handle +from ..output import safe, sentence +from .context import Context + +HELP = "list the groups you belong to" + + +def add_parser(sub, parent) -> None: + sub.add_parser("groups", parents=[parent], help=HELP, description=sentence(HELP) + + ": the node's state, your role, the visibility, the last activity. " + "The GROUP column is the path `ls`, `get` and `put` take.") + + +def node_state(g: dict) -> str: + if not g.get("hosted", True): + return "none" + return "up" if g.get("node_online") else "down" + + +def _when(iso: str | None) -> str: + if not iso: + return "" + try: + return datetime.fromisoformat(iso).astimezone().strftime("%Y-%m-%d %H:%M") + except ValueError: + return "" + + +async def run(ctx: Context) -> int: + await ctx.hub.check_version() + account = ctx.account() + try: + groups = await my_groups(account) + finally: + await account.close() + groups.sort(key=lambda g: (g["name"].lower(), g.get("owner_username") or "")) + if ctx.args.json: + json.dump([{"group": g["name"], "owner": g.get("owner_username"), "id": g["id"], + "path": handle(g), "node": node_state(g), + "role": "owner" if g.get("is_admin") else "member", + "visibility": g.get("visibility"), "join_policy": g.get("join_policy"), + "muted": bool(g.get("muted")), "description": g.get("description", ""), + "last_activity_at": g.get("last_activity_at")} + for g in groups], sys.stdout, indent=1) + print() + return 0 + if not groups: + print("You belong to no group yet.", file=sys.stderr) + return 0 + print(f"{'NODE':<6}{'ROLE':<8}{'VISIBILITY':<12}{'LAST ACTIVITY':<18}GROUP") + for g in groups: + role = "owner" if g.get("is_admin") else "member" + print(f"{node_state(g):<6}{role:<8}{safe(g.get('visibility') or ''):<12}" + f"{_when(g.get('last_activity_at')):<18}{safe(handle(g))}") + return 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/login.py b/packages/meshbay-cli/src/meshbay_cli/commands/login.py new file mode 100644 index 0000000..cbd16ef --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/login.py @@ -0,0 +1,71 @@ +"""`login`: sign in with the passphrase, and remember the session.""" + +import contextlib +import json +import sys + +from ..client.account import Account +from ..client.groups import my_groups +from ..client.store import FILE_KEY_NOTICE, KEY_FILE +from ..errors import USAGE, CliError +from ..output import err, safe, sentence +from .context import Context +from .logout import sign_out_stored + +HELP = "sign in, and remember the session so later runs need no passphrase" + + +def add_parser(sub, parent) -> None: + p = sub.add_parser("login", parents=[parent], help=HELP, description=sentence(HELP) + ".") + p.add_argument("--no-remember", action="store_true", + help="check the credentials and store nothing") + + +async def run(ctx: Context) -> int: + username = ctx.username or ctx.stored_username + if not username: + if not sys.stdin.isatty(): + raise CliError(USAGE, "Name the account with --user.") + username = input("Username: ").strip() + ctx.username = username + if not username: + raise CliError(USAGE, "No username given.") + ctx.username = username + + notice = await ctx.hub.check_version() + remember = not ctx.args.no_remember + + # Signing in again as the same account keeps its device key, its node + # identities and its pins; another account replaces the store. + previous = ctx.store.peek() + account = Account(ctx.hub, username, ctx.passphrase) + if previous.get("hub") == ctx.hub.url and previous.get("username") == username: + with contextlib.suppress(CliError): + ctx.store.load() + account.state, account.secrets = ctx.store.state, ctx.store.secrets + elif remember and previous: + await sign_out_stored(ctx) + + await account.sign_in(remember=remember) + groups = await my_groups(account) + + if remember: + kind = await account.remember(ctx.store) + if kind == KEY_FILE: + err(FILE_KEY_NOTICE) + else: + await account.close() + + n = len(groups) + if ctx.args.json: + print(json.dumps({"username": username, "hub": ctx.hub.url, "groups": n, + "remembered": remember, + "key_store": ctx.store.peek().get("key_store") if remember + else None})) + else: + verb = "Signed in" if remember else "Credentials accepted" + print(f"{verb}: {safe(username)} on {ctx.hub.url} — " + f"{n} group{'s' if n != 1 else ''}.") + if notice: + err(notice) + return 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/logout.py b/packages/meshbay-cli/src/meshbay_cli/commands/logout.py new file mode 100644 index 0000000..7ee3aa2 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/logout.py @@ -0,0 +1,43 @@ +"""`logout`: end the session on the hub, retire this machine's key, erase the store.""" + +import json + +from ..client.account import Account, no_passphrase +from ..client.hub import Hub +from ..errors import CliError +from ..output import err, sentence +from .context import Context + +HELP = "end the session, retire this machine's device key and erase what is stored" + + +def add_parser(sub, parent) -> None: + sub.add_parser("logout", parents=[parent], help=HELP, description=sentence(HELP) + ".") + + +async def sign_out_stored(ctx: Context) -> None: + """Whatever account the store holds, on whatever hub. Never asks anything.""" + stored = ctx.store.peek() + hub = ctx.hub if stored.get("hub") == ctx.hub.url else Hub(stored.get("hub", "")) + try: + account = Account.open(hub, None, no_passphrase, ctx.store) + warnings = await account.sign_out() + except CliError as e: + warnings = [f"The saved session could not be opened ({e.message}); it is erased " + "here, and ends on the hub when it expires."] + with ctx.store.locked(): + ctx.store.erase() + finally: + if hub is not ctx.hub: + await hub.aclose() + for w in warnings: + err(f"meshbay-cli: warning: {w}") + + +async def run(ctx: Context) -> int: + if not ctx.store.peek(): + print(json.dumps({"signed_out": False}) if ctx.args.json else "Not signed in.") + return 0 + await sign_out_stored(ctx) + print(json.dumps({"signed_out": True}) if ctx.args.json else "Signed out.") + return 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/ls.py b/packages/meshbay-cli/src/meshbay_cli/commands/ls.py new file mode 100644 index 0000000..bdff3c0 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/ls.py @@ -0,0 +1,208 @@ +""" +`ls`: your groups, or what a folder of a group holds — as the Files tab shows it. + +With no path the hub answers alone (`/v1/groups/mine`) and no node is dialled. +Each group is its own top-level folder and nothing is merged across groups +(docs/MESHBAY_DESIGN.md §9.11). +""" + +import asyncio +import json +import sys + +from ..client.groups import my_groups +from ..client.paths import encode_name, handle, parse, resolve_group +from ..client.pool import Pool +from ..client.tree import Item, Tree +from ..errors import NODE, NOT_FOUND, PARTIAL, CliError +from ..output import columns, date, err, human_size, is_tty, safe, sentence +from .context import Context + +HELP = "list your groups, or a folder of a group" + + +def add_parser(sub, parent) -> None: + p = sub.add_parser("ls", parents=[parent], help=HELP, description=sentence(HELP) + ".") + p.add_argument("path", nargs="?", default="", + help="group[@owner]/root/folder; none lists your groups") + p.add_argument("-l", dest="long", action="store_true", + help="long form: type, size, date added") + p.add_argument("-R", dest="recursive", action="store_true", help="recurse into folders") + p.add_argument("-a", dest="all", action="store_true", + help="include the content of unavailable roots") + p.add_argument("--sort", choices=("name", "size", "date"), default="name", + help="the order within folders and files (default: name)") + p.add_argument("-r", dest="reverse", action="store_true", help="reverse the order") + + +def _sorted(items: list[Item], key: str, reverse: bool) -> list[Item]: + """Folders first, as the Files tab; the order applies within each.""" + def k(i: Item): + if key == "size": + return (i.size, i.name.lower()) + if key == "date": + return (i.added_at, i.name.lower()) + return (i.name.lower(), i.name) + dirs = sorted((i for i in items if i.is_dir), key=k, reverse=reverse) + files = sorted((i for i in items if not i.is_dir), key=k, reverse=reverse) + return dirs + files + + +def _flags(item: Item) -> str: + flags = [] + if item.root is not None: + flags.append("writable" if item.root.get("writable") else "read-only") + if not item.available: + flags.append("unavailable") + return f" ({', '.join(flags)})" if flags else "" + + +def _line(item: Item, long: bool) -> str: + name = safe(item.name) + ("/" if item.is_dir else "") + if not long: + return name + kind = "root" if item.root is not None else ("dir" if item.is_dir else "file") + size = "" if item.is_dir else human_size(item.size) + return f"{kind:<5}{size:>10} {date(item.added_at):<16} {name}{_flags(item)}" + + +def _print_items(items: list[Item], long: bool) -> None: + if long or not is_tty(): + for i in items: + print(_line(i, long)) + elif items: + print(columns([_line(i, False) for i in items])) + + +def _json_item(group: dict, item: Item) -> dict: + base = handle(group) + out = {"group": group["name"], "owner": group.get("owner_username"), + "path": f"{base}/{'/'.join(encode_name(p) for p in item.path.split('/'))}", + "name": item.name, "available": item.available, + # `kind`, not `type`: an index entry's `type` is its media type. + "kind": "root" if item.root is not None else ("dir" if item.is_dir else "file")} + if item.root is not None: + out["writable"] = bool(item.root.get("writable")) + if item.entry is not None: + out.update({k: v for k, v in item.entry.items() if k not in ("name", "path")}) + return out + + +async def _list_groups(ctx: Context, account) -> int: + groups = sorted(await my_groups(account), key=lambda g: (g["name"].lower(), + g.get("owner_username") or "")) + if ctx.args.json: + if ctx.args.recursive: + return await _every_group(ctx, account, groups) + json.dump([{"group": g["name"], "owner": g.get("owner_username"), "id": g["id"], + "path": handle(g), "online": bool(g.get("node_online")), + "visibility": g.get("visibility"), "admin": bool(g.get("is_admin"))} + for g in groups], sys.stdout, indent=1) + print() + return 0 + if ctx.args.recursive: + return await _every_group(ctx, account, groups) + if ctx.args.long or not is_tty(): + for g in groups: + state = "up" if g.get("node_online") else "down" + print(f"{state:<6}{safe(handle(g))}" if ctx.args.long else safe(handle(g))) + else: + print(columns([safe(handle(g)) for g in groups])) + return 0 + + +async def _every_group(ctx: Context, account, groups: list[dict]) -> int: + """ + Every group's tree. All are dialled at once through the pool, so a node that + does not answer costs its own deadline and nobody else's; a group whose node + the hub reports as down is not dialled at all. Printed in name order. + """ + pool = Pool(account) + + async def tree_of(group: dict): + if not group.get("node_online"): + raise CliError(NODE, "no node serving it is online") + session = await pool.session(group) + return Tree.from_index(await session.index()) + + try: + results = await asyncio.gather(*(tree_of(g) for g in groups), return_exceptions=True) + finally: + await pool.close() + failed = 0 + listing_json = [] + first = True + for group, result in zip(groups, results): + if isinstance(result, BaseException): + if not isinstance(result, CliError): + raise result + failed += 1 + err(f"meshbay-cli: {safe(handle(group))}: {result.message}") + continue + if ctx.args.json: + listing_json += _json_listing(ctx, group, result, "") + continue + if not first: + print() + first = False + _list_in_group(ctx, group, result, "") + if ctx.args.json: + json.dump(listing_json, sys.stdout, indent=1) + print() + return PARTIAL if failed else 0 + + +async def run(ctx: Context) -> int: + await ctx.hub.check_version() + account = ctx.account() + try: + remote = parse(ctx.args.path) + if remote is None: + return await _list_groups(ctx, account) + group = resolve_group(await my_groups(account), remote.group) + pool = Pool(account) + try: + tree = Tree.from_index(await (await pool.session(group)).index()) + finally: + await pool.close() + return _list_in_group(ctx, group, tree, remote.inner) + finally: + await account.close() + + +def _list_in_group(ctx: Context, group: dict, tree: Tree, path: str) -> int: + a = ctx.args + if path and not tree.is_dir(path): + entry = tree.file(path) + if entry is None: + raise CliError(NOT_FOUND, f"No file or folder {path!r} in {handle(group)}.") + parent = path.rpartition("/")[0] + listing = [(None, [Item(entry["name"], path, False, entry=entry, + available=tree.root_available(parent))])] + elif a.recursive: + listing = list(tree.walk(path, include_unavailable=a.all)) + else: + listing = [(path, tree.children(path, include_unavailable=a.all))] + + if a.json: + json.dump([_json_item(group, i) for _, items in listing for i in items], + sys.stdout, indent=1) + print() + return 0 + _print_listing(a, group, listing) + return 0 + + +def _json_listing(ctx: Context, group: dict, tree: Tree, path: str) -> list[dict]: + return [_json_item(group, i) + for _, items in tree.walk(path, include_unavailable=ctx.args.all) for i in items] + + +def _print_listing(a, group: dict, listing) -> None: + for n, (folder, items) in enumerate(listing): + if a.recursive: + if n: + print() + shown = "/".join(encode_name(p) for p in folder.split("/")) if folder else "" + print(safe(f"{handle(group)}/{shown}".rstrip("/")) + ":") + _print_items(_sorted(items, a.sort, a.reverse), a.long) diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/put.py b/packages/meshbay-cli/src/meshbay_cli/commands/put.py new file mode 100644 index 0000000..01a7233 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/put.py @@ -0,0 +1,195 @@ +""" +`put`: upload files, or folders with -R, into a folder of a group. + +**Decided whole before anything is sent**, as a drop on the Files tab is +(docs/MESHBAY_DESIGN.md §6.4): a name already in the destination — compared +without case — or one the node does not accept refuses the run and lists every +offender, so a folder never arrives with holes in it. `-n` skips the names +already there instead. There is no overwrite: the node never replaces a file, +and if one appears while an upload runs, the upload takes a free name and the +CLI says which. +""" + +import asyncio +import contextlib +import os +import sys +import tempfile +import threading +from pathlib import Path + +from ..client.groups import my_groups +from ..client.paths import handle, parse, resolve_group +from ..client.pool import Pool +from ..client.transfers import TransferError, create_dir, upload +from ..client.tree import Tree +from ..client.upload_plan import UploadFile, plan_upload, refused_names, taken_names +from ..errors import NOT_FOUND, PARTIAL, REFUSED, USAGE, CliError +from ..output import Progress, err, human_size, safe, sentence +from .context import Context + +HELP = "upload files, or folders with -R, into a folder of a group" +PER_GROUP = 2 + + +def add_parser(sub, parent) -> None: + p = sub.add_parser("put", parents=[parent], help=HELP, description=sentence(HELP) + + ". With - as the source, stdin is sent and DEST names the file.") + p.add_argument("sources", nargs="+", metavar="SOURCE", + help="local files or folders, or - for stdin") + p.add_argument("dest", metavar="DEST", + help="the destination folder: group[@owner]/root/folder; " + "with - as the source, the file to write") + p.add_argument("-R", dest="recursive", action="store_true", help="folders, recursively") + p.add_argument("-n", dest="skip", action="store_true", + help="skip names already in the destination instead of refusing") + + +async def run(ctx: Context) -> int: + a = ctx.args + sources, dest_text = a.sources, a.dest + remote = parse(dest_text) + if remote is None or not remote.parts: + raise CliError(USAGE, "The destination is a folder of a group: " + "group[@owner]/root/folder") + from_stdin = sources == ["-"] + if "-" in sources and not from_stdin: + raise CliError(USAGE, "- (stdin) is the only source when it is used.") + + await ctx.hub.check_version() + account = ctx.account() + pool = Pool(account) + spool = None + try: + group = resolve_group(await my_groups(account), remote.group) + session = await pool.session(group) + tree = Tree.from_index(await session.index()) + + parts = remote.parts[:-1] if from_stdin else remote.parts + folder = "/".join(parts) + _check_folder(tree, group, folder) + + if from_stdin: + spool = await asyncio.to_thread(_spool_stdin) + name = remote.parts[-1] + plan_dirs: list[list[str]] = [] + files = [UploadFile(Path(spool.name), [name], os.path.getsize(spool.name))] + problems = refused_names([name], []) + problems += [f"{folder}/{t} is already there" + for t in sorted(taken_names(tree, folder, [name]))] + else: + plan_dirs, files, problems = plan_upload(sources, a.recursive) + tops = sorted({tuple(d[:1]) for d in plan_dirs} | {tuple(f.rel[:1]) for f in files}) + taken = taken_names(tree, folder, [t[0] for t in tops]) + if taken and a.skip: + plan_dirs = [d for d in plan_dirs if d[0] not in taken] + files = [f for f in files if f.rel[0] not in taken] + elif taken: + problems += [f"{folder}/{t} is already there (-n to skip it)" + for t in sorted(taken)] + problems += refused_names([f.rel[-1] for f in files], plan_dirs) + if problems: + raise CliError(REFUSED, "Nothing was sent:\n " + + "\n ".join(safe(p) for p in problems)) + return await _send(ctx, session, group, folder, plan_dirs, files, + display_names=[remote.parts[-1]] if from_stdin else None) + finally: + if spool is not None: + with contextlib.suppress(OSError): + os.unlink(spool.name) + await pool.close() + await account.close() + + +def _check_folder(tree: Tree, group: dict, folder: str) -> None: + shown = f"{handle(group)}/{folder}" + if not tree.is_dir(folder): + raise CliError(NOT_FOUND, f"No folder {safe(shown)}.") + root = tree.roots.get(folder.split("/", 1)[0], {}) + if not root.get("available", True): + raise CliError(REFUSED, f"{safe(shown)} is unavailable on the node " + "(its drive is disconnected).") + if not root.get("writable"): + raise CliError(REFUSED, f"{safe(shown)} is read-only: its operator does not " + "accept uploads there.") + + +def _spool_stdin(): + """stdin to a file first: every chunk names the total, so the size comes first.""" + f = tempfile.NamedTemporaryFile(prefix="meshbay-cli-", delete=False) + with f: + while block := sys.stdin.buffer.read(1024 * 1024): + f.write(block) + return f + + +async def _send(ctx: Context, session, group: dict, folder: str, dirs: list[list[str]], + files: list[UploadFile], display_names: list[str] | None) -> int: + a = ctx.args + progress = Progress("↑", quiet=a.quiet, as_json=a.json) + progress.plan(len(files), sum(f.size for f in files)) + + # Folders first, parents before children, one at a time. + remote_dir: dict[tuple, str] = {(): folder} + try: + for d in dirs: + remote_dir[tuple(d)] = await create_dir(session, remote_dir[tuple(d[:-1])], d[-1]) + except TransferError as e: + progress.clear() + created = len(remote_dir) - 1 + raise CliError(PARTIAL if created else e.code, + f"Stopped: {e.message}" + + (f" ({created} folder{'s' if created != 1 else ''} created)" + if created else "")) from e + + failed = 0 + stop: TransferError | None = None + gate = asyncio.Semaphore(PER_GROUP) + + async def one(job: UploadFile) -> None: + nonlocal failed, stop + async with gate: + if stop is not None: + return + name = job.rel[-1] + shown = f"{handle(group)}/{remote_dir[tuple(job.rel[:-1])]}/{name}" + progress.start(safe(name)) + lock = threading.Lock() + try: + with open(job.local, "rb") as fh: + def read(offset: int, length: int) -> bytes: + with lock: + fh.seek(offset) + return fh.read(length) + stored = await upload( + session, read, job.size, name, remote_dir[tuple(job.rel[:-1])], + progress=progress.advance, + on_queued=lambda n: progress.queued(safe(name), n)) + except OSError as e: + failed += 1 + progress.note(f"meshbay-cli: {job.local}: {e.strerror}") + return + except TransferError as e: + failed += 1 + progress.note(f"meshbay-cli: {safe(shown)}: {e.message}") + if e.fatal: + stop = e + return + if stored != name: + progress.note(f"meshbay-cli: {safe(name)} was stored as {safe(stored)} " + "(the name was taken meanwhile)") + progress.finish(safe(name), path=shown.rsplit("/", 1)[0] + "/" + stored, + size=job.size) + + await asyncio.gather(*(one(f) for f in files)) + progress.clear() + if stop is not None: + skipped = len(files) - progress.files_done - failed + err(f"meshbay-cli: stopped: {stop.message}" + + (f" ({skipped} file{'s' if skipped != 1 else ''} not sent)" if skipped else "")) + failed += skipped + if not a.quiet and not a.json: + done = progress.files_done + err(f"Uploaded {done} file{'s' if done != 1 else ''} " + f"({human_size(progress.bytes_done)})" + (f"; {failed} failed." if failed else ".")) + return PARTIAL if failed else 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/commands/trust.py b/packages/meshbay-cli/src/meshbay_cli/commands/trust.py new file mode 100644 index 0000000..f48a955 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/commands/trust.py @@ -0,0 +1,54 @@ +""" +`trust`: the node keys this machine pinned at first contact. + +A node presenting another key than the one pinned is refused outright (exit 6, +docs/MESHBAY_DESIGN.md §5.2). `trust reset NODE` is the deliberate way out, for a +rotation the node's operator has announced — the counterpart of the reset in +the application's Settings. Nothing resets a pin by itself. +""" + +import json +import sys + +from ..client.account import Account, no_passphrase +from ..errors import NOT_FOUND, USAGE, CliError +from ..output import safe, sentence +from .context import Context + +HELP = "the node keys pinned at first contact: list them, or forget one" + + +def add_parser(sub, parent) -> None: + p = sub.add_parser("trust", parents=[parent], help=HELP, description=sentence(HELP) + + ". A node whose key changed is refused until its pin is reset.") + p.add_argument("verb", choices=("list", "reset")) + p.add_argument("node", nargs="?", help="the node's id, as the refusal names it") + + +async def run(ctx: Context) -> int: + a = ctx.args + if not ctx.store.peek(): + pins = {} + account = None + else: + account = Account.open(ctx.hub, None, no_passphrase, ctx.store) + pins = account.state.get("pins", {}) if account.remembered else {} + if a.verb == "list": + if a.json: + json.dump(pins, sys.stdout, indent=1) + print() + else: + for node_id, pk in sorted(pins.items()): + print(f"{safe(node_id)} {safe(pk)}") + return 0 + if not a.node: + raise CliError(USAGE, "Name the node: meshbay-cli trust reset NODE") + if a.node not in pins: + raise CliError(NOT_FOUND, f"No key is pinned for the node {safe(a.node)}.") + + def drop(state: dict, secrets: dict) -> None: + state.get("pins", {}).pop(a.node, None) + await account.update(drop) + print(f"Forgot the key pinned for {safe(a.node)}; the next connection pins the key " + "it presents.") + return 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/errors.py b/packages/meshbay-cli/src/meshbay_cli/errors.py new file mode 100644 index 0000000..319645e --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/errors.py @@ -0,0 +1,24 @@ +""" +Every way a run can end badly, as an exit code a script can act on. + +The library raises `CliError` with one of these codes; only `__main__` turns it +into a message and an exit status. +""" + +USAGE = 1 +AUTH = 2 # sign-in refused, no session and no passphrase, locked out +NOT_FOUND = 3 # group, path, or a pattern that matches nothing +PARTIAL = 4 # some items failed +NODE = 5 # node unreachable, or it refused the connection +NODE_KEY = 6 # a node's key changed since it was pinned +VERSION = 7 # this client, or the protocol, is too old or too new +REFUSED = 8 # refused before anything was sent: a name taken or not accepted, + # a folder read-only or unavailable +INTERRUPTED = 130 + + +class CliError(Exception): + def __init__(self, code: int, message: str): + super().__init__(message) + self.code = code + self.message = message diff --git a/packages/meshbay-cli/src/meshbay_cli/manpage.py b/packages/meshbay-cli/src/meshbay_cli/manpage.py new file mode 100644 index 0000000..0633904 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/manpage.py @@ -0,0 +1,344 @@ +""" +The man page, generated from the parser: `python -m meshbay_cli.manpage`. + +Every action and option comes from `build_parser()`, so the page cannot list an +option the program lacks or miss one it has; the prose the parser cannot carry +(remote paths, the store, exit codes, environment) is written here. The +packages build it at build time and install it in /usr/share/man/man1. +""" + +import argparse +import sys + +from . import __version__, errors +from .__main__ import ACTIONS, build_parser + +EXIT_CODES = { + 0: "Success.", + errors.USAGE: "Usage error: an unknown option, a missing argument.", + errors.AUTH: "Sign-in refused, no stored session and no passphrase to sign in " + "with, or the account is locked out.", + errors.NOT_FOUND: "A group, a path, or a pattern that matches nothing.", + errors.PARTIAL: "Some items failed; the others were done. Each failure is " + "named on stderr.", + errors.NODE: "No node serving the group could be reached, or the node refused " + "the connection. Also: this machine is waiting for device " + "approval (the code is on stderr).", + errors.NODE_KEY: "A node presented another key than the one pinned at first " + "contact. Nothing was sent. See \\fBtrust\\fR.", + errors.VERSION: "This client, or the protocol it speaks, is too old or too new " + "for the hub or the node.", + errors.REFUSED: "Refused before anything was sent: a name already taken or not " + "accepted by the node, a folder read-only or unavailable.", + errors.INTERRUPTED: "Interrupted (Ctrl+C). A download resumes where it stopped " + "on the next run.", +} + +DESCRIPTION = r""" +.B meshbay-cli +is a member's view of their MeshBay groups from a terminal: sign in, list the +groups and their folders, download and upload files. +It talks to the hub only to sign in and to find a group's nodes; files go +directly between this machine and the node, end-to-end encrypted, never +through the hub. +.PP +Run with no argument at all, it opens the interactive mode (below). +Global options are accepted before the action and after it. +.PP +Administering a node is not this program's business: see +.BR meshbay-node (1). +""" + +PATHS = r""" +A remote path names a group, then a root the node shares, then folders and a +file: +.PP +.RS +.EX +group[@owner]/root/folder/file +.EE +.RE +.PP +.B @owner +is needed only when two of your groups share a name; group names compare +without regard to case. +A group name may itself contain +.B / +or +.BR @ : +type them as +.B %2F +and +.B %40 +(and a literal +.B % +as +.BR %25 ). +The GROUP column of +.B meshbay-cli groups +prints each group in this form. +A link copied from the application's address bar (containing +.BR #/ ) +is accepted as it is. +.PP +.B get +matches the wildcards +.BR * , +.B ? +and +.B [...] +against the group's index, not against local files: quote them. +""" + +INTERACTIVE = r""" +.B meshbay-cli +with no argument opens two panes: this machine's folders on the left, your +groups on the right, and the transfers below. +A sign-in screen comes first when no session is stored. +.TP +.B Tab +switch pane +.TP +.B Enter +open a folder or a group +.TP +.B Backspace +go up +.TP +.B Space +select an entry (several can be) +.TP +.B F5 +copy the selection, or the entry under the cursor, to the other pane's folder; +a name already there is asked about once for the whole batch +.TP +.B / +filter by name; +.B Esc +clears the filter +.TP +.B s +cycle the sort: name, size, date +.TP +.B r +refresh +.TP +.B q +quit +.PP +The right pane follows the node: a file added by someone else appears there. +""" + +SESSION = r""" +.B login +signs in with the account's passphrase and, unless +.B \-\-no\-remember +is given, keeps the session so that later runs need no passphrase. +The passphrase is read from, in order: +.BR \-\-password\-file , +.BR \-\-password , +.BR MESHBAY_PASSWORD , +then a prompt when there is a terminal. +.B \-\-password +puts it in the process list and the shell history; prefer +.B \-\-password\-file \- +(stdin). +.PP +The session and this machine's keys are kept encrypted in the data directory +(see FILES), under a key held by the system's key storage (Secret Service, +macOS Keychain, Windows Credential Manager). +Where there is none, such as a server with no desktop session, the key is kept +in a file only your user can read, like an SSH key without a passphrase; +.B login +says so once. +.B logout +ends the session on the hub, retires this machine's device key and erases +everything stored. +.PP +The first time this machine opens a group whose node holds no copy of your +keys (the application's "Browser access" turned off), it asks to be linked as +a new device and prints a code. +Approve that code from the application on a device already linked (the +group's Settings); with a terminal, +.B meshbay-cli +waits and goes on by itself, otherwise it exits with status 5 and the next run +goes on once the code is approved. +.PP +Each node's key is pinned at first contact. +A node presenting another key is refused, with status 6, until +.B meshbay-cli trust reset +.I NODE +forgets the pin; do that only for a key change the node's operator announced. +""" + +FILES = r""" +.TP +.I $XDG_DATA_HOME/meshbay\-cli/ \fR(default \fI~/.local/share/meshbay\-cli/\fR) +On Windows, +.IR %LOCALAPPDATA%\emeshbay\-cli\e . +Holds +.B state.json +(the hub, the account name, pinned node keys), +.B secrets.enc +(the session and keys, encrypted), +.B store.key +only where the system has no key storage, and +.B interactive.log +(the interactive mode's log). +.TP +\fIname\fB.meshbay\-part\fR, \fIname\fB.meshbay\-part.json\fR +A download in progress, next to where the file lands. +A later +.B get +of the same file resumes from it; the finished file replaces it. +""" + +ENVIRONMENT = r""" +.TP +.B MESHBAY_HUB +The hub, when +.B \-\-hub +is not given (default: the one signed in to, else https://meshbay.org). +.TP +.B MESHBAY_USER +The account name, when +.B \-u +is not given. +.TP +.B MESHBAY_PASSWORD +The passphrase, when neither +.B \-\-password\-file +nor +.B \-\-password +is given. +.TP +.B MESHBAY_CLI_HOME +The data directory, instead of the default (see FILES). +.TP +.B NO_COLOR +The interactive mode draws without colour. +""" + +EXAMPLES = r""" +.EX +meshbay\-cli login \-u alice +meshbay\-cli groups +meshbay\-cli ls \-l Family/photos/2024 +meshbay\-cli get \-R Family/photos/2024 \-o ~/Pictures/ +meshbay\-cli get 'Family/music/*.flac' +meshbay\-cli put \-R ~/scans Family/documents +tar c notes | meshbay\-cli put \- Family/documents/notes.tar +meshbay\-cli ls \-\-json \-R / | jq '.[] | .path' +.EE +""" + + +def _roff(text: str) -> str: + """ + Plain text into roff: no control line, no escape, no hyphen taken for a + dash, and anything past ASCII named, which troff reads whatever the locale. + """ + text = text.replace("\\", "\\e").replace("-", "\\-") + text = "".join(c if ord(c) < 128 else f"\\[u{ord(c):04X}]" for c in text) + return "\n".join("\\&" + line if line[:1] in (".", "'") else line + for line in text.splitlines()) + + +def _option_lines(action: argparse.Action) -> str: + if action.option_strings: + names = ", ".join(f"\\fB{_roff(s)}\\fR" for s in action.option_strings) + if action.choices: + names += " " + "|".join(f"\\fB{_roff(c)}\\fR" for c in action.choices) + elif action.nargs != 0: + names += f" \\fI{_roff(action.metavar or action.dest.upper())}\\fR" + else: + names = f"\\fI{_roff(action.metavar or action.dest.upper())}\\fR" + if action.choices: + names = " | ".join(f"\\fB{_roff(c)}\\fR" for c in action.choices) + help_text = action.help or "" + return f".TP\n{names}\n{_roff(help_text)}\n" + + +def _own_options(parser: argparse.ArgumentParser) -> list[argparse.Action]: + """The action's own arguments; the global ones are listed once, above.""" + late = {s for a in _globals(build_parser()) for s in a.option_strings} + return [a for a in parser._actions + if not isinstance(a, argparse._HelpAction) + and not (set(a.option_strings) & late)] + + +def _globals(parser: argparse.ArgumentParser) -> list[argparse.Action]: + return [a for a in parser._actions + if a.option_strings and not isinstance(a, argparse._SubParsersAction)] + + +def _subparsers(parser: argparse.ArgumentParser) -> dict[str, argparse.ArgumentParser]: + sub = next(a for a in parser._actions if isinstance(a, argparse._SubParsersAction)) + return dict(sub.choices) + + +def _synopsis(name: str, parser: argparse.ArgumentParser) -> str: + """`meshbay-cli ls [-l] ... [path]`, without the global options.""" + parts = [] + own = _own_options(parser) + for a in sorted(own, key=lambda a: not a.option_strings): # options first + if a.option_strings: + s = a.option_strings[0] + if a.choices: + s += " " + "|".join(a.choices) + elif a.nargs != 0: + s += f" {a.metavar or a.dest.upper()}" + parts.append(f"[{s}]") + elif a.choices: + parts.append(" | ".join(a.choices)) + else: + meta = a.metavar or a.dest.upper() + parts.append({"?": f"[{meta}]", "+": f"{meta}...", "*": f"[{meta}...]"} + .get(a.nargs, meta)) + return _roff(" ".join([name] + parts)) + + +def render() -> str: + parser = build_parser() + out = [ + f'.TH MESHBAY\\-CLI 1 "" "meshbay\\-cli {__version__}" "User Commands"', + ".nh", ".ad l", + ".SH NAME", + "meshbay\\-cli \\- a member's MeshBay groups and files, from a terminal", + ".SH SYNOPSIS", + ".B meshbay\\-cli", + ".br", + ".B meshbay\\-cli", + "[\\fIglobal options\\fR] \\fIaction\\fR [\\fIaction options\\fR] " + "[\\fIarguments\\fR]", + ".SH DESCRIPTION", + DESCRIPTION.strip(), + ".SH GLOBAL OPTIONS", + ] + out += [_option_lines(a).rstrip() for a in _globals(parser)] + out.append(".SH ACTIONS") + subs = _subparsers(parser) + for name in ACTIONS: + sp = subs[name] + out += [f".SS {_roff(name)}", f".B {_synopsis(name, sp)}", ".PP", + _roff(sp.description or "")] + out += [_option_lines(a).rstrip() for a in _own_options(sp)] + out += [".SH REMOTE PATHS", PATHS.strip(), + ".SH INTERACTIVE MODE", INTERACTIVE.strip(), + ".SH SESSION, DEVICES AND NODE KEYS", SESSION.strip(), + ".SH EXIT STATUS"] + for code, text in sorted(EXIT_CODES.items()): + out += [".TP", f".B {code}", text] + out += [".SH ENVIRONMENT", ENVIRONMENT.strip(), + ".SH FILES", FILES.strip(), + ".SH EXAMPLES", EXAMPLES.strip(), + ".SH SEE ALSO", ".BR meshbay\\-node (1)", ""] + return "\n".join(out) + + +def main() -> None: + sys.stdout.write(render()) + + +if __name__ == "__main__": + main() diff --git a/packages/meshbay-cli/src/meshbay_cli/output.py b/packages/meshbay-cli/src/meshbay_cli/output.py new file mode 100644 index 0000000..75a54ff --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/output.py @@ -0,0 +1,140 @@ +""" +Writing to the terminal. + +**Names are someone else's text.** A file or group name is chosen by whoever put +it in the group, and a raw ESC in it is an escape sequence the reader's +terminal executes; a bidirectional override makes it display as another name. +Every name printed goes through `safe()`. +""" + +import json +import shutil +import sys +import time +import unicodedata + +_BIDI = frozenset("") + + +def safe(text: str) -> str: + out = [] + for c in text: + if unicodedata.category(c) == "Cc" or c in _BIDI: + out.append(f"\\x{ord(c):02x}" if ord(c) < 0x100 else f"\\u{ord(c):04x}") + else: + out.append(c) + return "".join(out) + + +def human_size(n: int) -> str: + size = float(n) + for unit in ("B", "KB", "MB", "GB", "TB"): + if size < 1024 or unit == "TB": + return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}" + size /= 1024 + return f"{n} B" + + +def date(ts: int) -> str: + return time.strftime("%Y-%m-%d %H:%M", time.localtime(ts)) if ts else "" + + +def columns(names: list[str]) -> str: + """`ls`'s short form on a terminal: as many columns as fit, filled down.""" + if not names: + return "" + width = shutil.get_terminal_size((80, 24)).columns + col = max(len(n) for n in names) + 2 + ncols = max(1, width // col) + nrows = -(-len(names) // ncols) + lines = [] + for r in range(nrows): + row = [names[c * nrows + r] for c in range(ncols) if c * nrows + r < len(names)] + lines.append("".join(n.ljust(col) for n in row).rstrip()) + return "\n".join(lines) + + +def is_tty() -> bool: + return sys.stdout.isatty() + + +def err(message: str) -> None: + print(message, file=sys.stderr) + + +class Progress: + """ + One status line on stderr for a run of transfers, redrawn at most ten times + a second; JSON Lines with --json; nothing with -q or when stderr is not a + terminal. + """ + + def __init__(self, arrow: str, *, quiet: bool = False, as_json: bool = False): + self.arrow = arrow + self.live = sys.stderr.isatty() and not quiet and not as_json + self.as_json = as_json and not quiet + self.files_total = 0 + self.files_done = 0 + self.bytes_total = 0 + self.bytes_done = 0 + self.current = "" + self._start = time.monotonic() + self._drawn = 0.0 + + def plan(self, files: int, nbytes: int) -> None: + self.files_total, self.bytes_total = files, nbytes + + def event(self, **fields) -> None: + if self.as_json: + print(json.dumps(fields, ensure_ascii=False), file=sys.stderr, flush=True) + + def start(self, name: str) -> None: + self.current = name + self.event(event="start", name=name) + self._draw(force=True) + + def advance(self, nbytes: int) -> None: + self.bytes_done += nbytes + self._draw() + + def queued(self, name: str, ahead: int) -> None: + self.current = f"{name} (waiting for a free slot on the node, {ahead} ahead)" + self.event(event="queued", name=name, ahead=ahead) + self._draw(force=True) + + def finish(self, name: str, **fields) -> None: + self.files_done += 1 + self.event(event="done", name=name, **fields) + self._draw(force=True) + + def note(self, message: str) -> None: + """A line of its own, above the status line.""" + self.clear() + if not self.as_json: + print(message, file=sys.stderr) + self._draw(force=True) + + def _draw(self, force: bool = False) -> None: + if not self.live: + return + now = time.monotonic() + if not force and now - self._drawn < 0.1: + return + self._drawn = now + rate = self.bytes_done / max(now - self._start, 0.001) + line = (f"{self.arrow} {self.files_done}/{self.files_total} " + f"{human_size(self.bytes_done)} / {human_size(self.bytes_total)} " + f"{human_size(int(rate))}/s {self.current}") + width = shutil.get_terminal_size((80, 24)).columns + sys.stderr.write("\r\x1b[K" + line[:width - 1]) + sys.stderr.flush() + + def clear(self) -> None: + if self.live: + sys.stderr.write("\r\x1b[K") + sys.stderr.flush() + + +def sentence(text: str) -> str: + """The first letter up, the rest untouched: `str.capitalize` turns -R into -r.""" + return text[:1].upper() + text[1:] diff --git a/packages/meshbay-cli/src/meshbay_cli/selftest.py b/packages/meshbay-cli/src/meshbay_cli/selftest.py new file mode 100644 index 0000000..81453c4 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/selftest.py @@ -0,0 +1,61 @@ +""" +What a frozen build must be able to load, checked with no hub and no node. + +A PyInstaller build carries only the modules its analysis found, and a module +it missed fails at run time only — in the interactive mode, or the first time +the store asks for the system's key storage. The Windows build runs this +(`MESHBAY_CLI_SELFTEST=1 meshbay-cli`, see packaging/win/build-node-runtime.ps1) +on the frozen executable; nothing here is reachable through the command line. +""" + +import asyncio +import os +import sys +import tempfile + + +async def _interactive() -> None: + """The sign-in screen, drawn off-screen: Textual's widgets, CSS and Rich.""" + from .__main__ import build_parser + from .commands.context import Context + from .tui.app import MeshBayApp + from .tui.screens import SignInScreen + + app = MeshBayApp(Context(build_parser().parse_args(["--hub", "http://127.0.0.1:1"]))) + async with app.run_test(size=(100, 30)) as pilot: + for _ in range(100): + if isinstance(app.screen, SignInScreen): + break + await pilot.pause(0.05) + else: + raise RuntimeError("the sign-in screen never showed") + app.screen.query_one("#user") + + +async def _webrtc() -> None: + from aiortc import RTCPeerConnection + pc = RTCPeerConnection() + pc.createDataChannel("selftest") + await pc.setLocalDescription(await pc.createOffer()) + await pc.close() + + +def run() -> int: + with tempfile.TemporaryDirectory() as home: + os.environ["MESHBAY_CLI_HOME"] = home # nothing stored: the sign-in screen + # keyring finds its backends through its own entry points: without its + # metadata in the build, Windows Credential Manager is simply not + # there and the store falls back to a key file on every machine. + import keyring.backend + backends = [type(b).__module__ for b in keyring.backend.get_all_keyring()] + if sys.platform == "win32" and "keyring.backends.Windows" not in backends: + raise RuntimeError(f"no Windows Credential Manager backend in {backends}") + + from argon2.low_level import Type, hash_secret_raw + hash_secret_raw(b"x", b"saltsalt", 1, 8, 1, 16, Type.ID) + + from .__main__ import loop_factory + asyncio.run(_webrtc(), loop_factory=loop_factory()) + asyncio.run(_interactive(), loop_factory=loop_factory()) + print(f"self-test ok (keyring: {', '.join(backends)})") + return 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/tui/__init__.py b/packages/meshbay-cli/src/meshbay_cli/tui/__init__.py new file mode 100644 index 0000000..becd195 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/tui/__init__.py @@ -0,0 +1 @@ +"""The interactive mode: `meshbay-cli` with no argument.""" diff --git a/packages/meshbay-cli/src/meshbay_cli/tui/app.py b/packages/meshbay-cli/src/meshbay_cli/tui/app.py new file mode 100644 index 0000000..dbb773c --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/tui/app.py @@ -0,0 +1,76 @@ +""" +`meshbay-cli` with no argument: a full-screen terminal interface. + +Textual, because it runs on the same asyncio loop as aiortc and httpx rather +than beside them, and on Linux terminals and Windows' consoles alike. Signed in +already, it opens on the two panes; otherwise on the sign-in screen first. + +Nothing is printed while it runs — a line on stderr would tear the screen — so +the log goes to a file in the CLI's data directory instead. +""" + +import logging + +from textual.app import App + +from ..client.account import Account, no_passphrase +from ..client.hub import Hub +from ..client.store import data_dir +from ..commands.context import Context +from ..errors import CliError +from .screens import DeviceCode, MainScreen, SignInScreen + + +class MeshBayApp(App): + TITLE = "MeshBay" + + def __init__(self, ctx: Context): + super().__init__() + self.ctx = ctx + self.hub = ctx.hub + + async def use_hub(self, url: str) -> None: + if url != self.hub.url: + await self.hub.aclose() + self.hub = Hub(url) + self.ctx.hub = self.hub + self.ctx.hub_url = url + + def on_mount(self) -> None: + account = None + if self.ctx.store.peek().get("hub") == self.hub.url: + try: + account = Account.open(self.hub, None, no_passphrase, self.ctx.store) + except CliError as e: + self.notify(e.message, severity="warning", markup=False) + if account is not None and account.remembered: + self.start(account) + else: + self.push_screen(SignInScreen()) + + def start(self, account: Account) -> None: + account.on_device_code = DeviceCode(self) + if isinstance(self.screen, SignInScreen): + self.switch_screen(MainScreen(account)) + else: + self.push_screen(MainScreen(account)) + + +def _log_to_file() -> None: + root = logging.getLogger() + for h in list(root.handlers): + root.removeHandler(h) + try: + directory = data_dir() + directory.mkdir(parents=True, exist_ok=True) + handler = logging.FileHandler(directory / "interactive.log", encoding="utf-8") + except OSError: + handler = logging.NullHandler() + handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s")) + root.addHandler(handler) + + +async def run_interactive(ctx: Context) -> int: + _log_to_file() + await MeshBayApp(ctx).run_async() + return 0 diff --git a/packages/meshbay-cli/src/meshbay_cli/tui/panes.py b/packages/meshbay-cli/src/meshbay_cli/tui/panes.py new file mode 100644 index 0000000..d1e3dd0 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/tui/panes.py @@ -0,0 +1,372 @@ +""" +The two panes: this machine's folders on the left, the groups on the right. + +Both draw rows the same way — folders first, then files, each sorted by the +pane's key — and both mark a selection with Space. Names are someone else's +text on the right and may be on the left: every name is escaped (`safe`) and +handed to the table as plain text, never as markup. +""" + +import asyncio +import os +import stat +from dataclasses import dataclass, field +from pathlib import Path + +from rich.text import Text +from textual.app import ComposeResult +from textual.binding import Binding +from textual.containers import Vertical +from textual.widgets import DataTable, Input, Static + +from ..client.groups import my_groups +from ..client.paths import handle +from ..client.tree import Tree +from ..errors import CliError +from ..output import date, human_size, safe + +PARENT = ".." +SORTS = ("name", "size", "date") +SIZE_WIDTH, DATE_WIDTH = 9, 16 +# Cell padding on both sides of three columns, and the vertical scrollbar. +CHROME = 2 * 3 + 2 + + +@dataclass +class Row: + key: str + name: str + is_dir: bool + size: int | None = None + when: float | None = None + note: str = "" # "down", "read-only", ... + dim: bool = False + data: dict = field(default_factory=dict) + + +class Pane(Vertical, can_focus=False): + """A titled table with marks, a sort key and a filter.""" + + BINDINGS = [ + Binding("space", "mark", "Select"), + Binding("backspace", "up", "Up"), + Binding("slash", "filter", "Filter"), + Binding("s", "sort", "Sort"), + Binding("r", "refresh", "Refresh"), + ] + + DEFAULT_CSS = """ + Pane { width: 1fr; border: round $panel; } + Pane:focus-within { border: round $accent; } + Pane > .title { height: 1; padding: 0 1; background: $panel; } + Pane > DataTable { height: 1fr; } + Pane > Input { display: none; } + Pane > Input.shown { display: block; } + """ + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.rows: list[Row] = [] + self.marks: set[str] = set() + self.sort = "name" + self.filter_text = "" + self.name_width = 30 + self.show_date = True + + def compose(self) -> ComposeResult: + yield Static("", classes="title", markup=False) + table = DataTable(cursor_type="row", zebra_stripes=False) + self._add_columns(table) + yield table + yield Input(placeholder="filter — Enter to apply, Esc to clear") + + def _add_columns(self, table: DataTable) -> None: + table.add_column("Name", key="name", width=self.name_width) + table.add_column(Text("Size", justify="right"), key="size", width=SIZE_WIDTH) + if self.show_date: + table.add_column("Date", key="date", width=DATE_WIDTH) + + def on_resize(self) -> None: + """ + The name takes what the size and the date leave, never more: a long name + with no spaces in it must not push them out of the pane. In a pane too + narrow for all three (two panes in an 80-column terminal), the date goes. + """ + self._draw_title() + inner = self.table.size.width + show_date = inner - SIZE_WIDTH - DATE_WIDTH - CHROME >= 16 + width = max(8, inner - SIZE_WIDTH - (DATE_WIDTH if show_date else 0) + - (CHROME if show_date else CHROME - 2)) + if (width, show_date) != (self.name_width, self.show_date): + self.name_width, self.show_date = width, show_date + self.table.clear(columns=True) + self._add_columns(self.table) + self.redraw() + + @property + def table(self) -> DataTable: + return self.query_one(DataTable) + + def set_title(self, text: str) -> None: + self._title = text + self._draw_title() + + def _draw_title(self) -> None: + """One line; a long path loses its beginning, which is the part known.""" + text = getattr(self, "_title", "") + room = max(8, self.size.width - 4) + if len(text) > room: + text = "…" + text[-(room - 1):] + self.query_one(".title", Static).update(Text(text, no_wrap=True)) + + # ── Drawing ────────────────────────────────────────────────────────────── + + def show(self, rows: list[Row], *, keep_cursor: str | None = None) -> None: + self.rows = rows + self.marks &= {r.key for r in rows} + self.redraw(keep_cursor=keep_cursor) + + def _ordered(self) -> list[Row]: + def k(r: Row): + if self.sort == "size": + return (-(r.size or 0), r.name.lower()) + if self.sort == "date": + return (-(r.when or 0), r.name.lower()) + return (r.name.lower(), r.name) + visible = [r for r in self.rows if r.key != PARENT + and self.filter_text.lower() in r.name.lower()] + parent = [r for r in self.rows if r.key == PARENT] + return (parent + sorted((r for r in visible if r.is_dir), key=k) + + sorted((r for r in visible if not r.is_dir), key=k)) + + def redraw(self, *, keep_cursor: str | None = None) -> None: + table = self.table + current = keep_cursor or self.cursor_key() + table.clear() + for r in self._ordered(): + mark = "* " if r.key in self.marks else " " + name = Text(mark + safe(r.name) + ("/" if r.is_dir and r.key != PARENT else ""), + style="dim" if r.dim else ("bold" if r.key in self.marks else ""), + no_wrap=True, overflow="ellipsis") + if r.note: + name.append(f" ({r.note})", style="italic dim") + size = Text("" if r.is_dir or r.size is None else human_size(r.size), + justify="right") + cells = [name, size] + if self.show_date: + cells.append(Text(date(int(r.when)) if r.when else "")) + table.add_row(*cells, key=r.key) + keys = [r.key for r in self._ordered()] + if current in keys: + table.move_cursor(row=keys.index(current)) + + def cursor_key(self) -> str | None: + table = self.table + if table.row_count == 0: + return None + try: + return table.coordinate_to_cell_key(table.cursor_coordinate).row_key.value + except Exception: + return None + + def row(self, key: str | None) -> Row | None: + return next((r for r in self.rows if r.key == key), None) + + def selection(self) -> list[Row]: + """The marked rows, or the one under the cursor.""" + if self.marks: + return [r for r in self.rows if r.key in self.marks] + r = self.row(self.cursor_key()) + return [r] if r and r.key != PARENT else [] + + # ── Actions ────────────────────────────────────────────────────────────── + + def action_mark(self) -> None: + key = self.cursor_key() + if key is None or key == PARENT: + return + self.marks ^= {key} + row = self.table.cursor_row + self.redraw(keep_cursor=key) + self.table.move_cursor(row=min(row + 1, self.table.row_count - 1)) + + def action_sort(self) -> None: + self.sort = SORTS[(SORTS.index(self.sort) + 1) % len(SORTS)] + self.app.notify(f"Sorted by {self.sort}", timeout=2) + self.redraw() + + def action_filter(self) -> None: + box = self.query_one(Input) + box.add_class("shown") + box.value = self.filter_text + box.focus() + + def on_input_changed(self, event: Input.Changed) -> None: + self.filter_text = event.value + self.redraw() + + def on_input_submitted(self, event: Input.Submitted) -> None: + self.table.focus() + + def key_escape(self) -> None: + box = self.query_one(Input) + if box.has_class("shown"): + box.value = "" + box.remove_class("shown") + self.filter_text = "" + self.redraw() + self.table.focus() + + def on_data_table_row_selected(self, event: DataTable.RowSelected) -> None: + row = self.row(event.row_key.value) + if row is None: + return + if row.key == PARENT: + self.action_up() + elif row.is_dir: + self.enter(row) + + # Subclasses + def enter(self, row: Row) -> None: ... + def action_up(self) -> None: ... + def action_refresh(self) -> None: ... + + +class LocalPane(Pane): + def __init__(self, start: Path, **kwargs): + super().__init__(**kwargs) + self.path = start.resolve() + + def on_mount(self) -> None: + self.load() + + def load(self, keep: str | None = None) -> None: + self.set_title(f"Local: {self.path}") + self.run_worker(self._load(keep), exclusive=True, group="local") + + async def _load(self, keep: str | None) -> None: + try: + rows = await asyncio.to_thread(_scan, self.path) + except OSError as e: + self.app.notify(f"{self.path}: {e.strerror}", severity="error", markup=False) + rows = [] + if self.path.parent != self.path: + rows.insert(0, Row(PARENT, PARENT, True)) + self.show(rows, keep_cursor=keep) + + def enter(self, row: Row) -> None: + self.path = self.path / row.name + self.marks.clear() + self.load() + + def action_up(self) -> None: + if self.path.parent != self.path: + came_from = self.path.name + self.path = self.path.parent + self.marks.clear() + self.load(keep=came_from) + + def action_refresh(self) -> None: + # No position captured now: the cursor may move while the folder is read, + # and the redraw keeps wherever it is when the rows arrive. + self.load() + + +def _scan(path: Path) -> list[Row]: + rows = [] + with os.scandir(path) as it: + for e in it: + try: + st = e.stat() + except OSError: + continue + is_dir = stat.S_ISDIR(st.st_mode) + if not is_dir and not stat.S_ISREG(st.st_mode): + continue + rows.append(Row(e.name, e.name, is_dir, None if is_dir else st.st_size, + st.st_mtime)) + return rows + + +class RemotePane(Pane): + """Groups, then a group's roots and folders — the Files tab, never merged.""" + + def __init__(self, screen, **kwargs): + super().__init__(**kwargs) + self.main = screen + self.groups: list[dict] = [] + self.group: dict | None = None # None: the list of groups + self.path = "" # inside the group: root/folders + + def on_mount(self) -> None: + self.load() + + @property + def location(self) -> str: + if self.group is None: + return "Your groups" + return f"{handle(self.group)}/{self.path}".rstrip("/") + + def load(self, keep: str | None = None) -> None: + self.set_title(f"{safe(self.location)} (loading…)") + self.run_worker(self._load(keep), exclusive=True, group="remote") + + async def _load(self, keep: str | None) -> None: + try: + if self.group is None: + self.groups = await my_groups(self.main.account) + rows = [Row(g["id"], handle(g), True, + note="" if g.get("node_online") else "down", + dim=not g.get("node_online"), data=g) + for g in self.groups] + else: + tree = await self.main.tree(self.group) + rows = [Row(PARENT, PARENT, True)] + _tree_rows(tree, self.path) + except CliError as e: + self.set_title(safe(self.location)) + self.app.notify(e.message, severity="error", markup=False, timeout=8) + if self.group is not None: + self.group, self.path = None, "" + self.load() + return + self.set_title(safe(self.location)) + self.show(rows, keep_cursor=keep) + + def enter(self, row: Row) -> None: + self.marks.clear() + if self.group is None: + self.group, self.path = row.data, "" + else: + self.path = row.data["path"] + self.load() + + def action_up(self) -> None: + if self.group is None: + return + self.marks.clear() + if not self.path: + came_from, self.group = self.group["id"], None + else: + came_from = self.path + self.path = self.path.rpartition("/")[0] + self.load(keep=came_from) + + def action_refresh(self) -> None: + if self.group is not None: + self.main.forget_tree(self.group) + self.load() + + +def _tree_rows(tree: Tree, path: str) -> list[Row]: + rows = [] + for item in tree.children(path): + note = "" + if item.root is not None: + note = "" if item.root.get("writable") else "read-only" + if not item.available: + note = "unavailable" + rows.append(Row(item.path, item.name, item.is_dir, + None if item.is_dir else item.size, + item.added_at or None, note=note, dim=not item.available, + data={"path": item.path, "entry": item.entry})) + return rows diff --git a/packages/meshbay-cli/src/meshbay_cli/tui/screens.py b/packages/meshbay-cli/src/meshbay_cli/tui/screens.py new file mode 100644 index 0000000..9d19431 --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/tui/screens.py @@ -0,0 +1,354 @@ +""" +The sign-in screen, and the main screen: two panes and the transfers. + +F5 copies the selection to the other pane's folder — a download or an upload, +with the rules `get` and `put` follow: an upload is decided whole before it +starts (names the node refuses stop it, names already there are asked about +once for the batch), nothing on either side is replaced without being asked, +and a name from a group is made writable here before it is written. +""" + +import asyncio +import contextlib +import threading +from pathlib import Path + +from rich.text import Text +from textual.app import ComposeResult +from textual.binding import Binding +from textual.containers import Horizontal, Vertical +from textual.screen import Screen +from textual.widgets import Button, Footer, Input, Static + +from ..client.account import Account +from ..client.names import UnsafeName, local_path +from ..client.paths import handle +from ..client.pool import Pool +from ..client.store import FILE_KEY_NOTICE, KEY_FILE +from ..client.transfers import PART_SUFFIX, TransferError, create_dir, download, upload +from ..client.tree import Tree +from ..client.upload_plan import plan_upload, refused_names, taken_names +from ..errors import CliError +from ..output import safe +from .panes import LocalPane, RemotePane +from .widgets import Choice, TransferStrip + +PER_GROUP = 2 + + +class SignInScreen(Screen): + DEFAULT_CSS = """ + SignInScreen { align: center middle; } + SignInScreen > Vertical { width: 64; height: auto; border: thick $accent; + padding: 1 2; background: $surface; } + SignInScreen Input { margin-bottom: 1; } + SignInScreen #error { color: $error; height: auto; } + """ + + def compose(self) -> ComposeResult: + ctx = self.app.ctx + with Vertical(): + yield Static(Text("Sign in to MeshBay")) + yield Input(value=ctx.hub_url, placeholder="hub", id="hub") + yield Input(value=ctx.username or ctx.stored_username or "", + placeholder="username", id="user") + yield Input(placeholder="passphrase", password=True, id="pass") + yield Button("Sign in", id="go", variant="primary") + yield Static("", id="error", markup=False) + + def on_mount(self) -> None: + empty = next((i for i in self.query(Input) if not i.value), None) + (empty or self.query_one("#pass", Input)).focus() + + def on_input_submitted(self, event: Input.Submitted) -> None: + self._submit() + + def on_button_pressed(self, event: Button.Pressed) -> None: + self._submit() + + def _submit(self) -> None: + hub = self.query_one("#hub", Input).value.strip().rstrip("/") + user = self.query_one("#user", Input).value.strip() + passphrase = self.query_one("#pass", Input).value + if not (hub and user and passphrase): + self.query_one("#error", Static).update("Hub, username and passphrase.") + return + self.query_one("#error", Static).update("Signing in…") + self.run_worker(self._sign_in(hub, user, passphrase), exclusive=True) + + async def _sign_in(self, hub_url: str, user: str, passphrase: str) -> None: + app = self.app + try: + await app.use_hub(hub_url) + account = Account(app.hub, user, lambda: passphrase) + await account.hub.check_version() + await account.sign_in(remember=True) + kind = await account.remember(app.ctx.store) + except CliError as e: + self.query_one("#error", Static).update(e.message) + return + if kind == KEY_FILE: + app.notify(FILE_KEY_NOTICE, timeout=10, markup=False) + app.start(account) + + +class MainScreen(Screen): + BINDINGS = [ + Binding("f5", "copy", "Copy"), + Binding("q", "app.quit", "Quit"), + Binding("tab", "app.focus_next", "Switch", show=False), + ] + + DEFAULT_CSS = """ + MainScreen > Horizontal { height: 1fr; } + """ + + def __init__(self, account: Account): + super().__init__() + self.account = account + self.pool = Pool(account) + self._trees: dict[str, Tree] = {} + self._gates: dict[str, asyncio.Semaphore] = {} + self._watching: set[str] = set() + + def compose(self) -> ComposeResult: + with Horizontal(): + yield LocalPane(Path.cwd(), id="local") + yield RemotePane(self, id="remote") + yield TransferStrip(id="transfers") + yield Footer() + + def on_mount(self) -> None: + self.query_one(LocalPane).table.focus() + + @property + def local(self) -> LocalPane: + return self.query_one(LocalPane) + + @property + def remote(self) -> RemotePane: + return self.query_one(RemotePane) + + @property + def strip(self) -> TransferStrip: + return self.query_one(TransferStrip) + + # ── Groups ─────────────────────────────────────────────────────────────── + + async def tree(self, group: dict) -> Tree: + """The group's index, asked once and then kept current by the node's pushes.""" + if group["id"] not in self._trees: + session = await self.pool.session(group) + self._trees[group["id"]] = Tree.from_index(await session.index()) + if group["id"] not in self._watching: + self._watching.add(group["id"]) + session.ch.on_push("index_delta", lambda _msg, g=group: self._changed(g)) + return self._trees[group["id"]] + + def forget_tree(self, group: dict) -> None: + self._trees.pop(group["id"], None) + + def _changed(self, group: dict) -> None: + """Something changed in the group: the index is asked for again, and the + pane redrawn if it is showing that group.""" + self.forget_tree(group) + remote = self.remote + if remote.group is not None and remote.group["id"] == group["id"]: + remote.load() + + # ── F5 ─────────────────────────────────────────────────────────────────── + + def action_copy(self) -> None: + focused = self.app.focused + if focused is not None and self.remote in focused.ancestors_with_self: + self.run_worker(self._download(), group="copy") + else: + self.run_worker(self._upload(), group="copy") + + def _gate(self, group: dict) -> asyncio.Semaphore: + return self._gates.setdefault(group["id"], asyncio.Semaphore(PER_GROUP)) + + async def _ask(self, message: str, options: list[tuple[str, str]]) -> str: + return await self.app.push_screen(Choice(message, options), wait_for_dismiss=True) + + async def _download(self) -> None: + remote, local = self.remote, self.local + rows = remote.selection() + if remote.group is None or not rows: + self.app.notify("Open a group and choose what to download.", timeout=4) + return + group, dest = remote.group, local.path + try: + tree = await self.tree(group) + except CliError as e: + self.app.notify(e.message, severity="error", markup=False) + return + plan: list[tuple[dict, Path]] = [] + dirs: list[Path] = [] + problems: list[str] = [] + for row in rows: + path = row.data["path"] + top = path.rsplit("/", 1)[-1] + if not tree.root_available(path): + problems.append(f"{top}: its folder on the node is unavailable") + continue + if not row.is_dir: + try: + plan.append((row.data["entry"], local_path(dest, [top])[0])) + except UnsafeName as e: + problems.append(str(e)) + continue + for folder, items in tree.walk(path): + rel = [top] + [p for p in folder[len(path):].strip("/").split("/") if p] + try: + dirs.append(local_path(dest, rel)[0]) + plan += [(i.entry, local_path(dest, rel + [i.name])[0]) + for i in items if not i.is_dir] + except UnsafeName as e: + problems.append(str(e)) + for p in problems: + self.app.notify(p, severity="warning", markup=False) + existing = [t for _, t in plan if t.exists() + and not t.with_name(t.name + PART_SUFFIX).exists()] + overwrite = False + if existing: + names = ", ".join(safe(t.name) for t in existing[:5]) + ( + f" and {len(existing) - 5} more" if len(existing) > 5 else "") + answer = await self._ask( + f"{len(existing)} file(s) already in {dest}: {names}.", + [("skip", "Skip them"), ("overwrite", "Overwrite"), ("cancel", "Cancel")]) + if answer == "cancel": + return + overwrite = answer == "overwrite" + if not overwrite: + plan = [(e, t) for e, t in plan if t not in existing] + for d in dirs: + await asyncio.to_thread(d.mkdir, parents=True, exist_ok=True) + remote.marks.clear() + remote.redraw() + session = await self.pool.session(group) + await asyncio.gather(*(self._one_download(group, session, e, t, overwrite) + for e, t in plan)) + local.action_refresh() + + async def _one_download(self, group, session, entry: dict, target: Path, + overwrite: bool) -> None: + job = self.strip.add("↓", entry["name"], int(entry["size"])) + async with self._gate(group): + self.strip.update_job(job, state="running") + try: + await download( + session, entry, target, overwrite=overwrite, + progress=lambda n: self.strip.update_job(job, done=job.done + n), + on_queued=lambda n: self.strip.update_job(job, state="queued", + note=f"({n} ahead)")) + self.strip.update_job(job, state="done", + note="" if target.name == entry["name"] + else f"saved as {target.name}") + except TransferError as e: + self.strip.update_job(job, state="failed", note=e.message) + + async def _upload(self) -> None: + remote, local = self.remote, self.local + rows = local.selection() + if not rows: + return + if remote.group is None or not remote.path: + self.app.notify("Open a folder of a group on the right to upload into.", + timeout=4) + return + group, folder = remote.group, remote.path + try: + tree = await self.tree(group) + except CliError as e: + self.app.notify(e.message, severity="error", markup=False) + return + root = tree.roots.get(folder.split("/", 1)[0], {}) + if not root.get("writable") or not root.get("available", True): + self.app.notify(f"{safe(handle(group))}/{safe(folder)} does not accept " + "uploads (read-only or unavailable).", severity="error", + markup=False) + return + try: + dirs, files, problems = plan_upload([str(local.path / r.name) for r in rows], + recursive=True) + except CliError as e: + self.app.notify(e.message, severity="error", markup=False) + return + problems += refused_names([f.rel[-1] for f in files], dirs) + if problems: + await self._ask("Nothing was sent:\n" + "\n".join(safe(p) for p in problems), + [("ok", "OK")]) + return + tops = sorted({d[0] for d in dirs} | {f.rel[0] for f in files}) + taken = taken_names(tree, folder, tops) + if taken: + answer = await self._ask( + f"Already in {safe(folder)}: {', '.join(safe(t) for t in sorted(taken))}. " + "The node never replaces a file.", + [("skip", "Skip them"), ("cancel", "Cancel")]) + if answer != "skip": + return + dirs = [d for d in dirs if d[0] not in taken] + files = [f for f in files if f.rel[0] not in taken] + local.marks.clear() + local.redraw() + session = await self.pool.session(group) + remote_dir = {(): folder} + try: + for d in dirs: + remote_dir[tuple(d)] = await create_dir(session, remote_dir[tuple(d[:-1])], + d[-1]) + except TransferError as e: + self.app.notify(e.message, severity="error", markup=False) + return + await asyncio.gather(*(self._one_upload(group, session, f, + remote_dir[tuple(f.rel[:-1])]) + for f in files)) + self._changed(group) + + async def _one_upload(self, group, session, item, folder: str) -> None: + name = item.rel[-1] + job = self.strip.add("↑", name, item.size) + async with self._gate(group): + self.strip.update_job(job, state="running") + lock = threading.Lock() + try: + with open(item.local, "rb") as fh: + def read(offset: int, length: int) -> bytes: + with lock: + fh.seek(offset) + return fh.read(length) + stored = await upload( + session, read, item.size, name, folder, + progress=lambda n: self.strip.update_job(job, done=job.done + n), + on_queued=lambda n: self.strip.update_job(job, state="queued", + note=f"({n} ahead)")) + self.strip.update_job(job, state="done", note="" if stored == name + else f"stored as {stored}") + except OSError as e: + self.strip.update_job(job, state="failed", note=e.strerror or str(e)) + except TransferError as e: + self.strip.update_job(job, state="failed", note=e.message) + + async def on_unmount(self) -> None: + with contextlib.suppress(Exception): + await self.pool.close() + + +class DeviceCode: + """`account.on_device_code` for the interactive mode: the code in a + notification that stays until the approval lands; the node is retried meanwhile.""" + + def __init__(self, app): + self.app = app + + def __call__(self, group: dict, pending, *, first: bool) -> bool: + if first: + self.app.notify( + f"This machine is not yet a device of yours on the node hosting " + f"{safe(handle(group))}. Approve it from the MeshBay application or a " + f"browser: the group's Settings → \"Your devices on this node\" → " + f"enter {pending.link_code}. Waiting…", + title="Approve this machine", timeout=600, markup=False) + return True diff --git a/packages/meshbay-cli/src/meshbay_cli/tui/widgets.py b/packages/meshbay-cli/src/meshbay_cli/tui/widgets.py new file mode 100644 index 0000000..66fca4e --- /dev/null +++ b/packages/meshbay-cli/src/meshbay_cli/tui/widgets.py @@ -0,0 +1,109 @@ +"""A question with a few answers, and the strip that shows transfers.""" + +import time +from dataclasses import dataclass + +from rich.text import Text +from textual.app import ComposeResult +from textual.containers import Horizontal, Vertical +from textual.screen import ModalScreen +from textual.widgets import Button, Static + +from ..output import human_size, safe + + +class Choice(ModalScreen[str]): + """One question, asked once for a whole batch — never once per file.""" + + DEFAULT_CSS = """ + Choice { align: center middle; } + Choice > Vertical { width: 72; height: auto; max-height: 80%; + border: thick $accent; background: $surface; padding: 1 2; } + Choice Static { height: auto; margin-bottom: 1; } + Choice Horizontal { height: auto; align: center middle; } + Choice Button { margin: 0 1; } + """ + BINDINGS = [("escape", "dismiss('cancel')", "Cancel")] + + def __init__(self, message: str, options: list[tuple[str, str]]): + super().__init__() + self.message = message + self.options = options + + def compose(self) -> ComposeResult: + with Vertical(): + yield Static(Text(self.message)) + with Horizontal(): + for value, label in self.options: + yield Button(label, id=f"choice-{value}") + + def on_mount(self) -> None: + self.query(Button).first().focus() + + def on_button_pressed(self, event: Button.Pressed) -> None: + self.dismiss(event.button.id.removeprefix("choice-")) + + +@dataclass +class Job: + arrow: str + name: str + size: int + done: int = 0 + state: str = "waiting" # waiting, queued, running, done, failed + note: str = "" + started: float = 0.0 + + +class TransferStrip(Static): + """The running transfers, then the latest finished ones.""" + + DEFAULT_CSS = """ + TransferStrip { height: 6; border-top: solid $panel; padding: 0 1; } + """ + SHOWN = 5 + + def __init__(self, **kwargs): + super().__init__("", markup=False, **kwargs) + self.jobs: list[Job] = [] + + def on_mount(self) -> None: + self.refresh_text() + + def add(self, arrow: str, name: str, size: int) -> Job: + job = Job(arrow, name, size) + self.jobs.append(job) + self.refresh_text() + return job + + def update_job(self, job: Job, **fields) -> None: + for k, v in fields.items(): + setattr(job, k, v) + if job.state == "running" and not job.started: + job.started = time.monotonic() + self.refresh_text() + + def refresh_text(self) -> None: + active = [j for j in self.jobs if j.state in ("running", "queued", "waiting")] + ended = [j for j in self.jobs if j.state in ("done", "failed")] + lines = [self._line(j) for j in active[:self.SHOWN]] + if len(active) > self.SHOWN: + lines.append(f" … and {len(active) - self.SHOWN} more waiting") + for j in reversed(ended[-(self.SHOWN - len(lines)):] if len(lines) < self.SHOWN + else []): + lines.append(self._line(j)) + self.update(Text("\n".join(lines) or "No transfers.")) + + def _line(self, j: Job) -> str: + name = safe(j.name) + if j.state == "running": + pct = 100 * j.done // j.size if j.size else 100 + rate = j.done / max(time.monotonic() - j.started, 0.001) + return f"{j.arrow} {name} {pct}% {human_size(int(rate))}/s" + if j.state == "queued": + return f"{j.arrow} {name} waiting for a free slot on the node {j.note}" + if j.state == "waiting": + return f"{j.arrow} {name} waiting" + if j.state == "done": + return f"{j.arrow} {name} done{(' — ' + j.note) if j.note else ''}" + return f"{j.arrow} {name} FAILED: {j.note}" diff --git a/packages/meshbay-cli/tests/conftest.py b/packages/meshbay-cli/tests/conftest.py new file mode 100644 index 0000000..8ef9189 --- /dev/null +++ b/packages/meshbay-cli/tests/conftest.py @@ -0,0 +1,49 @@ +""" +Every test runs with its own home, and never touches the developer's own. + +Redirecting HOME alone isolates nothing on Windows — `Path.home()` reads +USERPROFILE there, and the CLI's directories come from LOCALAPPDATA — so all of +them are pointed at a throwaway directory. The OS key storage is replaced by +keyring's refusing backend: a test must never write into the developer's real +credential store, and the store's file fallback is what a headless machine uses +anyway. A test that needs a working key storage installs its own backend. +""" + +import logging + +import pytest + + +@pytest.fixture(autouse=True) +def isolated_home(tmp_path, monkeypatch): + home = tmp_path / "home" + home.mkdir() + for var in ("HOME", "USERPROFILE"): + monkeypatch.setenv(var, str(home)) + for var, sub in (("APPDATA", "AppData/Roaming"), ("LOCALAPPDATA", "AppData/Local"), + ("XDG_CONFIG_HOME", ".config"), ("XDG_DATA_HOME", ".local/share")): + monkeypatch.setenv(var, str(home / sub)) + for var in ("MESHBAY_CLI_HOME", "MESHBAY_HUB", "MESHBAY_USER", "MESHBAY_PASSWORD"): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv("PYTHON_KEYRING_BACKEND", "keyring.backends.fail.Keyring") + return home + + +@pytest.fixture(autouse=True) +def restored_logging(): + """`main()` configures logging, as an entry point does; a test that calls it + in-process must not leave that configuration to the next package's tests.""" + root = logging.getLogger() + names = ("meshbay_cli", "aiortc", "aioice", "httpx", "httpcore") + saved = (list(root.handlers), root.level, root.disabled, + {n: logging.getLogger(n).level for n in names}) + yield + handlers, level, disabled, levels = saved + for h in root.handlers: + if h not in handlers: + h.close() + root.handlers[:] = handlers + root.setLevel(level) + root.disabled = disabled + for n, lv in levels.items(): + logging.getLogger(n).setLevel(lv) diff --git a/packages/meshbay-cli/tests/fake_hub.py b/packages/meshbay-cli/tests/fake_hub.py new file mode 100644 index 0000000..ec1a0c0 --- /dev/null +++ b/packages/meshbay-cli/tests/fake_hub.py @@ -0,0 +1,108 @@ +""" +A hub that keeps the rules the CLI depends on, in memory. + +Its refresh tokens **rotate, and a spent one presented again revokes the +whole family**, as the real hub does. A lax stub would pass a client that +renews twice with the same token — which is exactly the failure two runs at +once produce. +""" + +import asyncio +import base64 +import secrets + +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey +from meshbay_cli.client.hub import HubError +from meshbay_cli.client.keys import auth_key + +USERNAME = "alice-test" +PASSPHRASE = "a passphrase for the fake hub" +USER_ID = "0b4f6f0e-5d7e-4e8a-9d2b-6a1c1b9e2f11" +PEPPER = base64.b64encode(b"\x07" * 32).decode() + + +class FakeHub: + def __init__(self, url: str = "https://hub.test"): + self.url = url + self.ak = auth_key(USERNAME, PASSPHRASE) + self.refresh_tokens: dict[str, dict] = {} # token → {family, spent} + self.revoked_families: set[str] = set() + self.access: set[str] = set() + self.devices: dict[str, str] = {} # id → pk + self.refresh_calls = 0 + self.reuse_detected = False + self.refresh_delay = 0.0 + self.groups = [{"id": "g1", "name": "Family", "owner_username": USERNAME, + "node_online": True, "visibility": "private", "is_admin": True}] + + def _session(self, family: str | None = None) -> dict: + family = family or secrets.token_hex(4) + rt, at = secrets.token_hex(8), secrets.token_hex(8) + self.refresh_tokens[rt] = {"family": family, "spent": False} + self.access.add(at) + return {"access_token": at, "refresh_token": rt, "expires_in": 14400} + + def _auth(self, token: str) -> None: + if token not in self.access: + raise HubError(401, "Invalid token") + + async def aclose(self) -> None: + pass + + async def check_version(self) -> None: + return None + + async def login(self, username, ak, known_browser): + from meshbay_cli.client.hub import _sign_in_refusal + if username != USERNAME or ak != self.ak: + raise _sign_in_refusal(HubError(401, "Invalid credentials")) + return {**self._session(), "bundle_pepper": PEPPER, "bundle_pepper_version": 1, + "known_browser": "kb-1"} + + async def me(self, token): + self._auth(token) + return {"user_id": USER_ID, "username": USERNAME} + + async def register_device(self, token, pk, label, ak): + self._auth(token) + assert ak == self.ak, "registering a device takes the passphrase" + for did, known in self.devices.items(): + if known == pk: + return {"id": did, "existing": True} + did = f"dev-{len(self.devices) + 1}" + self.devices[did] = pk + return {"id": did, "existing": False} + + async def delete_device(self, token, device_id): + self._auth(token) + self.devices.pop(device_id) + + async def device_auth(self, username, sk): + pk = base64.b64encode(sk.public_key().public_bytes_raw()).decode() + if pk not in self.devices.values(): + from meshbay_cli.errors import AUTH, CliError + raise CliError(AUTH, "Sign-in refused") + Ed25519PublicKey.from_public_bytes(sk.public_key().public_bytes_raw()) + return {**self._session(), "bundle_pepper": PEPPER} + + async def refresh(self, rt): + self.refresh_calls += 1 + row = self.refresh_tokens.get(rt) + if row is None or row["family"] in self.revoked_families: + raise HubError(401, "Invalid refresh token") + if row["spent"]: + self.reuse_detected = True + self.revoked_families.add(row["family"]) + raise HubError(401, "Token reuse detected — family revoked") + row["spent"] = True + await asyncio.sleep(self.refresh_delay) + return self._session(row["family"]) + + async def logout(self, rt): + row = self.refresh_tokens.get(rt) + if row: + self.revoked_families.add(row["family"]) + + async def my_groups(self, token): + self._auth(token) + return self.groups diff --git a/packages/meshbay-cli/tests/fake_node.py b/packages/meshbay-cli/tests/fake_node.py new file mode 100644 index 0000000..e733c56 --- /dev/null +++ b/packages/meshbay-cli/tests/fake_node.py @@ -0,0 +1,166 @@ +""" +A node in memory, at the level of the MNP messages the CLI sends. + +It does the real cryptography — chunks encrypted under the per-chunk key, +uploads unsealed under the group key, acks sealed back — and keeps the rules +that shape the client: a lease before any chunk, chunks of an upload in order, +the position an interrupted upload resumes from, no overwrite. Not a model of +the node's code: a node the CLI must work against. +""" + +import asyncio +import hashlib +import math +import secrets + +from meshbay_cli.client.session import GroupSession, NodeRefusal +from meshbay_common.groupbox import PURPOSE_INDEX, PURPOSE_UPLOAD, seal, unseal +from meshbay_common.protocol import UPLOAD_PROBE_INDEX, file_upload_ack_wire +from meshbay_common.webcrypto import chunk_key_aes, encrypt_chunk_aes + +GROUP_ID = "g1" +CHUNK = 1024 * 1024 + + +class FakeNode: + def __init__(self, *, writable: bool = True): + self.gek = secrets.token_bytes(32) + self.roots = [{"name": "share", "available": True, "writable": writable}] + self.dirs = {"share"} + self.files: dict[str, dict] = {} # id → entry, with "data" + self.leases: dict[str, str] = {} # tr → state + self.queue_first = 0 # leases to answer "queued" before granting + self.uploads: dict[tuple, dict] = {} # (dir, name) → {"next", "data"} + self.fail_chunk: tuple[str, int] | None = None # (file_id, index) → refused + self.refuse_upload_code = "" + self.chunk_requests = 0 + self.chunk_delay = 0.0 + + def add(self, path: str, name: str, data: bytes) -> dict: + fid = hashlib.blake2b(data + path.encode() + name.encode(), digest_size=32).hexdigest() + entry = {"id": fid, "name": name, "path": path, "size": len(data), + "type": "document", "added_at": 1_700_000_000, "data": data} + self.files[fid] = entry + parts = path.split("/") + for i in range(1, len(parts) + 1): + self.dirs.add("/".join(parts[:i])) + return entry + + def session(self) -> GroupSession: + return GroupSession(None, FakeChannel(self), GROUP_ID, "node-1", "pk", self.gek, {}) + + def stored(self, path: str) -> bytes | None: + parent, _, name = path.rpartition("/") + for e in self.files.values(): + if e["path"] == parent and e["name"] == name: + return e["data"] + return None + + +class FakeChannel: + def __init__(self, node: FakeNode): + self.node = node + self.closed = asyncio.Event() + self._push: dict[str, list] = {} + + def on_push(self, t, h): + self._push.setdefault(t, []).append(h) + + def off_push(self, t, h): + if h in self._push.get(t, []): + self._push[t].remove(h) + + def _emit(self, msg): + for h in list(self._push.get(msg["type"], [])): + h(msg) + + async def request(self, msg: dict, *expect: str, timeout: float = 60) -> dict: + await asyncio.sleep(self.node.chunk_delay if msg["type"] == "file_req" else 0) + reply = getattr(self, "_" + msg["type"])(msg) + if reply.get("type") == "error": + raise NodeRefusal(reply["detail"], reply.get("code", "")) + return reply + + # ── Messages ───────────────────────────────────────────────────────────── + + def _index_sync(self, msg): + n = self.node + payload = {"version": 1, "roots": n.roots, "dirs": sorted(n.dirs), + "entries": [{k: v for k, v in e.items() if k != "data"} + for e in n.files.values()]} + return {"type": "index_sync", "group_id": GROUP_ID, + **seal(n.gek, PURPOSE_INDEX, "index_sync", GROUP_ID, payload)} + + def _transfer_open(self, msg): + tr = msg["tr"] + if self.node.queue_first: + self.node.queue_first -= 1 + self.node.leases[tr] = "queued" + asyncio.get_running_loop().call_later(0.01, self._grant, tr) + return {"type": "transfer_state", "tr": tr, "state": "queued", "ahead": 1} + self.node.leases[tr] = "granted" + return {"type": "transfer_state", "tr": tr, "state": "granted"} + + def _grant(self, tr): + self.node.leases[tr] = "granted" + self._emit({"type": "transfer_state", "tr": tr, "state": "granted"}) + + def _transfer_close(self, msg): + self.node.leases.pop(msg["tr"], None) + return {"type": "transfer_state", "tr": msg["tr"], "state": "closed"} + + def _file_req(self, msg): + n = self.node + n.chunk_requests += 1 + if n.leases.get(msg.get("tr")) != "granted": + return {"type": "error", "detail": "no lease", "code": "lease_not_granted"} + if n.fail_chunk == (msg["file_id"], msg["chunk_index"]): + return {"type": "error", "detail": "The drive stopped answering"} + entry = n.files[msg["file_id"]] + i = msg["chunk_index"] + data = entry["data"][i * CHUNK:(i + 1) * CHUNK] + nonce, ct = encrypt_chunk_aes(chunk_key_aes(n.gek, bytes.fromhex(entry["id"]), i), data) + return {"type": "file_chunk", "chunk_index": i, "nonce": nonce, "ct": ct} + + def _dir_create(self, msg): + root = self.node.roots[0] + if not root["writable"]: + return {"type": "error", "detail": "read-only", "code": "root_read_only"} + path = f"{msg['dir']}/{msg['name']}" + self.node.dirs.add(path) + return {"type": "dir_create_ack", "dir": path} + + def _file_upload(self, msg): + n = self.node + if n.leases.get(msg.get("tr")) != "granted": + return {"type": "error", "detail": "no lease", "code": "lease_not_granted"} + p = unseal(n.gek, PURPOSE_UPLOAD, "file_upload", GROUP_ID, msg) + name, folder, i = p["filename"], p["dir"], msg["chunk_index"] + if n.refuse_upload_code: + return {"type": "error", "detail": "refused", "code": n.refuse_upload_code} + key = (folder, name) + state = n.uploads.get(key) + stored = name + while n.stored(f"{folder}/{stored}") is not None: + stem, dot, ext = name.partition(".") + stored = f"{stem} (1){dot}{ext}" + if i == UPLOAD_PROBE_INDEX: + return file_upload_ack_wire(n.gek, GROUP_ID, upload_id=msg["upload_id"], + chunk_index=i, filename=name, stored_as="", + dir=folder, resume_from=state["next"] if state else 0) + if state is None: + state = n.uploads[key] = {"next": 0, "data": b""} + if i != state["next"]: + return {"type": "error", "detail": "Out of order", "code": "bad_chunk_index"} + state["data"] += p["data"] + state["next"] += 1 + if state["next"] >= msg["total_chunks"]: + del n.uploads[key] + n.add(folder, stored, state["data"]) + return file_upload_ack_wire(n.gek, GROUP_ID, upload_id=msg["upload_id"], + chunk_index=i, filename=name, stored_as=stored, + dir=folder) + + +def chunks_of(size: int, chunk: int) -> int: + return max(1, math.ceil(size / chunk)) diff --git a/packages/meshbay-cli/tests/test_cli_actions.py b/packages/meshbay-cli/tests/test_cli_actions.py new file mode 100644 index 0000000..b88d93c --- /dev/null +++ b/packages/meshbay-cli/tests/test_cli_actions.py @@ -0,0 +1,238 @@ +""" +Every action, run through `main()` with the hub and the node stubbed. + +A branch nobody has run is covered by nothing — the node's `reload` once shipped +with a missing import. `test_every_action_is_run_here` refuses an action added +to the parser without a run below. +""" + +import io +import json + +import pytest +from fake_hub import PASSPHRASE, USERNAME, FakeHub +from meshbay_cli import __main__ as cli +from meshbay_cli.client import pool +from meshbay_cli.commands import context + +RUN: set[str] = set() + + +@pytest.fixture +def hub(monkeypatch): + fake = FakeHub() + monkeypatch.setattr(context, "Hub", lambda url: fake) + return fake + + +def main(*argv: str) -> int: + RUN.add(next(a for a in argv if a in cli.ACTIONS)) + return cli.main(list(argv)) + + +class FakeSession: + async def index(self): + return {"roots": [{"name": "share", "available": True, "writable": True}], + "dirs": ["share"], + "entries": [{"id": "a" * 64, "name": "notes.txt", "path": "share", + "size": 6, "type": "document", "added_at": 0}]} + + async def close(self): + pass + + +def test_login_ls_logout(hub, monkeypatch, capsys): + monkeypatch.setattr(pool, "open_group", lambda account, group: _session()) + + assert main("--hub", hub.url, "-u", USERNAME, "--password", PASSPHRASE, "login") == 0 + assert "Signed in: alice-test" in capsys.readouterr().out + + assert main("ls") == 0 + assert capsys.readouterr().out.strip() == f"Family@{USERNAME}" + + assert main("ls", "--json", "Family/share") == 0 + [item] = json.loads(capsys.readouterr().out) + assert item["path"] == f"Family@{USERNAME}/share/notes.txt" and item["kind"] == "file" + + assert main("logout") == 0 + assert "Signed out." in capsys.readouterr().out + assert not hub.devices + + assert main("ls") == 2 + assert "Not signed in" in capsys.readouterr().err + + +def test_login_without_remembering_stores_nothing(hub, isolated_home, capsys): + assert main("--hub", hub.url, "-u", USERNAME, "login", "--no-remember", + "--password", PASSPHRASE) == 0 + assert "Credentials accepted" in capsys.readouterr().out + assert not hub.devices + assert not any(isolated_home.rglob("secrets.enc")) + + +def test_a_wrong_passphrase_exits_2(hub, capsys): + assert main("--hub", hub.url, "-u", USERNAME, "--password", "nope", "login") == 2 + assert "wrong username or passphrase" in capsys.readouterr().err + + +def test_no_passphrase_and_no_terminal_exits_2(hub, monkeypatch, capsys): + monkeypatch.setattr("sys.stdin.isatty", lambda: False) + assert main("--hub", hub.url, "-u", USERNAME, "login") == 2 + assert "MESHBAY_PASSWORD" in capsys.readouterr().err + + +async def _session(): + return FakeSession() + + +@pytest.fixture +def node(hub, monkeypatch, capsys): + from fake_node import FakeNode + fake = FakeNode() + + async def open_group(account, group): + return fake.session() + monkeypatch.setattr(pool, "open_group", open_group) + assert main("--hub", hub.url, "-u", USERNAME, "--password", PASSPHRASE, "login") == 0 + capsys.readouterr() + return fake + + +def test_get_files_folders_and_patterns(node, tmp_path, monkeypatch, capsys): + node.add("share", "notes.txt", b"hello") + node.add("share/2024", "a.jpg", b"A" * 3000) + node.add("share/2024", "b.jpg", b"B" * 10) + node.add("share/2024/deep", "c.txt", b"C") + node.dirs.add("share/2024/empty") + monkeypatch.chdir(tmp_path) + + assert main("get", "Family/share/notes.txt") == 0 + assert (tmp_path / "notes.txt").read_bytes() == b"hello" + assert "Downloaded 1 file" in capsys.readouterr().err + + # Already there: refused by default, the run fails at the end; -f overwrites. + (tmp_path / "notes.txt").write_bytes(b"mine") + assert main("get", "Family/share/notes.txt") == 4 + assert "already exists" in capsys.readouterr().err + assert (tmp_path / "notes.txt").read_bytes() == b"mine" + assert main("get", "-f", "Family/share/notes.txt") == 0 + assert (tmp_path / "notes.txt").read_bytes() == b"hello" + + assert main("get", "Family/share/2024/*.jpg", "-o", "pics/") == 0 + assert sorted(p.name for p in (tmp_path / "pics").iterdir()) == ["a.jpg", "b.jpg"] + + assert main("get", "Family/share/2024") == 1 # a folder needs -R + assert "use -R" in capsys.readouterr().err + assert main("get", "-R", "Family/share/2024", "-o", "tree/") == 0 + assert (tmp_path / "tree/2024/deep/c.txt").read_bytes() == b"C" + assert (tmp_path / "tree/2024/empty").is_dir() + + assert main("get", "Family/share/nope*") == 3 + + assert main("get", "Family/share/notes.txt", "-o", "-") == 0 + assert capsys.readouterr().out == "hello" + + +def test_put_files_folders_and_stdin(node, tmp_path, monkeypatch, capsys): + node.add("share", "taken.txt", b"theirs") + (tmp_path / "a.txt").write_bytes(b"one") + (tmp_path / "Taken.TXT").write_bytes(b"mine") + scans = tmp_path / "Scans" + (scans / "2024").mkdir(parents=True) + (scans / "2024" / "p1.pdf").write_bytes(b"%PDF" + b"x" * 100_000) + (scans / "empty").mkdir() + + assert main("put", str(tmp_path / "a.txt"), "Family/share") == 0 + assert node.stored("share/a.txt") == b"one" + + # A name already there, whatever its case: nothing is sent at all. + assert main("put", str(tmp_path / "a.txt"), str(tmp_path / "Taken.TXT"), + "Family/share") == 8 + assert "already there" in capsys.readouterr().err + assert node.stored("share/Taken.TXT") is None + # -n skips it and sends the rest. + assert main("put", "-n", str(tmp_path / "Taken.TXT"), "Family/share") == 0 + + assert main("put", str(scans), "Family/share") == 1 # a folder needs -R + assert main("put", "-R", str(scans), "Family/share") == 0 + assert node.stored("share/Scans/2024/p1.pdf").startswith(b"%PDF") + assert "share/Scans/empty" in node.dirs + + bad = tmp_path / ".hidden" + bad.write_bytes(b"x") + assert main("put", str(bad), "Family/share") == 8 + assert node.stored("share/.hidden") is None + + monkeypatch.setattr("sys.stdin", io.TextIOWrapper(io.BytesIO(b"piped bytes"))) + assert main("put", "-", "Family/share/from-stdin.txt") == 0 + assert node.stored("share/from-stdin.txt") == b"piped bytes" + + +def test_put_into_a_read_only_folder_sends_nothing(node, tmp_path, capsys): + node.roots[0]["writable"] = False + (tmp_path / "a.txt").write_bytes(b"one") + assert main("put", str(tmp_path / "a.txt"), "Family/share") == 8 + assert "read-only" in capsys.readouterr().err + + +def test_trust_lists_and_forgets_pins(hub, capsys): + assert main("trust", "list") == 0 # nothing stored: nothing pinned + assert main("--hub", hub.url, "-u", USERNAME, "--password", PASSPHRASE, + "--json", "login") == 0 + assert json.loads(capsys.readouterr().out)["groups"] == 1 + from meshbay_cli.client.store import Store + store = Store() + store.load() + store.state["pins"] = {"node-1": "pk-1"} + store.save() + assert main("--json", "trust", "list") == 0 + assert json.loads(capsys.readouterr().out) == {"node-1": "pk-1"} + assert main("trust", "reset") == 1 + assert main("trust", "reset", "node-2") == 3 + assert main("trust", "reset", "node-1") == 0 + assert main("--json", "trust", "list") == 0 + capsys.readouterr() + assert main("--json", "logout") == 0 + assert json.loads(capsys.readouterr().out) == {"signed_out": True} + + +def test_a_usage_error_is_exit_1_not_the_sign_in_code(capsys): + """argparse's own 2 would read as a refused sign-in to a script.""" + assert cli.main(["ls", "--no-such-option"]) == 1 + assert cli.main(["no-such-action"]) == 1 + + + +def test_groups_lists_each_group_with_its_state(hub, capsys): + hub.groups.append({"id": "g2", "name": "Club/Photos", "owner_username": "bob-test", + "node_online": False, "visibility": "public", "is_admin": False, + "hosted": True, "last_activity_at": "2026-10-01T12:00:00+00:00"}) + hub.groups.append({"id": "g3", "name": "Draft\x1b[31m", "owner_username": USERNAME, + "visibility": "private", "is_admin": True, "hosted": False}) + assert main("groups") == 2 # signed in nowhere + capsys.readouterr() + assert main("--hub", hub.url, "-u", USERNAME, "--password", PASSPHRASE, "login") == 0 + capsys.readouterr() + + assert main("groups") == 0 + header, *lines = capsys.readouterr().out.splitlines() + assert header.split() == ["NODE", "ROLE", "VISIBILITY", "LAST", "ACTIVITY", "GROUP"] + # In name order; a "/" in a name is encoded so the path can be typed back. + assert lines[0].split()[:3] == ["down", "member", "public"] + assert lines[0].endswith("Club%2FPhotos@bob-test") + assert "2026-10-0" in lines[0] + assert lines[1].split()[:3] == ["none", "owner", "private"] + assert "\x1b" not in lines[1] and lines[1].endswith(f"Draft\\x1b[31m@{USERNAME}") + assert lines[2].split()[:3] == ["up", "owner", "private"] + + assert main("--json", "groups") == 0 + rows = json.loads(capsys.readouterr().out) + assert [(r["group"], r["node"], r["role"]) for r in rows] == [ + ("Club/Photos", "down", "member"), ("Draft\x1b[31m", "none", "owner"), + ("Family", "up", "owner")] + assert rows[0]["path"] == "Club%2FPhotos@bob-test" + + +def test_every_action_is_run_here(): + """Runs last in this module: every action the parser knows was run above.""" + assert RUN == set(cli.ACTIONS), f"never run: {set(cli.ACTIONS) - RUN}" diff --git a/packages/meshbay-cli/tests/test_cli_transfers.py b/packages/meshbay-cli/tests/test_cli_transfers.py new file mode 100644 index 0000000..c240896 --- /dev/null +++ b/packages/meshbay-cli/tests/test_cli_transfers.py @@ -0,0 +1,206 @@ +"""Downloads and uploads against a node in memory (`fake_node.py`).""" + +import io +import json +import secrets + +import pytest +from fake_node import CHUNK, FakeNode +from meshbay_cli.client.names import UnsafeName, local_path, upload_name_problem +from meshbay_cli.client.transfers import ( + PART_SUFFIX, + UPLOAD_CHUNK, + TransferError, + create_dir, + download, + upload, +) + + +def _reader(data: bytes): + buf = io.BytesIO(data) + + def read(offset, length): + buf.seek(offset) + return buf.read(length) + return read + + +# ── Download ───────────────────────────────────────────────────────────────── + +async def test_a_file_of_several_chunks_arrives_whole(tmp_path): + node = FakeNode() + data = secrets.token_bytes(2 * CHUNK + 12345) + entry = node.add("share", "big.bin", data) + target = tmp_path / "big.bin" + seen = [] + assert await download(node.session(), entry, target, progress=seen.append) == len(data) + assert target.read_bytes() == data + assert sum(seen) == len(data) + assert not list(tmp_path.glob(f"*{PART_SUFFIX}*")) + assert node.leases == {}, "the lease is closed at the end" + + +async def test_an_interrupted_download_resumes_from_its_last_whole_chunk(tmp_path): + node = FakeNode() + data = secrets.token_bytes(3 * CHUNK + 7) + entry = node.add("share", "big.bin", data) + target = tmp_path / "big.bin" + part = tmp_path / f"big.bin{PART_SUFFIX}" + # A chunk and a half on disk: only the whole chunk counts. + part.write_bytes(data[:CHUNK + CHUNK // 2]) + (tmp_path / f"big.bin{PART_SUFFIX}.json").write_text( + json.dumps({"id": entry["id"], "size": entry["size"]})) + await download(node.session(), entry, target) + assert target.read_bytes() == data + assert node.chunk_requests == 3 + + +async def test_a_part_of_another_file_is_started_over(tmp_path): + node = FakeNode() + data = secrets.token_bytes(2 * CHUNK) + entry = node.add("share", "big.bin", data) + (tmp_path / f"big.bin{PART_SUFFIX}").write_bytes(b"x" * CHUNK) + (tmp_path / f"big.bin{PART_SUFFIX}.json").write_text( + json.dumps({"id": "0" * 64, "size": entry["size"]})) + await download(node.session(), entry, tmp_path / "big.bin") + assert (tmp_path / "big.bin").read_bytes() == data + assert node.chunk_requests == 2 + + +async def test_a_refused_chunk_fails_the_file_and_keeps_the_part(tmp_path): + node = FakeNode() + entry = node.add("share", "big.bin", secrets.token_bytes(3 * CHUNK)) + node.fail_chunk = (entry["id"], 2) + with pytest.raises(TransferError) as e: + await download(node.session(), entry, tmp_path / "big.bin") + assert "chunk 2" in e.value.message and "stopped answering" in e.value.message + assert not (tmp_path / "big.bin").exists() + assert (tmp_path / f"big.bin{PART_SUFFIX}").stat().st_size == 2 * CHUNK + assert node.leases == {} + + +async def test_a_file_that_appeared_meanwhile_is_not_replaced(tmp_path): + node = FakeNode() + entry = node.add("share", "a.txt", b"from the group") + (tmp_path / "a.txt").write_bytes(b"mine") + with pytest.raises(TransferError): + await download(node.session(), entry, tmp_path / "a.txt") + assert (tmp_path / "a.txt").read_bytes() == b"mine" + await download(node.session(), entry, tmp_path / "a.txt", overwrite=True) + assert (tmp_path / "a.txt").read_bytes() == b"from the group" + + +async def test_a_queued_lease_waits_for_its_grant(tmp_path): + node = FakeNode() + node.queue_first = 1 + entry = node.add("share", "a.txt", b"hello") + queued = [] + await download(node.session(), entry, tmp_path / "a.txt", on_queued=queued.append) + assert queued == [1] and (tmp_path / "a.txt").read_bytes() == b"hello" + + +async def test_an_interrupted_download_keeps_its_part_and_closes_its_lease(tmp_path): + import asyncio + node = FakeNode() + node.chunk_delay = 0.05 + data = secrets.token_bytes(20 * CHUNK) + entry = node.add("share", "big.bin", data) + task = asyncio.ensure_future(download(node.session(), entry, tmp_path / "big.bin")) + while node.chunk_requests < 12: + await asyncio.sleep(0.01) + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert node.leases == {} + assert not (tmp_path / "big.bin").exists() + kept = (tmp_path / f"big.bin{PART_SUFFIX}").stat().st_size + assert kept > 0 and kept % CHUNK == 0 + node.chunk_delay = 0 + await download(node.session(), entry, tmp_path / "big.bin") + assert (tmp_path / "big.bin").read_bytes() == data + + +# ── Upload ─────────────────────────────────────────────────────────────────── + +async def test_an_upload_of_several_chunks_is_stored_whole(): + node = FakeNode() + data = secrets.token_bytes(5 * UPLOAD_CHUNK + 99) + stored = await upload(node.session(), _reader(data), len(data), "scan.pdf", "share") + assert stored == "scan.pdf" + assert node.stored("share/scan.pdf") == data + assert node.leases == {} + + +async def test_an_interrupted_upload_resumes_where_the_node_stopped(): + node = FakeNode() + data = secrets.token_bytes(4 * UPLOAD_CHUNK) + node.uploads[("share", "scan.pdf")] = {"next": 2, "data": data[:2 * UPLOAD_CHUNK]} + sent = [] + await upload(node.session(), _reader(data), len(data), "scan.pdf", "share", + progress=sent.append) + assert node.stored("share/scan.pdf") == data + assert sum(sent) == len(data) + + +async def test_a_name_taken_meanwhile_is_reported(): + node = FakeNode() + node.add("share", "scan.pdf", b"theirs") + stored = await upload(node.session(), _reader(b"mine"), 4, "scan.pdf", "share") + assert stored == "scan (1).pdf" + assert node.stored("share/scan.pdf") == b"theirs" + + +async def test_a_full_disk_is_fatal(): + node = FakeNode() + node.refuse_upload_code = "disk_full" + with pytest.raises(TransferError) as e: + await upload(node.session(), _reader(b"x"), 1, "a.txt", "share") + assert e.value.fatal + + +async def test_folders_are_created_where_asked(): + node = FakeNode() + assert await create_dir(node.session(), "share", "Scans") == "share/Scans" + assert "share/Scans" in node.dirs + + +# ── Names ──────────────────────────────────────────────────────────────────── + +NAMES = [ + "photo.jpg", "Été 2024", "東京", "track (1).flac", "[live] set", "a_b-c", + "rock & roll", "l’été", "x+y#z@w", "9lives", + ".DS_Store", "_private", "-dash", " space", "trailing ", "trailing.", + "a/b", "a\\b", "semi;colon", "x" * 128, "x" * 129, "", "tab\tname", +] + + +def test_the_cli_refuses_exactly_the_names_the_node_refuses(): + roots = pytest.importorskip("meshbay_node.roots") + differ = [n for n in NAMES + if bool(roots.SAFE_UPLOAD_NAME.match(n)) != (upload_name_problem(n, is_dir=True) + is None)] + assert not differ, f"the CLI and the node disagree on: {differ}" + for n in ["desktop.ini", "x.LNK", "a.url", "b.scf", "c.library-ms", "d.searchConnector-ms"]: + assert roots.shell_active(n) and upload_name_problem(n), n + + +def test_a_downloaded_name_is_made_writable_and_stays_inside(tmp_path): + path, renamed = local_path(tmp_path, ["evil\x1b[31mred.txt"]) + assert renamed and path.parent == tmp_path and "\x1b" not in path.name + path, renamed = local_path(tmp_path, ["CON.txt"]) + assert renamed and path.name != "CON.txt" + path, renamed = local_path(tmp_path, ["été", "plage de l'ouest.jpg"]) + assert not renamed and path == tmp_path / "été" / "plage de l'ouest.jpg" + path, _ = local_path(tmp_path, [".."]) + assert path.parent == tmp_path + + +def test_a_link_out_of_the_destination_is_refused(tmp_path): + outside = tmp_path / "outside" + outside.mkdir() + dest = tmp_path / "dest" + dest.mkdir() + (dest / "sub").symlink_to(outside) + with pytest.raises(UnsafeName): + local_path(dest, ["sub", "x.txt"]) diff --git a/packages/meshbay-cli/tests/test_e2e_local.py b/packages/meshbay-cli/tests/test_e2e_local.py new file mode 100644 index 0000000..4283819 --- /dev/null +++ b/packages/meshbay-cli/tests/test_e2e_local.py @@ -0,0 +1,570 @@ +""" +The real `meshbay-cli` against a real hub and a real node, both on loopback. + +Opt-in (MESHBAY_CLI_E2E=1): it starts a throwaway hub (SQLite, a fresh key) and +a node in a home of its own, which takes a while and runs Argon2 at 128 MB +several times. Nothing reaches a real hub or the developer's node. + +What it sets up, as an operator and a browser would: +- an account, activated directly in the throwaway hub's database (no mail); +- a group whose name holds a `/` and an `@`, which the hub allows and the path + syntax has to carry; +- a node linked to the account, hosting the group over a folder whose names + are the awkward ones real data has: spaces, accents, an apostrophe, a long + name without spaces, and a name with an escape character in it; +- the account's first contact with the node — a member invitation code, then a + keypair bundle left on the node — done here the way a browser does it, + because the CLI does not join groups. + +Then the CLI: login, ls of groups, of a root, recursive, JSON, a second run +from the identity it kept, and logout. + + MESHBAY_CLI_E2E=1 pytest packages/meshbay-cli/tests/test_e2e_local.py -v + +MESHBAY_CLI_E2E_BIN names another meshbay-cli to run in place of the one in +this venv: a packaged build, through a wrapper that mounts it where it installs. +""" + +import asyncio +import base64 +import json +import os +import re +import secrets +import socket +import sqlite3 +import subprocess +import sys +import time +from pathlib import Path + +import httpx +import pytest + +pytestmark = [ + pytest.mark.skipif(os.environ.get("MESHBAY_CLI_E2E") != "1", + reason="opt-in: set MESHBAY_CLI_E2E=1 (see the module docstring)"), + pytest.mark.timeout(600), +] + +USERNAME = "alice-e2e-cli" +PASSPHRASE = "a long enough test passphrase, nobody's" +GROUP_NAME = "Some Saga/Box @Set" +BIN = Path(sys.executable).parent + + +def _free_port() -> int: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +def _wait_http(url: str, timeout: float = 60) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + httpx.get(url, timeout=2) + return + except httpx.HTTPError: + time.sleep(0.3) + raise AssertionError(f"{url} did not come up") + + +@pytest.fixture(scope="module") +def world(tmp_path_factory): + root = tmp_path_factory.mktemp("e2e") + procs: list[subprocess.Popen] = [] + try: + yield _build(root, procs) + finally: + for p in procs: + p.terminate() + for p in procs: + try: + p.wait(10) + except subprocess.TimeoutExpired: + p.kill() + + +def _build(root: Path, procs: list) -> dict: + from cryptography.hazmat.primitives import serialization + from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + from meshbay_cli.client.keys import auth_key + + # ── The hub ────────────────────────────────────────────────────────────── + hub_home = root / "hub-home" + hub_home.mkdir() + key = root / "hub_private.pem" + key.write_bytes(Ed25519PrivateKey.generate().private_bytes( + serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, + serialization.NoEncryption())) + db = root / "hub.db" + port = _free_port() + hub_url = f"http://127.0.0.1:{port}" + hub_env = {**os.environ, "HOME": str(hub_home), + "MESHBAY_DATABASE_URL": f"sqlite+aiosqlite:///{db}", + "MESHBAY_HUB_KEY": str(key), "MESHBAY_HUB_ID": "e2e-hub", + "MESHBAY_CAPTCHA_ENABLED": "false"} + hub_log = open(root / "hub.log", "w") + procs.append(subprocess.Popen( + [sys.executable, "-m", "uvicorn", "meshbay_hub.app:create_app", "--factory", + "--host", "127.0.0.1", "--port", str(port)], + env=hub_env, stdout=hub_log, stderr=subprocess.STDOUT)) + _wait_http(f"{hub_url}/v1/hub/version") + + ak = auth_key(USERNAME, PASSPHRASE) + with httpx.Client(base_url=hub_url, timeout=60) as hub: + r = hub.post("/v1/users/register", json={ + "username": USERNAME, "email": "alice@example.invalid", "auth_key": ak}) + assert r.status_code == 201, r.text + with sqlite3.connect(db) as conn: + conn.execute("UPDATE users SET status='active' WHERE username=?", (USERNAME,)) + r = hub.post("/v1/users/login", json={"username": USERNAME, "auth_key": ak}) + assert r.status_code == 200, r.text + auth = {"Authorization": f"Bearer {r.json()['access_token']}"} + r = hub.post("/v1/groups", headers=auth, json={ + "name": GROUP_NAME, "visibility": "private", "join_policy": "invite"}) + assert r.status_code == 201, r.text + group_id = r.json()["group_id"] + + # ── The node ───────────────────────────────────────────────────────── + node_home = root / "node-home" + node_home.mkdir() + share = root / "share" + (share / "2024" / "été à la mer").mkdir(parents=True) + (share / "2024" / "été à la mer" / "plage de l'ouest.jpg").write_bytes( + secrets.token_bytes(3000)) + (share / "2024" / "A_Very_Long_Name_Without_Any_Spaces_In_It_1992-2000.flac")\ + .write_bytes(secrets.token_bytes(2_500_000)) + (share / "notes.txt").write_text("hello\n") + (share / "evil\x1b[31mred.txt").write_text("escape\n") + node_env = {**os.environ, "HOME": str(node_home), + "XDG_CONFIG_HOME": str(node_home / ".config"), + "XDG_DATA_HOME": str(node_home / ".local/share"), + "XDG_STATE_HOME": str(node_home / ".local/state")} + node = [str(BIN / "meshbay-node")] + + def run_node(*args: str) -> str: + p = subprocess.run(node + list(args), env=node_env, capture_output=True, + text=True, timeout=120) + assert p.returncode == 0, f"meshbay-node {args}: {p.stdout}\n{p.stderr}" + return p.stdout + + out = run_node("init", "--hub-url", hub_url, "--username", USERNAME) + node_pk = re.search(r"Node key: (\S+)", out).group(1) + cfg = next(node_home.rglob("node.toml")) + ui_port = _free_port() + cfg.write_text(cfg.read_text().replace("ui_port = 18000", + f"ui_port = {ui_port}")) + r = hub.put("/v1/users/me/node_key", headers=auth, + json={"pk_node_ed25519": node_pk}) + assert r.status_code == 200, r.text + + node_log = open(root / "node.log", "w") + procs.append(subprocess.Popen(node, env=node_env, stdout=node_log, + stderr=subprocess.STDOUT)) + _wait_http(f"http://127.0.0.1:{ui_port}/", timeout=90) + + # Attaching reloads the node by itself; the key needs the group loaded. + run_node("group", "add", group_id, "--dir", str(share)) + deadline = time.monotonic() + 90 + while "GEK ready" not in (out := _try(run_node, "gek", "init", "--group", group_id)): + assert time.monotonic() < deadline, f"gek init never succeeded: {out}" + time.sleep(2) + code = re.search(r"INVITATION CODE\s+(\S+)", + run_node("member", "invite", USERNAME, "--group", group_id)).group(1) + + # The node is online at the hub once it lists itself for the group. + deadline = time.monotonic() + 90 + while not hub.get(f"/v1/groups/{group_id}/nodes", headers=auth).json()["nodes"]: + assert time.monotonic() < deadline, "the node never reached the hub" + time.sleep(1) + + app = App(hub_url, group_id) + asyncio.run(app.first_contact(code)) + return {"root": root, "hub_url": hub_url, "group_id": group_id, "db": db, + "share": share, "app": app} + + +def _try(fn, *args) -> str: + try: + return fn(*args) + except AssertionError as e: + return str(e) + + +class App: + """The application's side of the test: a device already linked to the node.""" + + def __init__(self, hub_url: str, group_id: str): + self.hub_url, self.group_id = hub_url, group_id + self.ident = None + + async def session(self, code: str = ""): + """Signed in, handshake done, joined: (channel, close, node_pk, nonce_s, user_id).""" + import os as _os + + from aiortc import ( + RTCConfiguration, + RTCIceServer, + RTCPeerConnection, + RTCSessionDescription, + ) + from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey + from meshbay_cli.client.account import Account + from meshbay_cli.client.hub import Hub + from meshbay_cli.client.keys import Identity, b64, unb64 + from meshbay_cli.client.session import STUN_SERVERS, Channel, _dtls_fingerprint + from meshbay_common import MNP_VERSION + from meshbay_common.crypto import unwrap_gek_aes + from meshbay_common.handshake import MNP_MIN_SUPPORTED, make_proof, webrtc_binding + from meshbay_common.join import join_transcript + + hub = Hub(self.hub_url) + account = Account(hub, USERNAME, lambda: PASSPHRASE) + await account.sign_in(remember=False) + token = await account.token() + node = (await hub.group_nodes(token, self.group_id))[0] + mnp = await hub.mnp_token(token, node["pk_node"], self.group_id) + pc = RTCPeerConnection(RTCConfiguration( + iceServers=[RTCIceServer(urls=u) for u in STUN_SERVERS])) + dc = pc.createDataChannel("mnp", ordered=True) + ch = Channel(dc) + opened = asyncio.Event() + dc.on("open", opened.set) + await pc.setLocalDescription(await pc.createOffer()) + offer = pc.localDescription.sdp + answer = await hub.offer(token, node["node_id"], offer) + await pc.setRemoteDescription(RTCSessionDescription(sdp=answer, type="answer")) + await asyncio.wait_for(opened.wait(), 30) + binding = webrtc_binding(_dtls_fingerprint(offer), _dtls_fingerprint(answer)) + nonce_c = _os.urandom(32) + chal = await ch.request({"type": "handshake", "v": MNP_VERSION, + "v_min": MNP_MIN_SUPPORTED, "token": mnp, + "group_id": self.group_id, "nonce": b64(nonce_c)}, + "handshake_challenge") + nonce_s, node_pk = unb64(chal["nonce"]), chal["node_pk"] + if self.ident is None: + self.ident = Identity(Ed25519PrivateKey.generate(), X25519PrivateKey.generate()) + ident, ts = self.ident, int(time.time()) + joined = await ch.request({ + "type": "join_request", "v": MNP_VERSION, "group_id": self.group_id, + "pk_ed25519": ident.pk_ed, "pk_x25519": ident.pk_x, "code": code, "ts": ts, + "sig": b64(ident.sk_ed.sign(join_transcript( + node_pk_b64=node_pk, group_id=self.group_id, user_id=account.user_id, + pk_ed25519_b64=ident.pk_ed, pk_x25519_b64=ident.pk_x, + nonce_node=nonce_s, ts=ts)))}, "join_result") + assert joined.get("ok"), joined + gek = unwrap_gek_aes(joined, ident.sk_x.private_bytes_raw(), + ident.sk_x.public_key().public_bytes_raw()) + await ch.request({"type": "handshake_response", "v": MNP_VERSION, + "proof": b64(make_proof(gek, "client", self.group_id, nonce_c, + nonce_s, binding))}, "handshake_ack") + + async def close(): + await pc.close() + await account.close() + await hub.aclose() + return ch, close, node_pk, nonce_s, account + + async def first_contact(self, code: str) -> None: + """What a browser does the first time: join with the code, leave a bundle.""" + import os as _os + + from cryptography.hazmat.primitives.ciphers.aead import AESGCM + from meshbay_cli.client.keys import node_bundle_key + from meshbay_common import MNP_VERSION + + ch, close, node_pk, _, account = await self.session(code) + # MBK3, as keyderive.js encryptBundle writes it. + nonce = _os.urandom(12) + aad = f"meshbay:bundle:v3|{account.user_id}|{node_pk}".encode() + ct = AESGCM(node_bundle_key(account.master, node_pk)).encrypt( + nonce, json.dumps(self.ident.to_pkcs8()).encode(), aad) + bundle = base64.b64encode(b"MBK3" + bytes([1]) + nonce + ct).decode() + stored = await ch.request({"type": "keypair_bundle_store", "v": MNP_VERSION, + "bundle_enc": bundle}, "ack") + assert stored.get("detail") == "keypair_bundle_stored", stored + await close() + + async def withdraw_bundle(self) -> None: + """Browser access turned off: the application withdraws the node's copy.""" + from meshbay_common import MNP_VERSION + ch, close, *_ = await self.session() + r = await ch.request({"type": "keypair_bundle_delete", "v": MNP_VERSION}, "ack") + assert r.get("detail") == "keypair_bundle_deleted", r + await close() + + async def approve(self, code: str) -> None: + """Approve a device by the code it shows, as the group's Settings tab does.""" + from meshbay_cli.client.keys import b64 + from meshbay_common import MNP_VERSION + from meshbay_common.device import device_add_transcript, device_code_hash + ch, close, node_pk, nonce_s, account = await self.session() + listed = await ch.request({"type": "device_lookup", "v": MNP_VERSION}, + "device_lookup_result") + raw = code.replace("-", "").upper() + match = next(r for r in listed["requests"] + if device_code_hash(raw, r["pk_ed25519"], r["pk_x25519"]) + == r["code_hash"]) + ts = int(time.time()) + sig = self.ident.sk_ed.sign(device_add_transcript( + node_pk_b64=node_pk, user_id=account.user_id, + pk_ed25519_b64=match["pk_ed25519"], pk_x25519_b64=match["pk_x25519"], + nonce_node=nonce_s, ts=ts)) + await ch.request({"type": "device_add", "v": MNP_VERSION, + "pk_ed25519": match["pk_ed25519"], + "pk_x25519": match["pk_x25519"], + "code_hash": match["code_hash"], "ts": ts, "sig": b64(sig)}, + "device_add_ack") + await close() + + +def cli(world: dict, *args: str, stdin: str | None = None, + home: str = "cli") -> subprocess.CompletedProcess: + env = {**os.environ, "MESHBAY_CLI_HOME": str(world["root"] / home), + "PYTHON_KEYRING_BACKEND": "keyring.backends.fail.Keyring"} + env.pop("MESHBAY_PASSWORD", None) + exe = os.environ.get("MESHBAY_CLI_E2E_BIN") or str(BIN / "meshbay-cli") + return subprocess.run([exe, *args], env=env, input=stdin, + capture_output=True, text=True, timeout=180) + + +def _handle() -> str: + return GROUP_NAME.replace("%", "%25").replace("/", "%2F") + "@" + USERNAME + + +def test_the_cli_against_a_real_hub_and_node(world): + hub = world["hub_url"] + + r = cli(world, "--hub", hub, "-u", USERNAME, "login", "--password-file", "-", + stdin=PASSPHRASE + "\n") + assert r.returncode == 0, r.stderr + assert "Signed in" in r.stdout and "1 group" in r.stdout + # No key storage in this environment: the file fallback, said once. + assert "No OS key storage here" in r.stderr + + # The groups, from the hub alone; the stored session needs no passphrase. + r = cli(world, "ls") + assert r.returncode == 0, r.stderr + assert r.stdout.strip() == _handle() + r = cli(world, "groups") + assert r.returncode == 0, r.stderr + print(r.stdout) + [line] = r.stdout.splitlines()[1:] + assert line.split()[:3] == ["up", "owner", "private"] and line.endswith(_handle()) + + # One group, by its handle: its root. + r = cli(world, "-vv", "ls", "-l", _handle()) + assert r.returncode == 0, r.stderr + assert "keypair_bundle_fetch" in r.stderr # first contact: the node's copy + assert re.search(r"^root\s.*share/\s+\(writable\)$", r.stdout, re.M), r.stdout + + # Recursive: the awkward names arrive, the escape character is escaped. + r = cli(world, "-vv", "ls", "-R", "-l", f"{_handle()}/share") + assert r.returncode == 0, r.stderr + assert "keypair_bundle_fetch" not in r.stderr # the identity was kept + out = r.stdout + assert "plage de l'ouest.jpg" in out + assert "A_Very_Long_Name_Without_Any_Spaces_In_It_1992-2000.flac" in out + assert "\x1b" not in out and "evil\\x1b[31mred.txt" in out + assert "été à la mer/" in out + + # By name alone, case-insensitively, and as JSON. + r = cli(world, "--json", "ls", "some saga%2Fbox @set/share/2024") + assert r.returncode == 0, r.stderr + items = json.loads(r.stdout) + names = {i["name"]: i for i in items} + assert names["été à la mer"]["kind"] == "dir" + flac = names["A_Very_Long_Name_Without_Any_Spaces_In_It_1992-2000.flac"] + assert flac["kind"] == "file" and flac["type"] == "audio" + assert flac["size"] == 2_500_000 and len(flac["id"]) == 64 + + # Not found is exit 3; an unknown group too. + assert cli(world, "ls", f"{_handle()}/share/nope").returncode == 3 + assert cli(world, "ls", "No Such Group").returncode == 3 + + # ── get ────────────────────────────────────────────────────────────────── + out_dir = world["root"] / "downloads" + flac = "A_Very_Long_Name_Without_Any_Spaces_In_It_1992-2000.flac" + r = cli(world, "get", f"{_handle()}/share/2024/{flac}", "-o", f"{out_dir}/") + assert r.returncode == 0, r.stderr + assert (out_dir / flac).read_bytes() == (world["share"] / "2024" / flac).read_bytes() + r = cli(world, "get", "-R", f"{_handle()}/share", "-o", f"{out_dir}/tree/") + assert r.returncode == 0, r.stderr + got = out_dir / "tree" / "share" + assert (got / "2024" / "été à la mer" / "plage de l'ouest.jpg").read_bytes() == \ + (world["share"] / "2024" / "été à la mer" / "plage de l'ouest.jpg").read_bytes() + # The escape character could not be written as it is: renamed, and said so. + assert "evil\\x1b[31mred.txt is saved as evil_[31mred.txt" in r.stderr + assert (got / "evil_[31mred.txt").exists() + r = cli(world, "get", f"{_handle()}/share/notes.txt", "-o", "-") + assert r.returncode == 0 and r.stdout == "hello\n" + + # ── put ────────────────────────────────────────────────────────────────── + local = world["root"] / "to-send" + (local / "Scans" / "2024").mkdir(parents=True) + scan = secrets.token_bytes(300_000) + (local / "Scans" / "2024" / "page 1.pdf").write_bytes(scan) + (local / "Scans" / "empty").mkdir() + (local / "Notes.TXT").write_bytes(b"clash") + r = cli(world, "put", "-R", str(local / "Scans"), f"{_handle()}/share/2024") + assert r.returncode == 0, r.stderr + assert (world["share"] / "2024" / "Scans" / "2024" / "page 1.pdf").read_bytes() == scan + assert (world["share"] / "2024" / "Scans" / "empty").is_dir() + # notes.txt is there already, whatever the case: nothing sent. + r = cli(world, "put", str(local / "Notes.TXT"), f"{_handle()}/share") + assert r.returncode == 8 and "already there" in r.stderr + assert not (world["share"] / "Notes.TXT").exists() + r = cli(world, "put", "-", f"{_handle()}/share/piped.txt", stdin="through stdin") + assert r.returncode == 0, r.stderr + assert (world["share"] / "piped.txt").read_text() == "through stdin" + # And what was sent can be fetched back, once the node has indexed it. + deadline = time.monotonic() + 60 + while True: + r = cli(world, "get", f"{_handle()}/share/2024/Scans/2024/page 1.pdf", "-o", + str(out_dir / "back.pdf")) + if r.returncode == 0 or time.monotonic() > deadline: + break + time.sleep(1) + assert r.returncode == 0, r.stderr + assert (out_dir / "back.pdf").read_bytes() == scan + + # ── ls -R over every group ─────────────────────────────────────────────── + deadline = time.monotonic() + 60 + while True: + r = cli(world, "ls", "-R", "/") + assert r.returncode == 0, r.stderr + if "piped.txt" in r.stdout or time.monotonic() > deadline: + break + time.sleep(1) # the node indexes what it received on its own schedule + assert "page 1.pdf" in r.stdout and "piped.txt" in r.stdout, r.stdout + + # The device key was registered, and logout retires it. + with sqlite3.connect(world["db"]) as conn: + assert conn.execute("SELECT count(*) FROM user_devices").fetchone()[0] == 1 + r = cli(world, "logout") + assert r.returncode == 0, r.stderr + assert "Signed out." in r.stdout + with sqlite3.connect(world["db"]) as conn: + assert conn.execute("SELECT count(*) FROM user_devices").fetchone()[0] == 0 + assert not (world["root"] / "cli" / "secrets.enc").exists() + + # Signed out: a run needs credentials again, and says so. + r = cli(world, "ls") + assert r.returncode == 2 and "Not signed in" in r.stderr + + +def test_a_changed_node_key_is_refused_until_reset(world): + def run(*args, stdin=None): + return cli(world, *args, stdin=stdin, home="cli-trust") + + r = run("--hub", world["hub_url"], "-u", USERNAME, "login", "--password-file", "-", + stdin=PASSPHRASE + "\n") + assert r.returncode == 0, r.stderr + assert run("ls", _handle()).returncode == 0 + + r = run("--json", "trust", "list") + [(node_id, pk)] = json.loads(r.stdout).items() + state_path = world["root"] / "cli-trust" / "state.json" + state = json.loads(state_path.read_text()) + state["pins"][node_id] = base64.b64encode(b"\x01" * 32).decode() + state_path.write_text(json.dumps(state)) + r = run("ls", _handle()) + assert r.returncode == 6 and "trust reset" in r.stderr, r.stderr + assert run("trust", "reset", "no-such-node").returncode == 3 + r = run("trust", "reset", node_id) + assert r.returncode == 0, r.stderr + assert run("ls", _handle()).returncode == 0 + assert json.loads(run("--json", "trust", "list").stdout) == {node_id: pk} + assert run("logout").returncode == 0 + + +def test_a_machine_with_no_identity_there_is_linked_as_a_device(world): + """Browser access off: the node keeps no copy, so a new machine asks to be + approved, shows a code, and joins with its own keys once approved.""" + app = world["app"] + asyncio.run(app.withdraw_bundle()) + + def run(*args, stdin=None): + return cli(world, *args, stdin=stdin, home="cli2") + + r = run("--hub", world["hub_url"], "-u", USERNAME, "login", "--password-file", "-", + stdin=PASSPHRASE + "\n") + assert r.returncode == 0, r.stderr + + # No terminal here: the code is shown, and the run ends rather than waits. + r = run("ls", _handle()) + assert r.returncode == 5, r.stderr + found = re.search(r"enter\s+([0-9A-Z]{4}-[0-9A-Z]{4})", r.stderr) + assert found, r.stderr + code = found.group(1) + # A second run shows the same code: the request it filed is still pending. + r = run("ls", _handle()) + assert r.returncode == 5 and code in r.stderr, r.stderr + + asyncio.run(app.approve(code)) + r = run("ls", "-l", _handle()) + assert r.returncode == 0, r.stderr + assert "share/" in r.stdout + + assert run("logout").returncode == 0 + + +async def test_the_interactive_mode_against_the_real_node(world, monkeypatch, tmp_path): + """The panes, opened on a real group, and F5 bringing a file back.""" + from meshbay_cli import __main__ as main_module + from meshbay_cli.commands.context import Context + from meshbay_cli.tui.app import MeshBayApp + from meshbay_cli.tui.panes import RemotePane + from meshbay_cli.tui.screens import MainScreen + + # Browser access off, whichever test ran before: this machine must be linked. + await world["app"].withdraw_bundle() + r = await asyncio.to_thread(cli, world, "--hub", world["hub_url"], "-u", USERNAME, + "login", "--password-file", "-", + stdin=PASSPHRASE + "\n", home="cli-tui") + assert r.returncode == 0, r.stderr + monkeypatch.setenv("MESHBAY_CLI_HOME", str(world["root"] / "cli-tui")) + monkeypatch.chdir(tmp_path) + args = main_module.build_parser().parse_args(["--hub", world["hub_url"]]) + app = MeshBayApp(Context(args)) + + def names(pane): + return [pane.table.get_row_at(i)[0].plain.split(" (")[0].strip() + for i in range(pane.table.row_count)] + + async def until(pilot, cond, timeout=60): + deadline = time.monotonic() + timeout + while not cond(): + shown = (names(app.screen.query_one(RemotePane)) + if isinstance(app.screen, MainScreen) else None) + assert time.monotonic() < deadline, ( + f"timed out on {type(app.screen).__name__}; " + f"notified: {[n.message for n in app._notifications]}; remote: {shown}") + await pilot.pause(0.1) + + async with app.run_test(size=(140, 40)) as pilot: + await until(pilot, lambda: isinstance(app.screen, MainScreen)) + remote = app.screen.query_one(RemotePane) + name = _handle() + "/" + await until(pilot, lambda: name in names(remote)) + remote.table.move_cursor(row=names(remote).index(name)) + remote.table.focus() + await pilot.press("enter") + # The node keeps no copy of the identity: this machine is a new device, + # shows its code, and waits while it is approved elsewhere — then the + # group opens by itself. + await until(pilot, lambda: any("enter " in n.message for n in app._notifications)) + note = next(n.message for n in app._notifications if "enter " in n.message) + code = re.search(r"enter ([0-9A-Z]{4}-[0-9A-Z]{4})", note).group(1) + await world["app"].approve(code) + await until(pilot, lambda: "share/" in names(remote)) + remote.table.move_cursor(row=names(remote).index("share/")) + await pilot.press("enter") + await until(pilot, lambda: "notes.txt" in names(remote)) + remote.table.move_cursor(row=names(remote).index("notes.txt")) + await pilot.press("f5") + await until(pilot, lambda: (tmp_path / "notes.txt").exists() + and (tmp_path / "notes.txt").read_text() == "hello\n") diff --git a/packages/meshbay-cli/tests/test_interactive.py b/packages/meshbay-cli/tests/test_interactive.py new file mode 100644 index 0000000..680b9e0 --- /dev/null +++ b/packages/meshbay-cli/tests/test_interactive.py @@ -0,0 +1,159 @@ +""" +The interactive mode, driven by Textual's pilot against the in-memory hub and +node: what a person does with the keys, and what lands on disk and on the node. +""" + +import asyncio +import time + +import pytest +from fake_hub import PASSPHRASE, USERNAME, FakeHub +from fake_node import FakeNode +from meshbay_cli import __main__ as cli +from meshbay_cli.client import pool +from meshbay_cli.commands import context +from meshbay_cli.tui.app import MeshBayApp +from meshbay_cli.tui.panes import LocalPane, RemotePane +from meshbay_cli.tui.screens import MainScreen, SignInScreen +from meshbay_cli.tui.widgets import Choice + + +@pytest.fixture +def world(monkeypatch, tmp_path): + hub, node = FakeHub(), FakeNode() + monkeypatch.setattr(context, "Hub", lambda url: hub) + + async def open_group(account, group): + return node.session() + monkeypatch.setattr(pool, "open_group", open_group) + work = tmp_path / "work" + work.mkdir() + monkeypatch.chdir(work) + return hub, node, work + + +def _app() -> MeshBayApp: + args = cli.build_parser().parse_args(["--hub", "https://hub.test"]) + return MeshBayApp(context.Context(args)) + + +async def _until(pilot, condition, timeout: float = 10) -> None: + deadline = time.monotonic() + timeout + while not condition(): + assert time.monotonic() < deadline, "timed out" + await pilot.pause(0.05) + + +def _names(pane) -> list[str]: + """Row names as shown, without the mark or a note such as "(read-only)".""" + return [pane.table.get_row_at(i)[0].plain.split(" (")[0].strip() + for i in range(pane.table.row_count)] + + +async def _open(pilot, pane, name: str) -> None: + await _until(pilot, lambda: name in _names(pane)) + pane.table.move_cursor(row=_names(pane).index(name)) + pane.table.focus() + await pilot.press("enter") + + +async def test_signing_in_then_downloading_and_uploading(world): + hub, node, work = world + node.add("share", "notes.txt", b"hello") + node.add("share/2024", "a.jpg", b"A" * 3000) + node.add("share", "[bold]x[/bold]\x1b.txt", b"tricky") + (work / "mine.txt").write_bytes(b"from here") + + app = _app() + async with app.run_test(size=(140, 40)) as pilot: + # Nothing stored: the sign-in screen, then the panes. + await _until(pilot, lambda: isinstance(app.screen, SignInScreen)) + app.screen.query_one("#user").value = USERNAME + app.screen.query_one("#pass").value = PASSPHRASE + app.screen.query_one("#pass").focus() + await pilot.press("enter") + await _until(pilot, lambda: isinstance(app.screen, MainScreen)) + assert len(hub.devices) == 1, "signing in here remembers the machine" + + remote = app.screen.query_one(RemotePane) + local = app.screen.query_one(LocalPane) + await _open(pilot, remote, f"Family@{USERNAME}/") + await _open(pilot, remote, "share/") + await _until(pilot, lambda: "notes.txt" in _names(remote)) + + # A name somebody else chose is shown as text, escapes and all. + assert "[bold]x[/bold]\\x1b.txt" in _names(remote) + + # Mark two, F5: both land in the local folder. + for name in ("notes.txt", "2024/"): + remote.table.move_cursor(row=_names(remote).index(name)) + await pilot.press("space") + await pilot.press("f5") + await _until(pilot, lambda: (work / "2024" / "a.jpg").exists() + and (work / "notes.txt").exists()) + assert (work / "notes.txt").read_bytes() == b"hello" + assert (work / "2024" / "a.jpg").read_bytes() == b"A" * 3000 + + # Again: already there, asked once for the batch. + remote.table.move_cursor(row=_names(remote).index("notes.txt")) + remote.table.focus() + (work / "notes.txt").write_bytes(b"changed here") + await pilot.press("f5") + await _until(pilot, lambda: isinstance(app.screen, Choice)) + await pilot.click("#choice-skip") + await pilot.pause(0.2) + assert (work / "notes.txt").read_bytes() == b"changed here" + + # Upload from the left into the open folder. + await _until(pilot, lambda: isinstance(app.screen, MainScreen)) + local.action_refresh() + await _until(pilot, lambda: "mine.txt" in _names(local)) + local.table.move_cursor(row=_names(local).index("mine.txt")) + local.table.focus() + await pilot.press("f5") + await _until(pilot, lambda: node.stored("share/mine.txt") == b"from here") + # The node's index changed: the right pane shows it. + await _until(pilot, lambda: "mine.txt" in _names(remote)) + + # A name already there is asked about, never replaced. + local.table.focus() + await pilot.press("f5") + await _until(pilot, lambda: isinstance(app.screen, Choice)) + await pilot.click("#choice-cancel") + await pilot.pause(0.2) + assert node.stored("share/mine.txt") == b"from here" + + +async def test_a_remembered_session_opens_on_the_panes(world, capsys): + hub, node, _ = world + assert await asyncio.to_thread(cli.main, ["--hub", "https://hub.test", "-u", USERNAME, + "--password", PASSPHRASE, "login"]) == 0 + app = _app() + async with app.run_test(size=(120, 30)) as pilot: + await _until(pilot, lambda: isinstance(app.screen, MainScreen)) + remote = app.screen.query_one(RemotePane) + await _until(pilot, lambda: f"Family@{USERNAME}/" in _names(remote)) + + +async def test_a_read_only_folder_takes_no_upload(world): + hub, node, work = world + node.roots[0]["writable"] = False + node.add("share", "notes.txt", b"hello") + (work / "mine.txt").write_bytes(b"x") + assert await asyncio.to_thread(cli.main, ["--hub", "https://hub.test", "-u", USERNAME, + "--password", PASSPHRASE, "login"]) == 0 + app = _app() + async with app.run_test(size=(120, 30)) as pilot: + await _until(pilot, lambda: isinstance(app.screen, MainScreen)) + remote = app.screen.query_one(RemotePane) + local = app.screen.query_one(LocalPane) + await _open(pilot, remote, f"Family@{USERNAME}/") + await _open(pilot, remote, "share/") + await _until(pilot, lambda: "notes.txt" in _names(remote)) + await _until(pilot, lambda: "mine.txt" in _names(local)) + local.table.move_cursor(row=_names(local).index("mine.txt")) + local.table.focus() + await pilot.press("f5") + await pilot.pause(0.3) + assert node.stored("share/mine.txt") is None + assert any("does not accept uploads" in n.message for n in app._notifications) diff --git a/packages/meshbay-cli/tests/test_listing.py b/packages/meshbay-cli/tests/test_listing.py new file mode 100644 index 0000000..407074e --- /dev/null +++ b/packages/meshbay-cli/tests/test_listing.py @@ -0,0 +1,73 @@ +"""The index as the Files tab draws it, and names printed safely.""" + +import pytest +from meshbay_cli.client.tree import Tree +from meshbay_cli.errors import NOT_FOUND, CliError +from meshbay_cli.output import human_size, safe + +INDEX = { + "roots": [{"name": "share", "available": True, "writable": True}, + {"name": "usb", "available": False, "writable": False}], + "dirs": ["share", "share/2024", "share/empty", "usb", "usb/old"], + "entries": [ + {"id": "a" * 64, "name": "notes.txt", "path": "share", "size": 6, "type": "document"}, + {"id": "b" * 64, "name": "x.flac", "path": "share/2024", "size": 9, "type": "audio"}, + # A folder the node's walk did not list still exists through its file. + {"id": "c" * 64, "name": "y.jpg", "path": "share/2024/late", "size": 1, + "type": "image"}, + {"id": "d" * 64, "name": "z.txt", "path": "usb/old", "size": 1, "type": "document"}, + ], +} + + +def _names(items): + return sorted(i.name for i in items) + + +def test_the_top_is_the_roots(): + tree = Tree.from_index(INDEX) + top = tree.children("") + assert _names(top) == ["share", "usb"] + assert {i.name: i.available for i in top} == {"share": True, "usb": False} + + +def test_a_folder_holds_its_folders_and_files(): + tree = Tree.from_index(INDEX) + assert _names(tree.children("share")) == ["2024", "empty", "notes.txt"] + assert _names(tree.children("share/2024")) == ["late", "x.flac"] + + +def test_an_unavailable_root_shows_nothing_unless_asked(): + tree = Tree.from_index(INDEX) + assert tree.children("usb") == [] + assert _names(tree.children("usb/old", include_unavailable=True)) == ["z.txt"] + + +def test_walk_visits_every_folder(): + tree = Tree.from_index(INDEX) + assert [p for p, _ in tree.walk("share")] == [ + "share", "share/2024", "share/2024/late", "share/empty"] + + +def test_a_missing_folder_is_not_found(): + with pytest.raises(CliError) as e: + Tree.from_index(INDEX).children("share/nope") + assert e.value.code == NOT_FOUND + + +def test_a_file_is_found_by_its_path(): + tree = Tree.from_index(INDEX) + assert tree.file("share/2024/x.flac")["id"] == "b" * 64 + assert tree.file("share/2024/nope") is None + + +def test_control_characters_and_bidi_overrides_are_escaped(): + assert safe("evil\x1b[31mred") == "evil\\x1b[31mred" + assert safe("a\nb\x7fc\x9b") == "a\\x0ab\\x7fc\\x9b" + assert safe("photogpj.exe") == "photo\\u202egpj.exe" + assert safe("été 🎵 l'ouest") == "été 🎵 l'ouest" + + +def test_sizes(): + assert human_size(6) == "6 B" + assert human_size(2_500_000) == "2.4 MB" diff --git a/packages/meshbay-cli/tests/test_manpage.py b/packages/meshbay-cli/tests/test_manpage.py new file mode 100644 index 0000000..ebd71c8 --- /dev/null +++ b/packages/meshbay-cli/tests/test_manpage.py @@ -0,0 +1,70 @@ +""" +The man page is generated from the parser; these keep the generator honest. + +An option added to an action must reach the page, every exit code must be +explained, and troff must read the result without a warning. +""" + +import argparse +import os +import shutil +import subprocess + +import pytest +from meshbay_cli import __main__ as cli +from meshbay_cli import errors, manpage + + +def _section(page: str, start: str) -> str: + return page.split(start + "\n", 1)[1].split("\n.S", 1)[0] + + +def _entry(option: str) -> str: + """How an option opens its entry: `.TP`, then the option in bold.""" + return f"\\fB{manpage._roff(option)}\\fR" + + +def test_every_action_and_option_is_in_its_section(): + page = manpage.render() + parser = cli.build_parser() + glob = _section(page, ".SH GLOBAL OPTIONS") + for a in parser._actions: + for option in a.option_strings: + assert _entry(option) in glob, f"global {option} missing" + sub = next(a for a in parser._actions if isinstance(a, argparse._SubParsersAction)) + assert list(sub.choices) == list(cli.ACTIONS) + for name, sp in sub.choices.items(): + own = _section(page, f".SS {name}") + for a in sp._actions: + if isinstance(a, argparse._HelpAction): + continue + if a.option_strings and _entry(a.option_strings[0]) in glob: + continue # a global one, listed once + if a.option_strings: + for option in a.option_strings: + assert _entry(option) in own, f"{name} {option} missing" + elif a.choices: + for choice in a.choices: + assert _entry(choice) in own, f"{name} {choice} missing" + else: + meta = a.metavar or a.dest.upper() + assert f"\\fI{manpage._roff(meta)}\\fR" in own, f"{name} {meta} missing" + + +def test_every_exit_code_is_explained(): + codes = {v for k, v in vars(errors).items() if k.isupper() and isinstance(v, int)} + assert codes | {0} == set(manpage.EXIT_CODES) + + +def test_names_and_text_cannot_break_the_page(): + assert manpage._roff(".hidden") == "\\&.hidden" + assert manpage._roff("a\\b — c-d") == "a\\eb \\[u2014] c\\-d" + + +@pytest.mark.skipif(not shutil.which("groff"), reason="groff not installed") +def test_troff_reads_it_without_a_warning(tmp_path): + page = tmp_path / "meshbay-cli.1" + page.write_text(manpage.render(), encoding="ascii") # nothing past ASCII left + r = subprocess.run(["groff", "-man", "-Tutf8", "-ww", "-z", str(page)], + capture_output=True, text=True, env={**os.environ, "LC_ALL": "C"}) + assert r.returncode == 0 and not r.stderr, r.stderr diff --git a/packages/meshbay-cli/tests/test_remote_paths.py b/packages/meshbay-cli/tests/test_remote_paths.py new file mode 100644 index 0000000..5192e7b --- /dev/null +++ b/packages/meshbay-cli/tests/test_remote_paths.py @@ -0,0 +1,77 @@ +"""Remote paths and group handles — `docs/MESHBAY_DESIGN.md` §8.4, `group-link.js`.""" + +import pytest +from meshbay_cli.client.paths import encode_name, handle, parse, resolve_group +from meshbay_cli.errors import NOT_FOUND, USAGE, CliError + +GROUPS = [ + {"id": "1", "name": "Family", "owner_username": "alice"}, + {"id": "2", "name": "Family", "owner_username": "bob"}, + {"id": "3", "name": "Some Saga/Box @Set", "owner_username": "alice"}, + {"id": "4", "name": "me@home", "owner_username": "carol"}, + {"id": "5", "name": "Work", "owner_username": "Dave"}, +] + + +def test_nothing_or_a_slash_is_every_group(): + assert parse("") is None + assert parse("/") is None + + +def test_a_typed_path_decodes_only_the_slash_and_the_percent(): + p = parse("Some Saga%2FBox @Set@alice/share/100% done.txt") + assert p.group == "Some Saga/Box @Set@alice" + assert p.parts == ["share", "100% done.txt"] + assert parse("G/a%252Fb").parts == ["a%2Fb"] + + +def test_a_link_copied_from_the_application_is_decoded_in_full(): + p = parse("https://hub.example/#/Some%20Saga%2FBox%20@Set@alice/share/b%C3%A9b%C3%A9.jpg") + assert p.from_link + assert p.group == "Some Saga/Box @Set@alice" + assert p.parts == ["share", "bébé.jpg"] + + +def test_dot_segments_are_refused(): + with pytest.raises(CliError) as e: + parse("Family/share/../etc") + assert e.value.code == USAGE + + +def test_names_are_nfc(): + decomposed = "été" + assert parse(f"G/{decomposed}").parts == ["été"] + + +def test_the_owner_is_after_the_last_at(): + assert resolve_group(GROUPS, "Family@bob")["id"] == "2" + assert resolve_group(GROUPS, "Some Saga/Box @Set@alice")["id"] == "3" + assert resolve_group(GROUPS, "me@home@carol")["id"] == "4" + + +def test_a_name_with_an_at_of_its_own_still_resolves_without_owner(): + assert resolve_group(GROUPS, "me@home")["id"] == "4" + + +def test_names_and_owners_compare_without_case(): + assert resolve_group(GROUPS, "some saga/box @set")["id"] == "3" + assert resolve_group(GROUPS, "work@dave")["id"] == "5" + + +def test_two_groups_of_one_name_ask_for_the_owner(): + with pytest.raises(CliError) as e: + resolve_group(GROUPS, "Family") + assert e.value.code == USAGE + assert "Family@alice" in e.value.message and "Family@bob" in e.value.message + + +def test_an_unknown_group_is_not_found(): + with pytest.raises(CliError) as e: + resolve_group(GROUPS, "Nope") + assert e.value.code == NOT_FOUND + + +def test_a_handle_can_be_typed_back(): + for g in GROUPS: + assert resolve_group(GROUPS, parse(handle(g)).group)["id"] == g["id"] + assert encode_name("100%/x") == "100%25%2Fx" diff --git a/packages/meshbay-cli/tests/test_selftest.py b/packages/meshbay-cli/tests/test_selftest.py new file mode 100644 index 0000000..ea49fa1 --- /dev/null +++ b/packages/meshbay-cli/tests/test_selftest.py @@ -0,0 +1,16 @@ +"""The frozen build's self-test, run here from source so it cannot rot unseen.""" + +from meshbay_cli import __main__ as cli + + +def test_the_self_test_passes_from_source(monkeypatch, tmp_path, capsys): + monkeypatch.setenv("MESHBAY_CLI_SELFTEST", "1") + monkeypatch.setenv("MESHBAY_CLI_HOME", str(tmp_path)) # restored after + assert cli.main([]) == 0 + assert "self-test ok" in capsys.readouterr().out + + +def test_it_is_not_reachable_with_arguments(monkeypatch, capsys): + monkeypatch.setenv("MESHBAY_CLI_SELFTEST", "1") + assert cli.main(["--version"]) == 0 + assert "self-test" not in capsys.readouterr().out diff --git a/packages/meshbay-cli/tests/test_store_and_session.py b/packages/meshbay-cli/tests/test_store_and_session.py new file mode 100644 index 0000000..42458e2 --- /dev/null +++ b/packages/meshbay-cli/tests/test_store_and_session.py @@ -0,0 +1,276 @@ +""" +The local store, and the hub session it keeps. + +The case that matters most is two runs renewing at once: the hub rotates refresh +tokens and treats a spent one as theft, so a second renewal with the token the +first has just spent would sign the account out everywhere. +""" + +import asyncio +import os +import sys +import time + +import keyring +import pytest +from fake_hub import PASSPHRASE, USERNAME, FakeHub +from keyring.backend import KeyringBackend +from meshbay_cli.client.account import Account, no_passphrase +from meshbay_cli.client.store import KEY_FILE, KEY_OS, Store +from meshbay_cli.errors import AUTH, CliError + + +class MemoryKeyring(KeyringBackend): + priority = 1 + + def __init__(self): + super().__init__() + self.items: dict[tuple[str, str], str] = {} + + def get_password(self, service, username): + return self.items.get((service, username)) + + def set_password(self, service, username, password): + self.items[(service, username)] = password + + def delete_password(self, service, username): + self.items.pop((service, username), None) + + +@pytest.fixture +def memory_keyring(): + backend = MemoryKeyring() + previous = keyring.get_keyring() + keyring.set_keyring(backend) + yield backend + keyring.set_keyring(previous) + + +def _saved(root, secrets=None, state=None) -> Store: + store = Store(root) + store.state = {"hub": "https://hub.test", "username": USERNAME, "user_id": "u", + **(state or {})} + store.secrets = {"access_token": "old", "access_exp": 0, "refresh_token": "rt0", + **(secrets or {})} + store.save(new=True) + return store + + +# ── The store ──────────────────────────────────────────────────────────────── + +def test_without_key_storage_the_key_is_a_private_file(tmp_path): + store = _saved(tmp_path / "s") + assert store.state["key_store"] == KEY_FILE + if sys.platform != "win32": + for name in ("store.key", "secrets.enc", "state.json"): + assert os.stat(tmp_path / "s" / name).st_mode & 0o077 == 0, name + again = Store(tmp_path / "s") + assert again.load() and again.secrets["refresh_token"] == "rt0" + + +def test_with_key_storage_the_key_is_kept_there(tmp_path, memory_keyring): + a = _saved(tmp_path / "a") + _saved(tmp_path / "b") + assert a.state["key_store"] == KEY_OS and not (tmp_path / "a" / "store.key").exists() + # One entry per store: a second store never overwrites the first's key. + assert len(memory_keyring.items) == 2 + assert Store(tmp_path / "a").load() and Store(tmp_path / "b").load() + a.erase() + assert len(memory_keyring.items) == 1 + assert not Store(tmp_path / "a").exists() + + +def test_secrets_are_not_readable_in_the_file(tmp_path): + _saved(tmp_path / "s", secrets={"refresh_token": "a-very-recognisable-token"}) + assert b"a-very-recognisable-token" not in (tmp_path / "s" / "secrets.enc").read_bytes() + + +def test_a_store_whose_key_storage_vanished_says_so(tmp_path, memory_keyring): + _saved(tmp_path / "s") + keyring.set_keyring(__import__("keyring.backends.fail", fromlist=["Keyring"]).Keyring()) + with pytest.raises(CliError) as e: + Store(tmp_path / "s").load() + assert e.value.code == AUTH and "OS key storage" in e.value.message + + +def test_peek_reads_the_state_without_the_key(tmp_path): + _saved(tmp_path / "s") + (tmp_path / "s" / "store.key").unlink() + assert Store(tmp_path / "s").peek()["username"] == USERNAME + + +# ── Renewal ────────────────────────────────────────────────────────────────── + +def _open(root, hub) -> Account: + return Account.open(hub, None, no_passphrase, Store(root)) + + +async def test_two_runs_renewing_at_once_spend_the_token_once(tmp_path): + hub = FakeHub() + first = hub._session() + _saved(tmp_path / "s", secrets={"refresh_token": first["refresh_token"]}) + hub.refresh_delay = 0.2 # wide enough for the second run to arrive meanwhile + + a, b = _open(tmp_path / "s", hub), _open(tmp_path / "s", hub) + ta, tb = await asyncio.gather(a.token(), b.token()) + + assert not hub.reuse_detected + assert hub.refresh_calls == 1 + assert ta == tb and ta in hub.access + # And the stored token is the live one: a third run renews without incident. + c = _open(tmp_path / "s", hub) + c.secrets["access_exp"] = 0 + await c.token(rejected=c.secrets["access_token"]) + assert not hub.reuse_detected + + +async def test_a_token_still_fresh_is_used_as_it_is(tmp_path): + hub = FakeHub() + s = hub._session() + _saved(tmp_path / "s", secrets={"access_token": s["access_token"], + "access_exp": time.time() + 3600, + "refresh_token": s["refresh_token"]}) + assert await _open(tmp_path / "s", hub).token() == s["access_token"] + assert hub.refresh_calls == 0 + + +async def test_a_refused_refresh_falls_back_to_the_device_key(tmp_path): + from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + from meshbay_cli.client.keys import b64 + hub = FakeHub() + sk = Ed25519PrivateKey.generate() + hub.devices["dev-1"] = b64(sk.public_key().public_bytes_raw()) + _saved(tmp_path / "s", secrets={"refresh_token": "revoked", + "device_sk": b64(sk.private_bytes_raw())}) + token = await _open(tmp_path / "s", hub).token() + assert token in hub.access + + +async def test_a_session_that_ended_asks_for_login(tmp_path): + hub = FakeHub() + _saved(tmp_path / "s", secrets={"refresh_token": "revoked"}) + with pytest.raises(CliError) as e: + await _open(tmp_path / "s", hub).token() + assert e.value.code == AUTH and "meshbay-cli login" in e.value.message + + +async def test_a_401_renews_once_and_replays(tmp_path): + hub = FakeHub() + s = hub._session() + _saved(tmp_path / "s", secrets={"access_token": "revoked-by-the-hub", + "access_exp": time.time() + 3600, + "refresh_token": s["refresh_token"]}) + groups = await _open(tmp_path / "s", hub).call(hub.my_groups) + assert groups[0]["name"] == "Family" and hub.refresh_calls == 1 + + +# ── Signing in ─────────────────────────────────────────────────────────────── + +async def test_sign_in_derives_the_keys_and_registers_one_device(tmp_path): + hub = FakeHub() + acc = Account(hub, USERNAME, lambda: PASSPHRASE) + await acc.sign_in(remember=True) + assert acc.master and acc.legacy and acc.user_id + assert len(hub.devices) == 1 and acc.state["device_id"] == "dev-1" + # Signing in again keeps the same device key rather than adding one. + await acc.sign_in(remember=True) + assert len(hub.devices) == 1 + + +async def test_a_wrong_passphrase_is_an_auth_error(): + with pytest.raises(CliError) as e: + await Account(FakeHub(), USERNAME, lambda: "wrong").sign_in(remember=False) + assert e.value.code == AUTH + + +async def test_a_renewal_by_passphrase_is_what_gets_stored(tmp_path): + """The fallback replaces the session wholesale; the store must get the new one.""" + hub = FakeHub() + _saved(tmp_path / "s", secrets={"refresh_token": "revoked"}) + acc = Account.open(hub, None, lambda: PASSPHRASE, Store(tmp_path / "s")) + await acc.token() + again = Store(tmp_path / "s") + again.load() + assert again.secrets["refresh_token"] == acc.secrets["refresh_token"] + assert again.secrets["refresh_token"] in hub.refresh_tokens + assert again.secrets.get("master") + + +async def test_a_run_signed_out_meanwhile_says_so(tmp_path): + hub = FakeHub() + _saved(tmp_path / "s") + acc = _open(tmp_path / "s", hub) + Store(tmp_path / "s").erase() + with pytest.raises(CliError) as e: + await acc.token() + assert e.value.code == AUTH and "another run" in e.value.message + + +async def test_a_pending_device_request_is_kept_then_adopted(tmp_path): + from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey + from meshbay_cli.client.keys import Identity + hub = FakeHub() + _saved(tmp_path / "s") + acc = _open(tmp_path / "s", hub) + ident = Identity(Ed25519PrivateKey.generate(), X25519PrivateKey.generate()) + await acc.keep_pending("node-pk", ident, "ABCD-EFGH", time.time() + 600) + + # A later run shows the same code and joins with the same keys. + again = _open(tmp_path / "s", hub) + pending = again.pending_identity("node-pk") + assert pending.code == "ABCD-EFGH" and pending.identity.pk_ed == ident.pk_ed + assert again.identity("node-pk") is None + await again.adopt_pending("node-pk") + assert again.pending_identity("node-pk") is None + assert _open(tmp_path / "s", hub).identity("node-pk").pk_ed == ident.pk_ed + + +async def test_an_expired_device_request_is_not_reused(tmp_path): + from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey + from meshbay_cli.client.keys import Identity + hub = FakeHub() + _saved(tmp_path / "s") + acc = _open(tmp_path / "s", hub) + ident = Identity(Ed25519PrivateKey.generate(), X25519PrivateKey.generate()) + await acc.keep_pending("node-pk", ident, "ABCD-EFGH", time.time() - 1) + assert _open(tmp_path / "s", hub).pending_identity("node-pk") is None + + +async def test_a_machine_waiting_for_approval_retries_until_it_is_approved(monkeypatch): + from meshbay_cli.client import groups + from meshbay_cli.client.session import DeviceLinkPending + monkeypatch.setattr(groups, "LINK_POLL", 0) + tries, shown = [], [] + + async def connect(account, group_id, node): + tries.append(node["node_id"]) + if len(tries) < 3: + raise DeviceLinkPending("ABCD-EFGH", time.time() + 600) + return "session" + monkeypatch.setattr(groups, "connect", connect) + + class Acc: + def on_device_code(self, group, pending, *, first): + shown.append((pending.link_code, first)) + return True + got = await groups._connect_linking(Acc(), {"id": "g"}, {"node_id": "n1"}) + assert got == "session" and tries == ["n1", "n1", "n1"] + assert shown == [("ABCD-EFGH", True), ("ABCD-EFGH", False)] + + +async def test_without_a_terminal_the_code_is_shown_and_the_run_ends(monkeypatch): + from meshbay_cli.client import groups + from meshbay_cli.client.session import DeviceLinkPending + + async def connect(account, group_id, node): + raise DeviceLinkPending("ABCD-EFGH", time.time() + 600) + monkeypatch.setattr(groups, "connect", connect) + + class Acc: + def on_device_code(self, group, pending, *, first): + return False + with pytest.raises(DeviceLinkPending) as e: + await groups._connect_linking(Acc(), {"id": "g"}, {"node_id": "n1"}) + assert e.value.code == 5 and e.value.link_code == "ABCD-EFGH" diff --git a/packages/meshbay-client/build/appx-extensions.xml b/packages/meshbay-client/build/appx-extensions.xml index 3326dd8..db6e345 100644 --- a/packages/meshbay-client/build/appx-extensions.xml +++ b/packages/meshbay-client/build/appx-extensions.xml @@ -13,3 +13,10 @@ <uap5:ExecutionAlias Alias="meshbay-node.exe" /> </uap5:AppExecutionAlias> </uap5:Extension> + <!-- `meshbay-cli` in a terminal, the command-line client: the same alias + mechanism, for the same reason. --> + <uap5:Extension Category="windows.appExecutionAlias" Executable="app\resources\node-runtime\meshbay-cli.exe" EntryPoint="Windows.FullTrustApplication"> + <uap5:AppExecutionAlias> + <uap5:ExecutionAlias Alias="meshbay-cli.exe" /> + </uap5:AppExecutionAlias> + </uap5:Extension> diff --git a/packages/meshbay-client/build/appx-manifest.xml b/packages/meshbay-client/build/appx-manifest.xml index a8dba41..c097fdd 100644 --- a/packages/meshbay-client/build/appx-manifest.xml +++ b/packages/meshbay-client/build/appx-manifest.xml @@ -44,10 +44,11 @@ ${extensions} </Application> </Applications> - <!-- The node accepts connections from peers (WebRTC, QUIC, casting). Rules - declared here are created by Windows at install with no administrator - prompt, follow the executable's versioned path on every update, and go - with the package. A rule the app made itself named that path and was + <!-- The node accepts connections from peers (WebRTC, QUIC, casting), and + the command-line client does during a transfer (WebRTC). Rules declared + here are created by Windows at install with no administrator prompt, + follow the executable's versioned path on every update, and go with the + package. A rule the app made itself named that path and was stale after the next update; the network capabilities create rules for sandboxed apps only, which a full-trust process is not. Both measured on a real install. --> @@ -64,5 +65,10 @@ <desktop2:Rule Direction="in" IPProtocol="UDP" Profile="all" /> </desktop2:FirewallRules> </desktop2:Extension> + <desktop2:Extension Category="windows.firewallRules"> + <desktop2:FirewallRules Executable="app\resources\node-runtime\meshbay-cli.exe"> + <desktop2:Rule Direction="in" IPProtocol="UDP" Profile="all" /> + </desktop2:FirewallRules> + </desktop2:Extension> </Extensions> </Package> diff --git a/packages/meshbay-hub/tests/test_bundle_kdf_parity.py b/packages/meshbay-hub/tests/test_bundle_kdf_parity.py index a4bee43..358d6ce 100644 --- a/packages/meshbay-hub/tests/test_bundle_kdf_parity.py +++ b/packages/meshbay-hub/tests/test_bundle_kdf_parity.py @@ -158,7 +158,7 @@ const fp = async (key) => Buffer.from(await webcrypto.subtle.encrypt( Buffer.from('ed-private'), Buffer.from('x-private'), kNode, { userId: v.user_id, nodePk: v.node_pk, pepperVersion: 1 }); out.push({ node: await fp(kNode), playlists: await fp(await derivePlaylistKey(sk.v3)), - bundle }); + bundle, auth: await K.deriveAuthKey(v.password, v.username) }); } process.stdout.write(JSON.stringify(out)); })().catch((e) => { console.error(e); process.exit(1); }); @@ -231,3 +231,37 @@ def test_an_mbk3_bundle_opens_from_the_specification(chain): plain = json.loads(AESGCM(_reference(v)[0]).decrypt(raw[5:17], raw[17:], aad)) assert base64.b64decode(plain["skEd"]) == b"ed-private" assert base64.b64decode(plain["skX"]) == b"x-private" + + +# ── The command-line client derives the same keys ────────────────────────── +# +# `meshbay-cli` runs its own Python copy of the chain (it is a client, not a +# test reference), so it is held here to the page, as the reference is. + +def test_the_cli_derives_what_the_page_derives(chain): + cli_keys = pytest.importorskip("meshbay_cli.client.keys") + import base64 + vectors, js = chain + for v, got in zip(vectors, js): + assert cli_keys.auth_key(v["username"], v["password"]) == got["auth"], ( + f"auth_key disagrees for {v['username']!r}: the CLI could not sign in") + a = cli_keys.passphrase_half(v["username"], v["password"]) + assert a.hex() == _python_hash(v["username"], v["password"]) + m = cli_keys.bundle_master_key(a, base64.b64decode(v["pepper"]), v["user_id"]) + assert _fp(cli_keys.node_bundle_key(m, v["node_pk"])) == got["node"] + + +def test_the_cli_opens_a_bundle_the_page_sealed(chain): + cli_keys = pytest.importorskip("meshbay_cli.client.keys") + import base64 + vectors, js = chain + for v, got in zip(vectors, js): + a = cli_keys.passphrase_half(v["username"], v["password"]) + m = cli_keys.bundle_master_key(a, base64.b64decode(v["pepper"]), v["user_id"]) + keys = cli_keys.open_bundle(got["bundle"], master=m, legacy=None, + user_id=v["user_id"], node_pk=v["node_pk"]) + assert base64.b64decode(keys["skEd"]) == b"ed-private" + # Bound to its account: served for another, it does not open. + with pytest.raises(cli_keys.BundleError): + cli_keys.open_bundle(got["bundle"], master=m, legacy=None, + user_id="someone-else", node_pk=v["node_pk"]) diff --git a/packages/meshbay-node/tests/test_licensing.py b/packages/meshbay-node/tests/test_licensing.py index 239633d..0cb85cc 100644 --- a/packages/meshbay-node/tests/test_licensing.py +++ b/packages/meshbay-node/tests/test_licensing.py @@ -11,6 +11,7 @@ LGPL. import importlib.util import json import re +import sys import tomllib from importlib import metadata from pathlib import Path @@ -22,7 +23,9 @@ VENDOR = STATIC / "vendor" DESKTOP = PACKAGES / "meshbay-client" / "src" ANDROID = PACKAGES / "meshbay-android" / "app" / "src" / "main" / "kotlin" KEYS = ANDROID / "org" / "meshbay" / "client" / "keys" +CLI = PACKAGES / "meshbay-cli" / "src" / "meshbay_cli" SPDX_LGPL = "// SPDX-License-Identifier: LGPL-3.0-or-later\n" +SPDX_LGPL_PY = "# SPDX-License-Identifier: LGPL-3.0-or-later\n" # The protocol layer in the clients (README, "Licence"): what a program needs to # speak to a hub and a node, under the LGPL in every language it exists in. LGPL_FILES = sorted( @@ -31,11 +34,13 @@ LGPL_FILES = sorted( + sorted(STATIC.glob("transport-*.js")) + [DESKTOP / f for f in ("keyring.js", "transcripts.js", "argon2-wasm.js")] + [KEYS / f for f in ("Kdf.kt", "Keyring.kt", "Transcripts.kt")] + + [CLI / "client" / "keys.py"] ) EXPECTED = { "meshbay-common": ("LGPL-3.0-or-later", ["COPYING", "COPYING.LESSER"]), "meshbay-hub": ("AGPL-3.0-or-later", ["LICENSE"]), "meshbay-node": ("AGPL-3.0-or-later", ["LICENSE"]), + "meshbay-cli": ("AGPL-3.0-or-later", ["LICENSE"]), } @@ -63,7 +68,7 @@ def test_licence_texts_are_the_right_ones(): agpl = (ROOT / "LICENSE").read_text(encoding="utf-8") assert "GNU AFFERO GENERAL PUBLIC LICENSE" in agpl and "Version 3" in agpl # Copies, because a wheel's license-files cannot reach outside its package. - for pkg in ("meshbay-hub", "meshbay-node"): + for pkg in ("meshbay-hub", "meshbay-node", "meshbay-cli"): assert (PACKAGES / pkg / "LICENSE").read_text(encoding="utf-8") == agpl, pkg common = PACKAGES / "meshbay-common" assert "GNU LESSER GENERAL PUBLIC LICENSE" in (common / "COPYING.LESSER").read_text( @@ -73,7 +78,8 @@ def test_licence_texts_are_the_right_ones(): def test_rpm_specs_and_the_client_agree_with_the_packages(): - for pkg in ("meshbay-common", "meshbay-hub", "meshbay-node", "meshbay-client"): + for pkg in ("meshbay-common", "meshbay-hub", "meshbay-node", "meshbay-cli", + "meshbay-client"): spec = (ROOT / "packaging" / "rpm" / f"{pkg}.spec").read_text(encoding="utf-8") want = EXPECTED.get(pkg, ("AGPL-3.0-or-later",))[0] assert re.search(rf"^License:\s+{re.escape(want)}\s*$", spec, re.M), pkg @@ -136,14 +142,18 @@ def test_every_vendored_file_has_its_provenance_and_licence(): def _sources(): - for tree, pattern in ((STATIC, "*.js"), (DESKTOP, "*.js"), (ANDROID, "**/*.kt")): + for tree, pattern in ((STATIC, "*.js"), (DESKTOP, "*.js"), (ANDROID, "**/*.kt"), + (CLI, "**/*.py")): for f in tree.glob(pattern): if "vendor" not in f.parts and "locales" not in f.parts: yield f def test_the_lgpl_files_are_exactly_the_ones_that_say_so(): - marked = sorted(f for f in _sources() if f.read_text(encoding="utf-8").startswith(SPDX_LGPL)) + def spdx(f: Path) -> str: + return SPDX_LGPL_PY if f.suffix == ".py" else SPDX_LGPL + + marked = sorted(f for f in _sources() if f.read_text(encoding="utf-8").startswith(spdx(f))) assert marked == LGPL_FILES @@ -202,6 +212,22 @@ def test_the_lgpl_layer_depends_on_nothing_under_the_agpl(): } for f in LGPL_FILES: src = f.read_text(encoding="utf-8") + if f.suffix == ".py": + # The standard library, meshbay-common (the LGPL layer itself) and + # permissive packages; never the client package it sits in. + for mod in re.findall(r"^\s*(?:from|import)\s+([\w.]+)", src, re.M): + top_name = mod.split(".")[0] + assert top_name and top_name != "meshbay_cli" and not mod.startswith("."), ( + f"{f.name} imports {mod}") + if top_name in sys.stdlib_module_names or top_name == "meshbay_common": + continue + dist = metadata.packages_distributions()[top_name][0] + md = metadata.distribution(dist).metadata + label = " ".join([md.get("License-Expression") or "", md.get("License") or ""] + + (md.get_all("Classifier") or [])) + assert "GPL" not in label, f"{f.name} imports {mod} ({dist}: {label[:80]})" + assert not re.search(r"^\s*from\s+\.", src, re.M), f"{f.name} has a relative import" + continue if f.suffix == ".kt": for imp in re.findall(r"^import (org\.meshbay\.[\w.]+)", src, re.M): owner = ( diff --git a/packages/meshbay-node/tests/test_packaging_win.py b/packages/meshbay-node/tests/test_packaging_win.py index 69eefc8..ed8104f 100644 --- a/packages/meshbay-node/tests/test_packaging_win.py +++ b/packages/meshbay-node/tests/test_packaging_win.py @@ -173,7 +173,10 @@ def test_the_frozen_exe_carries_a_version_resource(): spec = (WIN / "meshbay-node.spec").read_text(encoding="utf-8") assert "VSVersionInfo" in spec assert "version=_version_info" in spec, "EXE() is not given the version resource" - assert 'StringStruct("ProductName", "MeshBay Node")' in spec + assert '_version_info = _version_resource("meshbay-node", "MeshBay Node")' in spec + assert 'StringStruct("ProductName", description)' in spec + # The command-line client beside it, under its own name. + assert 'version=_version_resource("meshbay-cli", "MeshBay command-line client")' in spec assert '_pkg_version("meshbay-node")' in spec, "version is hardcoded, not read from the package" diff --git a/packaging/README.md b/packaging/README.md index 6081fc7..55cc39a 100644 --- a/packaging/README.md +++ b/packaging/README.md @@ -1,12 +1,13 @@ # MeshBay — Packaging -Four packages, installed under `/opt/<PACKAGE_NAME>/`: +Five packages, installed under `/opt/<PACKAGE_NAME>/`: | Package | RPM spec | DEB control | Description | |---|---|---|---| | `meshbay-common` | `rpm/meshbay-common.spec` | `deb/meshbay-common/` | Shared Python venv + all pip deps | | `meshbay-hub` | `rpm/meshbay-hub.spec` | `deb/meshbay-hub/` | Hub server (FastAPI + PostgreSQL) | | `meshbay-node` | `rpm/meshbay-node.spec` | `deb/meshbay-node/` | Node daemon + local UI | +| `meshbay-cli` | `rpm/meshbay-cli.spec` | `deb/meshbay-cli/` | Command-line client (a member's files from a terminal) + its man page | | `meshbay-client` | `rpm/meshbay-client.spec` | `deb/meshbay-client/` | Electron desktop app | ## Building packages @@ -22,7 +23,7 @@ bash packaging/build/build-packages.sh rpm bash packaging/build/build-packages.sh deb --arm64 ``` -Output: `meshbay-{common,hub,node,client}_<version>_{arch}.{deb,rpm}` in +Output: `meshbay-{common,hub,node,cli,client}_<version>_{arch}.{deb,rpm}` in `/tmp/meshbay-build/out/` (override with `MESHBAY_BUILD_DIR`). The version comes from each package's `pyproject.toml`. @@ -33,12 +34,12 @@ Always install `meshbay-common` first (owns the shared Python venv). ```bash # Ubuntu/Debian sudo dpkg -i meshbay-common_*.deb -sudo dpkg -i meshbay-hub_*.deb meshbay-node_*.deb +sudo dpkg -i meshbay-hub_*.deb meshbay-node_*.deb meshbay-cli_*.deb sudo dpkg -i meshbay-client_*.deb # Fedora sudo rpm -ivh meshbay-common-*.rpm -sudo rpm -ivh meshbay-hub-*.rpm meshbay-node-*.rpm +sudo rpm -ivh meshbay-hub-*.rpm meshbay-node-*.rpm meshbay-cli-*.rpm sudo rpm -ivh meshbay-client-*.rpm ``` @@ -75,8 +76,33 @@ already remembers the hub address — under `videoDecodeProbe`, and ## Architecture meshbay-common owns the shared Python virtual environment at -`/opt/meshbay-common/venv/`. Hub and node install their code into that venv. -The client is a standalone Electron app at `/opt/meshbay-client/`. +`/opt/meshbay-common/venv/`. Hub, node and command-line client install their +code into that venv. The client is a standalone Electron app at +`/opt/meshbay-client/`. + +### meshbay-cli + +The command-line client is its own package, for a desktop that wants it and a +server that does not. `build-common.sh` installs it into the venv last, after +listing what the hub and the node need (`servers-site.txt`, +`servers-bin.txt` in the staging directory); `build-cli.sh` moves whatever that +install added — the client and the packages only it uses (Textual, Rich, +keyring, argon2-cffi, ...) — into `meshbay-cli`, so a server carries none of +them. The list of what moved is installed as `/opt/meshbay-cli/site-packages.list`: +the maintainer scripts compile those files once at install (an unprivileged +user cannot write bytecode into `/opt`, and Textual is slow to compile on every +run) and remove that bytecode before removal. The rpm's `%files` takes the +same entries from a list `build-cli.sh` writes (`meshbay-cli-root.files`), +since they change with every dependency upgrade. + +Its architecture is the machine's, not `all`: argon2-cffi's bindings are +compiled code. Its licence notices are in +`/opt/meshbay-cli/THIRD-PARTY-NOTICES.txt`. + +The man page, `meshbay-cli(1)`, is generated at build time from the client's +own argument parser (`python -m meshbay_cli.manpage`), so it lists the options +the program has; `tests/test_manpage.py` fails when an action or option misses +its section, and checks that troff reads it without a warning. No pip runs at install time — everything is pre-built. diff --git a/packaging/build/build-cli.sh b/packaging/build/build-cli.sh new file mode 100755 index 0000000..b697f10 --- /dev/null +++ b/packaging/build/build-cli.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Build the meshbay-cli package: the command-line client, and the Python +# packages only it needs, on top of meshbay-common. +# +# Expects build-common.sh to have run first: it installed the CLI last, after +# listing the servers' packages in servers-site.txt and servers-bin.txt, so +# everything not in those lists is the CLI's and moves here. +# +# Usage: ./build-cli.sh [staging-dir] +set -euo pipefail + +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +STAGING="${1:-/tmp/meshbay-build}" +COMMON_ROOT="$STAGING/meshbay-common-root" +ROOT="$STAGING/meshbay-cli-root" + +VENV="$COMMON_ROOT/opt/meshbay-common/venv" +PYVER=$(basename "$VENV"/lib/python3.*) +SITE="$VENV/lib/$PYVER/site-packages" + +VERSION=$(python3 -c " +import tomllib, pathlib +p = pathlib.Path('$REPO/packages/meshbay-cli/pyproject.toml') +print(tomllib.loads(p.read_text())['project']['version']) +") +echo "==> meshbay-cli $VERSION" + +[ -d "$VENV" ] || { echo "!! shared venv not found — run build-common.sh first" >&2; exit 1; } +[ -f "$STAGING/servers-site.txt" ] || { + echo "!! $STAGING/servers-site.txt missing — run build-common.sh first" >&2; exit 1; } +[ -d "$SITE/meshbay_cli" ] || { + echo "!! meshbay_cli not in the shared venv — was build-cli.sh already run?" >&2; exit 1; } + +rm -rf "$ROOT" +mkdir -p "$ROOT/opt/meshbay-cli" + +# Run from the staged venv with no bytecode written: whatever it imports +# would otherwise leave __pycache__ in files other packages own. +run_venv() { PYTHONDONTWRITEBYTECODE=1 "$VENV/bin/python" "$@"; } + +# --- Before moving anything: the notices and the man page ----------------- +# Both need the CLI importable, which it is only where build-common.sh put it. +echo " writing THIRD-PARTY-NOTICES.txt" +run_venv "$REPO/packaging/third_party_notices.py" \ + -o "$ROOT/opt/meshbay-cli/THIRD-PARTY-NOTICES.txt" meshbay-cli + +echo " generating the man page" +mkdir -p "$ROOT/usr/share/man/man1" +run_venv -m meshbay_cli.manpage | gzip -9n > "$ROOT/usr/share/man/man1/meshbay-cli.1.gz" +[ "$(gzip -dc "$ROOT/usr/share/man/man1/meshbay-cli.1.gz" | head -c 3)" = ".TH" ] || { + echo "!! the man page did not generate" >&2; exit 1; } + +# --- The CLI and its own dependencies, out of the common tree --------------- +# MOVE (not copy), as build-hub.sh and build-node.sh do: dpkg/rpm then know +# these files belong to meshbay-cli. The list is kept for the maintainer +# scripts, which remove the bytecode compiled next to these files. +CLI_SITE="$ROOT/opt/meshbay-common/venv/lib/$PYVER/site-packages" +CLI_BIN="$ROOT/opt/meshbay-common/venv/bin" +mkdir -p "$CLI_SITE" "$CLI_BIN" +: > "$ROOT/opt/meshbay-cli/site-packages.list" +while IFS= read -r entry; do + mv "$SITE/$entry" "$CLI_SITE/" + echo "$entry" >> "$ROOT/opt/meshbay-cli/site-packages.list" +done < <(LC_ALL=C comm -13 "$STAGING/servers-site.txt" <(LC_ALL=C ls -1A "$SITE")) +while IFS= read -r entry; do + mv "$VENV/bin/$entry" "$CLI_BIN/" +done < <(LC_ALL=C comm -13 "$STAGING/servers-bin.txt" <(LC_ALL=C ls -1A "$VENV/bin")) +echo " moved: $(tr '\n' ' ' < "$ROOT/opt/meshbay-cli/site-packages.list")" + +# Only the client is on PATH; the dependencies' own scripts (pygmentize, +# keyring, ...) stay inside the venv. +mkdir -p "$ROOT/usr/bin" +ln -sf /opt/meshbay-common/venv/bin/meshbay-cli "$ROOT/usr/bin/meshbay-cli" + +find "$ROOT" -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true + +# What the venv holds of this package, for the rpm spec's %files: the entries +# differ with every dependency upgrade, so no list is kept by hand. +( cd "$ROOT" && find opt/meshbay-common/venv/lib/*/site-packages opt/meshbay-common/venv/bin \ + -mindepth 1 -maxdepth 1 | LC_ALL=C sort | sed 's|^|/|' ) > "$ROOT.files" + +echo "==> meshbay-cli staging ready at $ROOT" diff --git a/packaging/build/build-common.sh b/packaging/build/build-common.sh index 5f119c6..2010d1b 100755 --- a/packaging/build/build-common.sh +++ b/packaging/build/build-common.sh @@ -29,9 +29,9 @@ echo " creating venv at $VENV_BUILD" python3 -m venv "$VENV_BUILD" "$VENV_BUILD/bin/pip" install --upgrade pip wheel 2>&1 | tail -1 -# --- Build wheels for all three packages ----------------------------------- +# --- Build wheels for every package ---------------------------------------- echo " building wheels" -for pkg in meshbay-common meshbay-hub meshbay-node; do +for pkg in meshbay-common meshbay-hub meshbay-node meshbay-cli; do "$VENV_BUILD/bin/pip" wheel \ --no-deps \ --wheel-dir "$WHEEL_DIR" \ @@ -44,6 +44,17 @@ echo " installing all packages + dependencies" --find-links "$WHEEL_DIR" \ meshbay-common meshbay-hub meshbay-node 2>&1 | tail -3 +# The command-line client goes in last, after a list of what the servers need: +# whatever its install adds is the CLI's alone, and build-cli.sh moves it into +# the meshbay-cli package, so a server carries no Textual. +SITE_BUILD=$(echo "$VENV_BUILD"/lib/python3.*/site-packages) +LC_ALL=C ls -1A "$SITE_BUILD" > "$STAGING/servers-site.txt" +LC_ALL=C ls -1A "$VENV_BUILD/bin" > "$STAGING/servers-bin.txt" +echo " installing meshbay-cli + its own dependencies" +"$VENV_BUILD/bin/pip" install \ + --find-links "$WHEEL_DIR" \ + meshbay-cli 2>&1 | tail -1 + # --- Third-party notices: every package the venv ships, with its licence ------- echo " writing THIRD-PARTY-NOTICES.txt" "$VENV_BUILD/bin/python" "$REPO/packaging/third_party_notices.py" \ @@ -64,7 +75,9 @@ for f in "$VENV_BUILD/bin"/*; do [ -f "$f" ] || continue [ -L "$f" ] && continue head -1 "$f" | grep -q "^#!" || continue - sed -i "1s|$ROOT||" "$f" + # Line 2 too: past 127 characters, pip writes a `#!/bin/sh` trampoline that + # names the interpreter on the next line. + sed -i "1,2s|$ROOT||" "$f" done sed -i "s|$ROOT||g" "$VENV_BUILD/pyvenv.cfg" diff --git a/packaging/build/build-packages.sh b/packaging/build/build-packages.sh index 8ca0e23..7bbf6b5 100755 --- a/packaging/build/build-packages.sh +++ b/packaging/build/build-packages.sh @@ -63,19 +63,23 @@ mkdir -p "$OUT" # --- Step 1: Build staging trees ------------------------------------------ -echo "--- [1/4] meshbay-common (shared venv + all deps) ---" +echo "--- [1/5] meshbay-common (shared venv + all deps) ---" bash "$SCRIPT_DIR/build-common.sh" "$STAGING" echo "" -echo "--- [2/4] meshbay-hub ---" +echo "--- [2/5] meshbay-hub ---" bash "$SCRIPT_DIR/build-hub.sh" "$STAGING" echo "" -echo "--- [3/4] meshbay-node ---" +echo "--- [3/5] meshbay-node ---" bash "$SCRIPT_DIR/build-node.sh" "$STAGING" echo "" -echo "--- [4/4] meshbay-client ---" +echo "--- [4/5] meshbay-cli ---" +bash "$SCRIPT_DIR/build-cli.sh" "$STAGING" + +echo "" +echo "--- [5/5] meshbay-client ---" bash "$SCRIPT_DIR/build-client.sh" "$STAGING" $ARM64_FLAG # --- Step 2: Package ------------------------------------------------------ @@ -143,6 +147,7 @@ if [ "$FORMAT" = "deb" ]; then build_deb meshbay-common build_deb meshbay-hub build_deb meshbay-node + build_deb meshbay-cli build_deb meshbay-client elif [ "$FORMAT" = "rpm" ]; then @@ -175,6 +180,7 @@ elif [ "$FORMAT" = "rpm" ]; then build_rpm meshbay-common build_rpm meshbay-hub build_rpm meshbay-node + build_rpm meshbay-cli build_rpm meshbay-client fi @@ -187,5 +193,5 @@ ls -lh "$OUT"/ echo "" echo "Install order:" echo " 1. meshbay-common" -echo " 2. meshbay-hub and/or meshbay-node" +echo " 2. meshbay-hub, meshbay-node and/or meshbay-cli" echo " 3. meshbay-client" diff --git a/packaging/deb/meshbay-cli/DEBIAN/control b/packaging/deb/meshbay-cli/DEBIAN/control new file mode 100644 index 0000000..6c64c09 --- /dev/null +++ b/packaging/deb/meshbay-cli/DEBIAN/control @@ -0,0 +1,17 @@ +Package: meshbay-cli +Version: __VERSION__ +Section: net +Priority: optional +Architecture: __ARCH__ +Maintainer: MeshBay Team <devel@meshbay.org> +Homepage: https://meshbay.org +Depends: meshbay-common (= __VERSION__) +Description: MeshBay command-line client — a member's groups and files, from a terminal + meshbay-cli signs in to a MeshBay hub, lists the groups the account belongs + to and their folders, and downloads and uploads files, directly with the + node serving each group and end-to-end encrypted. With no argument it opens + an interactive two-pane view. + . + Installs into the shared venv at /opt/meshbay-common/venv/, with the Python + packages only the client needs (Textual, keyring, argon2-cffi). Administering + a node is meshbay-node's business, not this package's. diff --git a/packaging/deb/meshbay-cli/DEBIAN/postinst b/packaging/deb/meshbay-cli/DEBIAN/postinst new file mode 100755 index 0000000..e09d713 --- /dev/null +++ b/packaging/deb/meshbay-cli/DEBIAN/postinst @@ -0,0 +1,17 @@ +#!/bin/sh +set -e + +# The client runs as an unprivileged user, who cannot write bytecode into +# /opt: compiled here once, or every run compiles Textual again in memory. +case "$1" in + configure) + if command -v py3compile >/dev/null 2>&1; then + site=$(echo /opt/meshbay-common/venv/lib/python3.*/site-packages) + while IFS= read -r entry; do + [ -d "$site/$entry" ] && py3compile "$site/$entry" 2>/dev/null || true + done < /opt/meshbay-cli/site-packages.list + fi + ;; +esac + +#DEBHELPER# diff --git a/packaging/deb/meshbay-cli/DEBIAN/prerm b/packaging/deb/meshbay-cli/DEBIAN/prerm new file mode 100755 index 0000000..96e5930 --- /dev/null +++ b/packaging/deb/meshbay-cli/DEBIAN/prerm @@ -0,0 +1,18 @@ +#!/bin/sh +set -e + +# The client's files live in meshbay-common's venv, whose own cleanup runs only +# after this package is gone: the bytecode compiled next to them is removed +# here, or dpkg cannot remove the directories it sits in. +case "$1" in + remove|upgrade|deconfigure) + site=$(echo /opt/meshbay-common/venv/lib/python3.*/site-packages) + while IFS= read -r entry; do + [ -n "$entry" ] && [ -d "$site/$entry" ] || continue + find "$site/$entry" -depth -type d -name __pycache__ \ + -exec rm -rf {} + 2>/dev/null || true + done < /opt/meshbay-cli/site-packages.list + ;; +esac + +#DEBHELPER# diff --git a/packaging/rpm/meshbay-cli.spec b/packaging/rpm/meshbay-cli.spec new file mode 100644 index 0000000..5d9aeb6 --- /dev/null +++ b/packaging/rpm/meshbay-cli.spec @@ -0,0 +1,69 @@ +Name: meshbay-cli +Version: __VERSION__ +Release: 1%{?dist} +Summary: MeshBay command-line client — a member's groups and files, from a terminal +# The Python packages only the client needs go with it, each under its own +# licence: /opt/meshbay-cli/THIRD-PARTY-NOTICES.txt lists them. +License: AGPL-3.0-or-later +URL: https://meshbay.org + +AutoReqProv: no +# Not noarch: argon2-cffi's bindings are compiled code, as meshbay-common's are. + +Requires: meshbay-common = %{version} + +%description +meshbay-cli signs in to a MeshBay hub, lists the groups the account belongs +to and their folders, and downloads and uploads files, directly with the node +serving each group and end-to-end encrypted. With no argument it opens an +interactive two-pane view. + +Installs into the shared venv at /opt/meshbay-common/venv/, with the Python +packages only the client needs (Textual, keyring, argon2-cffi). + +%install +cp -a %{_staging_root}/* %{buildroot}/ +install -Dm644 -t %{buildroot}%{_licensedir}/%{name} %{_repo_root}/LICENSE + +# After the whole transaction, so that meshbay-common's venv is in place. The +# client runs as an unprivileged user, who cannot write bytecode into /opt: +# compiled here once, or every run compiles Textual again in memory. +%posttrans +site=$(echo /opt/meshbay-common/venv/lib/python3.*/site-packages) +while IFS= read -r entry; do + [ -d "$site/$entry" ] || continue + /opt/meshbay-common/venv/bin/python -m compileall -q "$site/$entry" >/dev/null 2>&1 || true +done < /opt/meshbay-cli/site-packages.list + +# The client's files live in meshbay-common's venv: the bytecode compiled next +# to them is removed, or rpm cannot remove the directories it sits in. Before +# an upgrade too, while the list still names what the old version shipped. +%pre +if [ "$1" -ge 2 ] && [ -f /opt/meshbay-cli/site-packages.list ]; then + site=$(echo /opt/meshbay-common/venv/lib/python3.*/site-packages) + while IFS= read -r entry; do + [ -n "$entry" ] && [ -d "$site/$entry" ] || continue + find "$site/$entry" -depth -type d -name __pycache__ \ + -exec rm -rf {} + 2>/dev/null || true + done < /opt/meshbay-cli/site-packages.list +fi + +%preun +if [ "$1" -eq 0 ]; then + site=$(echo /opt/meshbay-common/venv/lib/python3.*/site-packages) + while IFS= read -r entry; do + [ -n "$entry" ] && [ -d "$site/$entry" ] || continue + find "$site/$entry" -depth -type d -name __pycache__ \ + -exec rm -rf {} + 2>/dev/null || true + done < /opt/meshbay-cli/site-packages.list +fi + +%files -f %{_staging_root}.files +%license %{_licensedir}/%{name} +/opt/meshbay-cli +/usr/bin/meshbay-cli +%{_mandir}/man1/meshbay-cli.1* + +%changelog +* Wed Oct 14 2026 MeshBay Team <devel@meshbay.org> - %{version}-1 +- First package: the command-line client diff --git a/packaging/win/README.md b/packaging/win/README.md index 4ce1459..89185fe 100644 --- a/packaging/win/README.md +++ b/packaging/win/README.md @@ -11,13 +11,15 @@ Linux). `meshbay-common` rides along inside the node runtime. ├─ MeshBay.exe Electron client ├─ resources\ │ ├─ app.asar src/ + ui/ (the interface ships in the package) -│ ├─ firewall.ps1 adds/removes the four inbound rules (see below) +│ ├─ firewall.ps1 adds/removes the five inbound rules (see below) │ ├─ service.ps1 install/remove/status/run/end the boot-time task │ ├─ service-mode.ps1 elevated helper: service.ps1 + firewall.ps1 in one UAC prompt │ └─ node-runtime\ │ ├─ meshbay-node.exe frozen daemon and CLI (PyInstaller onedir) │ ├─ meshbay-nodew.exe the same daemon without a console (the Store │ │ package's startup task runs it) +│ ├─ meshbay-cli.exe the command-line client, a member's files from +│ │ a terminal (Full and Store, not Light) │ ├─ _internal\ … its Python + deps (aiortc, av, aioquic, …) │ ├─ ffmpeg.exe, ffprobe.exe bundled by default, see Video (ffmpeg) below │ ├─ av*.dll, swscale/swresample*.dll what those two link against @@ -33,8 +35,9 @@ as this same signed-in user (S4U, see below), so it is the same profile either way — not LocalSystem/NetworkService, which would have none of this. `build/installer.nsh` also adds `…\resources\node-runtime` to the **per-user** -`Path` (`HKCU\Environment`) so `meshbay-node` works in a terminal, and takes it -back out on uninstall. New shells only — a `WM_SETTINGCHANGE` broadcast nudges +`Path` (`HKCU\Environment`) so `meshbay-node` and `meshbay-cli` work in a +terminal, and takes it back out on uninstall. The Store package declares an +execution alias for each instead (`build/appx-extensions.xml`). New shells only — a `WM_SETTINGCHANGE` broadcast nudges open ones. It uses stock `WordFunc.nsh` (the `EnVar` plugin is not in electron-builder's NSIS bundle). @@ -151,7 +154,7 @@ That runs [`build-win.ps1`](build-win.ps1): | 2 | `npm ci` + download Electron's Chromium | | 3 | bump Electron to the latest release (Chromium CVE policy; `-NoElectronBump` to skip) | | 4 | `npm run sync-ui` — copy the interface from `meshbay-hub/.../static` | -| 5 | [`build-node-runtime.ps1`](build-node-runtime.ps1) — PyInstaller freeze → `packages/meshbay-client/node-runtime/`, then [`smoke-node-runtime.ps1`](smoke-node-runtime.ps1) starts the frozen daemon in a throwaway profile and checks it answers with the right version and writes its log, and [`smoke-connection-process.py`](smoke-connection-process.py) starts both executables as connection processes (`connection_processes` above 1) and carries a connection through each | +| 5 | [`build-node-runtime.ps1`](build-node-runtime.ps1) — PyInstaller freeze → `packages/meshbay-client/node-runtime/`, then [`smoke-node-runtime.ps1`](smoke-node-runtime.ps1) starts the frozen daemon in a throwaway profile and checks it answers with the right version and writes its log, and [`smoke-connection-process.py`](smoke-connection-process.py) starts both executables as connection processes (`connection_processes` above 1) and carries a connection through each; `meshbay-cli.exe` is run too — `--version`, `groups --help`, and its self-test (`MESHBAY_CLI_SELFTEST=1 meshbay-cli`), which loads what a frozen build can miss without failing until run: the interactive mode drawn off-screen, keyring's backends, the WebRTC stack | | 6 | `electron-builder --win nsis` → `packages/meshbay-client/dist/MeshBay-Setup-<version>.exe` | ### Video (ffmpeg) @@ -215,7 +218,7 @@ publish their host candidate as an unresolvable `<uuid>.local` mDNS name that `aioice` discards — the browser always dials the node, never the reverse. So the node has to accept unsolicited inbound UDP from its peers. -**Windows Defender Firewall.** The setup wizard offers to add four inbound +**Windows Defender Firewall.** The setup wizard offers to add five inbound rules in one step — it needs one admin confirmation (`build/installer.nsh` runs `firewall.ps1` via NSIS `ExecShellWait "runas"`; the per-user install itself never elevates): @@ -224,6 +227,7 @@ itself never elevates): |---|---|---|---| | `MeshBay` | `MeshBay.exe` | UDP, any port | WebRTC (ICE checks touch the client too) | | `MeshBay Node` | `meshbay-node.exe` | UDP, any port | WebRTC (the node — unsolicited inbound, see above) | +| `MeshBay CLI` | `meshbay-cli.exe` | UDP, any port | WebRTC (the command-line client during a transfer; ICE checks touch it as they do the app) | | `MeshBay Cast` | `MeshBay.exe` | TCP 19550–19553 | LAN cast HTTP relay (`src/cast-relay.js`) | | `MeshBay Cast Discovery` | `MeshBay.exe` | UDP 5353 | Chromecast/Smart TV mDNS discovery (`bonjour-service`) | @@ -284,6 +288,12 @@ FFmpeg DLLs), `aioquic`, `pydantic_core`, `uvicorn`, `watchdog`, `guessit`, add the package to `COLLECT_ALL` / `HIDDEN` / `COPY_META` in the spec — that is the expected way the list grows. +The command-line client has its own analysis in the same spec +(`CLI_COLLECT_ALL`, `CLI_COPY_META`): Textual loads its widgets lazily, Rich its +Unicode tables by version, keyring its backends through entry points, which +is why keyring's metadata is copied. The build's self-test run catches what +those lists miss. + ## Not done - **Authenticode signing** — Phase 13.9. Unsigned installer triggers SmartScreen. diff --git a/packaging/win/build-node-runtime.ps1 b/packaging/win/build-node-runtime.ps1 index c6a8af1..28b96e8 100644 --- a/packaging/win/build-node-runtime.ps1 +++ b/packaging/win/build-node-runtime.ps1 @@ -1,10 +1,12 @@ <# .SYNOPSIS - Freeze the MeshBay node daemon into a relocatable Windows binary. + Freeze the MeshBay node daemon, and the command-line client beside it, + into relocatable Windows binaries. .DESCRIPTION - Produces packages/meshbay-client/node-runtime/meshbay-node.exe (+ its - onedir payload) with PyInstaller. electron-builder then carries that tree + Produces packages/meshbay-client/node-runtime/meshbay-node.exe, + meshbay-nodew.exe and meshbay-cli.exe (+ their shared onedir payload) + with PyInstaller. electron-builder then carries that tree into the NSIS installer as an extraResource (see package.json build.win). The frozen binary is what the desktop client spawns and what the W3 @@ -76,11 +78,12 @@ if ($verMajor -lt 3 -or ($verMajor -eq 3 -and $verMinor -lt 12)) { Step "python $verMajor.$verMinor ($Python)" # --- 2. dependencies ----------------------------------------------------- -Step "installing meshbay-common, meshbay-node and PyInstaller" +Step "installing meshbay-common, meshbay-node, meshbay-cli and PyInstaller" & $Python -m pip install --upgrade pip --quiet & $Python -m pip install --quiet ` (Join-Path $Repo "packages\meshbay-common") ` (Join-Path $Repo "packages\meshbay-node") ` + (Join-Path $Repo "packages\meshbay-cli") ` "pyinstaller>=6.10" ` "tzdata" # tzdata: Windows ships no IANA zone database, so zoneinfo (pulled in @@ -105,7 +108,7 @@ finally { } $frozen = Join-Path $pyiDist "meshbay-node" -foreach ($name in "meshbay-node.exe", "meshbay-nodew.exe") { +foreach ($name in "meshbay-node.exe", "meshbay-nodew.exe", "meshbay-cli.exe") { if (-not (Test-Path (Join-Path $frozen $name))) { throw "PyInstaller did not produce $name at $frozen" } @@ -124,7 +127,7 @@ Copy-Item (Join-Path $Repo "packages\meshbay-common\COPYING.LESSER") (Join-Path Copy-Item (Join-Path $Repo "packages\meshbay-common\COPYING") (Join-Path $frozen "LICENSE-GPL-3.0.txt") & $Python (Join-Path $Repo "packaging\third_party_notices.py") ` -o (Join-Path $frozen "THIRD-PARTY-NOTICES.txt") ` - meshbay-node tzdata --extra pyinstaller --with-python + meshbay-node meshbay-cli tzdata --extra pyinstaller --with-python if ($LASTEXITCODE -ne 0) { throw "third_party_notices.py failed" } # --- 4. ffmpeg (bundled by default) ----------------------------------- @@ -198,6 +201,29 @@ Step "smoke test: the frozen node as a connection process" & $Python (Join-Path $WinDir "smoke-connection-process.py") $exe (Join-Path $OutDir "meshbay-nodew.exe") if ($LASTEXITCODE -ne 0) { throw "a frozen connection process did not carry a connection" } +# The client is frozen from its own analysis, so what the daemon's checks +# proved says nothing about it: a module PyInstaller missed fails at run time +# only. --version and `groups --help` build the parser, which imports every +# action and the client library; the self-test loads what the parser does not +# -- the interactive mode drawn off-screen (Textual, Rich), keyring's backends +# (Windows Credential Manager), an aiortc offer. The CLI has the node's version +# (all packages share one). +Step "smoke test: meshbay-cli" +$cli = Join-Path $OutDir "meshbay-cli.exe" +$out = (& $cli --version 2>&1) -join "`n" +if ($LASTEXITCODE -ne 0 -or $out -notmatch "meshbay-cli $expect") { + throw "frozen meshbay-cli --version exited $LASTEXITCODE`n$out" +} +$out = (& $cli groups --help 2>&1) -join "`n" +if ($LASTEXITCODE -ne 0 -or $out -notmatch "List the groups") { + throw "frozen meshbay-cli groups --help exited $LASTEXITCODE`n$out" +} +$env:MESHBAY_CLI_SELFTEST = "1" +try { $out = (& $cli 2>&1) -join "`n" } finally { Remove-Item Env:MESHBAY_CLI_SELFTEST } +if ($LASTEXITCODE -ne 0 -or $out -notmatch "self-test ok") { + throw "frozen meshbay-cli self-test exited $LASTEXITCODE`n$out" +} + if ($createdVenv -and -not $KeepBuildVenv) { Remove-Item -Recurse -Force $BuildVenv -ErrorAction SilentlyContinue } diff --git a/packaging/win/cli-entry.py b/packaging/win/cli-entry.py new file mode 100644 index 0000000..908d4c7 --- /dev/null +++ b/packaging/win/cli-entry.py @@ -0,0 +1,14 @@ +"""PyInstaller entry point for the command-line client, `meshbay-cli.exe`. + +`meshbay_cli.__main__:main` is a module function; PyInstaller freezes a script. +Built beside meshbay-node.exe in the same runtime directory (meshbay-node.spec), +which the Full installer puts on the per-user PATH and the Store package names +in an execution alias. +""" + +import sys + +from meshbay_cli.__main__ import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/packaging/win/firewall.ps1 b/packaging/win/firewall.ps1 index 939a58e..827daf5 100644 --- a/packaging/win/firewall.ps1 +++ b/packaging/win/firewall.ps1 @@ -6,7 +6,9 @@ Two independent things need an inbound allow, and Windows prompts "Allow access" for each the first time it happens if there is no rule: - 1. WebRTC (MeshBay.exe and meshbay-node.exe). It binds an ephemeral UDP + 1. WebRTC (MeshBay.exe, meshbay-node.exe and meshbay-cli.exe, the + command-line client, which carries files over WebRTC as the app does). + It binds an ephemeral UDP port per connection, and the browser always dials the node -- aioice cannot resolve the peer's mDNS `.local` candidate to call back -- so the node (and, during ICE connectivity checks, the client) must accept @@ -23,12 +25,12 @@ are fixed and known. Matches packaging/firewall/*/meshbay-cast.xml. The installer runs this once, elevated, so the user answers one UAC prompt - instead of up to four dialogs spread across first use of chat, downloads + instead of up to five dialogs spread across first use of chat, downloads and casting. Declining the installer's offer is fine -- the dialogs are the fallback, one per program/port combination as each is first used. Shipped as an extraResource at <install>\resources\firewall.ps1, so it - locates the two executables from its own path and takes no arguments + locates the executables from its own path and takes no arguments beyond the action. add/remove run elevated and windowless, leaving a trace at %TEMP%\meshbay-firewall.log; check is a plain read (no admin needed -- Get-NetFirewallRule does not require it, only New/Remove do) @@ -59,12 +61,14 @@ $GROUP = "MeshBay" $client = Join-Path $install "MeshBay.exe" $node = Join-Path $resources "node-runtime\meshbay-node.exe" +$cli = Join-Path $resources "node-runtime\meshbay-cli.exe" # Program-scoped, any port: the ephemeral-UDP-port problem WebRTC always has. # Port-scoped as well: the two fixed, known ports LAN casting actually uses. $rules = @( @{ Name = "MeshBay"; Path = $client; Protocol = "UDP"; LocalPort = "Any" } @{ Name = "MeshBay Node"; Path = $node; Protocol = "UDP"; LocalPort = "Any" } + @{ Name = "MeshBay CLI"; Path = $cli; Protocol = "UDP"; LocalPort = "Any" } @{ Name = "MeshBay Cast"; Path = $client; Protocol = "TCP"; LocalPort = "19550-19553" } @{ Name = "MeshBay Cast Discovery"; Path = $client; Protocol = "UDP"; LocalPort = "5353" } ) diff --git a/packaging/win/meshbay-node.spec b/packaging/win/meshbay-node.spec index a1173a2..8a6e384 100644 --- a/packaging/win/meshbay-node.spec +++ b/packaging/win/meshbay-node.spec @@ -1,6 +1,8 @@ -# PyInstaller spec for the bundled Windows node daemon. +# PyInstaller spec for the bundled Windows node daemon, and the command-line +# client beside it. # -# Output: dist/meshbay-node/meshbay-node.exe (--onedir, relocatable) +# Output: dist/meshbay-node/meshbay-node.exe, meshbay-nodew.exe and +# meshbay-cli.exe (--onedir, relocatable, one _internal\ for the three) # Consumed by: packaging/win/build-node-runtime.ps1, which moves the tree to # packages/meshbay-client/node-runtime/ for electron-builder to carry as an # extraResource. @@ -35,21 +37,27 @@ _vtuple = tuple( int(p) for p in (_ver.split("+")[0].split(".") + ["0", "0", "0", "0"])[:4] ) -_version_info = VSVersionInfo( - ffi=FixedFileInfo(filevers=_vtuple, prodvers=_vtuple), - kids=[ - StringFileInfo([StringTable("040904B0", [ - StringStruct("CompanyName", "MeshBay"), - StringStruct("FileDescription", "MeshBay Node"), - StringStruct("FileVersion", _ver), - StringStruct("InternalName", "meshbay-node"), - StringStruct("OriginalFilename", "meshbay-node.exe"), - StringStruct("ProductName", "MeshBay Node"), - StringStruct("ProductVersion", _ver), - ])]), - VarFileInfo([VarStruct("Translation", [0x0409, 1200])]), - ], -) + + +def _version_resource(name: str, description: str) -> VSVersionInfo: + return VSVersionInfo( + ffi=FixedFileInfo(filevers=_vtuple, prodvers=_vtuple), + kids=[ + StringFileInfo([StringTable("040904B0", [ + StringStruct("CompanyName", "MeshBay"), + StringStruct("FileDescription", description), + StringStruct("FileVersion", _ver), + StringStruct("InternalName", name), + StringStruct("OriginalFilename", f"{name}.exe"), + StringStruct("ProductName", description), + StringStruct("ProductVersion", _ver), + ])]), + VarFileInfo([VarStruct("Translation", [0x0409, 1200])]), + ], + ) + + +_version_info = _version_resource("meshbay-node", "MeshBay Node") # Packages taken in full: data files + shared libs + every submodule. COLLECT_ALL = [ @@ -177,11 +185,96 @@ exe_windowless = EXE( icon="../../packages/meshbay-client/build/icon.ico", version=_version_info, ) + +# --- The command-line client --------------------------------------------------- +# Its own analysis: another entry point, and packages the daemon does not use +# (Textual loads its widgets lazily, Rich its Unicode tables by version, +# Pygments its lexers by name, keyring its backends through entry points — +# hence keyring's metadata). What both import (aiortc, av, cryptography, ...) +# lands once in _internal\: COLLECT keeps one copy of a file the two share. +CLI_COLLECT_ALL = [ + "meshbay_cli", + "meshbay_common", + "aiortc", + "aioice", + "av", + "cryptography", + "pylibsrtp", + "pyee", + "google_crc32c", + "httpx", + "httpcore", + "h11", + "anyio", + "dns", + "argon2", + "_argon2_cffi_bindings", + "keyring", + "win32ctypes", + "textual", + "rich", + "pygments", + "markdown_it", + "mdit_py_plugins", + "linkify_it", + "platformdirs", + "blake3", + "msgpack", + "jwt", + "tzdata", +] +CLI_COPY_META = ["meshbay-cli", "meshbay-common", "keyring", "textual"] + +cli_datas, cli_binaries, cli_hidden = [], [], [] +for pkg in CLI_COLLECT_ALL: + d, b, h = collect_all(pkg) + cli_datas += d + cli_binaries += b + cli_hidden += h +for dist in CLI_COPY_META: + cli_datas += copy_metadata(dist) + +cli_a = Analysis( + ["cli-entry.py"], + pathex=[], + binaries=cli_binaries, + datas=cli_datas, + hiddenimports=cli_hidden, + hookspath=[], + hooksconfig={}, + runtime_hooks=[], + excludes=["tkinter", "pytest", "ruff", "IPython", "meshbay_node", "meshbay_hub"], + noarchive=False, +) +cli_pyz = PYZ(cli_a.pure) +exe_cli = EXE( + cli_pyz, + cli_a.scripts, + [], + exclude_binaries=True, + name="meshbay-cli", + debug=False, + bootloader_ignore_signals=False, + strip=False, + upx=False, + console=True, + disable_windowed_traceback=False, + argv_emulation=False, + target_arch=None, + codesign_identity=None, + entitlements_file=None, + icon="../../packages/meshbay-client/build/icon.ico", + version=_version_resource("meshbay-cli", "MeshBay command-line client"), +) + coll = COLLECT( exe, exe_windowless, + exe_cli, a.binaries, a.datas, + cli_a.binaries, + cli_a.datas, strip=False, upx=False, upx_exclude=[], diff --git a/pyproject.toml b/pyproject.toml index 0c99c1a..a4d7a60 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,6 +3,7 @@ members = [ "packages/meshbay-common", "packages/meshbay-hub", "packages/meshbay-node", + "packages/meshbay-cli", ] [tool.ruff] @@ -13,9 +14,13 @@ target-version = "py312" select = ["E", "F", "I", "UP"] [tool.pytest.ini_options] +# The node's tests `from conftest import ...`, and with no __init__.py every +# tests/conftest.py is the module `conftest`: the last one loaded wins. So the +# node stays last here. testpaths = [ "packages/meshbay-common/tests", "packages/meshbay-hub/tests", + "packages/meshbay-cli/tests", "packages/meshbay-node/tests", ] asyncio_mode = "auto" |