| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A root that has spun down, or that lives on a network mount, answers its first
syscall in seconds rather than microseconds. Made from the event loop, that
stalls the whole node: no other group is served, no stream is fed, no chat
message is delivered and the hub socket is not read, for as long as the platter
takes to come back. It was found from the other end — a client's connection
attempt timing out on a node with one member, while the disk woke up — and it is
AV9's lesson with the disk in the place of the mail server.
Every filesystem call on a group's content now goes through `off_disk`, onto a
thread that belongs to that group's root set. The stat goes with the read: a stat
is what *wakes* a sleeping disk, so offloading only the read would move the stall
rather than remove it, and the read would then find the disk already awake.
`_locate` is the one place allowed to call `entry_abs_path`, which is
`Path.resolve()` and therefore syscalls too.
Five handlers touched: a chunk request, an audio transcode, a subtitle request, a
video stream and a delete. The delete became a coroutine, which its one caller
already was.
One worker per root set, not a pool and not one for the node. One worker for the
same reason the indexer's executor has one — two interleaved reads of a spinning
drive seek-thrash rather than go faster — and it keeps two threads from being
inside the same file at once, which is what makes the `seek`/`read` pair safe
without a lock. Per root set, because a node serves several groups and their
roots are not all on the same volume: a single worker would put one group's
sleeping USB drive in front of another's SSD, which is this symptom one level
down. The thread is created on the first read, so a group nobody downloads from
never starts one, and the daemon stops them all on the way out.
The tests measure rather than read: a ticker counts its own wake-ups beside a
request made slow on purpose, and a handler that blocks the loop takes every one
of them with it. Put either call back inline and both tests report zero wake-ups,
which was checked before they were trusted.
QUIC still reads on its loop. Its handler is synchronous by construction, no
client speaks it, and its distance from parity is already recorded in the design
document (§15.3, L3); moving it is part of bringing it to parity, not of this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A 1080p film is decoded by whoever watches it and re-encoded by the node
when no browser can decode the source. Both were on the CPU, and on an
Atom or Celeron mini-PC neither reaches real time — which is what
`transcode_incompatible_video` exists to refuse. This adds the mechanism
that makes refusing it unnecessary.
Node — `hwaccel.py`: VA-API on Linux, Quick Sync or NVENC on Windows,
established by encoding 1080p and reading the file back with ffprobe.
Nothing is accepted that does not produce the exact profile and level
`stream_init` announces, since the client checks that string before it
trusts a byte: an encoder that wrote another level would make the node's
own codec string a lie, and ffmpeg takes `-level 4.1` and `-level 41`
from h264_qsv without saying which it understood. Three modes per
stream — hardware decode and encode, hardware encode alone, libx264 —
demoted per source codec, because a GPU that decodes HEVC may have no
decoder for MPEG-4 Part 2 and only asking it finds out. A mode that
fails is detected on an empty stdout before `stream_init` goes out, so
the viewer sees one working stream and never an error.
Client — Chromium ships VA-API off on Linux. It is enabled where a
render node and a driver are present, then verified through
`navigator.mediaCapabilities`: a no moves to the next GL backend on the
next launch and an exhausted list drops the switches, so a renamed
feature cannot pass for a feature that is on and `--ignore-gpu-blocklist`
cannot survive on a machine it did not help. Feature lists now merge
rather than overwrite — `appendSwitch` replaces the value, and a second
caller would have silently cancelled the mDNS switch aiortc depends on.
Packaging — the drivers are weak dependencies on all four formats, so a
machine without a GPU installs exactly as before. `dpkg -i` and
`rpm -ivh` ignore weak deps; `packaging/README.md` now says so. Windows
needs no driver: the bundled ffmpeg already carries h264_qsv and
h264_nvenc, and a re-pin that dropped them would cost every low-power
Windows node its hardware encoding silently.
AMD on Windows (AMF) and macOS (VideoToolbox) are named gaps, not
oversights: neither could be tried anywhere in this project, and both
re-encode in software as before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Verifying the feature across a library turned up two faults, one of which
broke it outright.
**The extraction budget was a constant and the cost is not.** Extracting
demuxes the whole container, so the file sets the price: measured at 9.8 s per
GB on a library held on an external disk — 36 s for a 3.9 GB title, 71 s for a
7.3 GB one. Against a flat 60 s that worked on most of a library and failed on
the big films, which to whoever is watching one is indistinguishable from a
broken feature. `Avatar.The.Way.of.Water` timed out every time. The budget is
now three times the measured rate per gigabyte, floored at the old 60 s and
capped so no container can pin a transcode slot indefinitely. The client's
own timeout follows it: the node always answers, so that one is a backstop
against a silent peer rather than a deadline for the work.
**Two clicks ran two extractions.** The cache is consulted on the way in, so a
second request arriving while the first was still running missed it and ran
its own — seen in the log as two identical extractions of one 4.3 GB file
overlapping, each holding a transcode slot and reading the file end to end.
Latecomers now wait on the answer the first is producing.
The in-flight entry is registered *before* the first await, not after. The
first version registered it after the probe, two concurrent requests both got
past the check while neither had registered, and the test for it failed —
which is the only reason this note can be written from the right side.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous fix read ffprobe's key frames and took the last one at or before
the request. It was wrong twice, and a viewer felt the difference: subtitles
went from 5 s early to 2–3 s late.
Matroska's Cues index only some keyframes, so an index seek backs off to an
indexed one that the frame list does not single out. And the landing point
moves with **which streams are mapped**, because the container is positioned
where every mapped stream has data — on the reported title, a seek to 4913.7 s
landed at 4909.863 with video alone and at 4907.236 with the second audio
track mapped beside it. The frame scan gave the first number; the stream
delivered the second; the gap was 2.65 s, and the measured audio displacement
in the served stream was 2.65 s.
So the node asks ffmpeg instead: the same seek, the same mapping, one copied
frame under `-copyts`, and the answer read back off the result. 0.06–0.07 s,
cheaper than the scan it replaces. The `-ss` argument stays at the request, so
the bytes served are exactly the ones served before — only the number naming
them changes. The probe runs after the audio track is resolved, because it
cannot be right before that is known.
An answer after the request, or further before it than any real keyframe gap,
is discarded in favour of the old label: a number wrong by seconds beats a
fabricated one.
Found by decoding the served stream and locating its first frame in the
source, which put it at 4907.213 s against an announced 4909.863 s. The test
does the same thing rather than comparing the announced number against a
second reading of the same probe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A cached track replied without leaving a line, so the journal showed nothing
for a viewer who had asked for subtitles and been served instantly. That made
"no request in the log" look like evidence nobody had asked — which it was not
entitled to be, and which cost a wrong conclusion about whether a fix had been
exercised at all.
The request is logged on arrival now, before the cache is consulted, and a
cache hit says so with the size it served. Second time in this feature that a
silent success was read as an absence; the first was an attach-time state that
was correct while the screen stayed empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Copied video can only start on a keyframe, so `-ss t` with `-c:v copy`
delivers the keyframe at or before `t`. The node reported `t` anyway and the
client builds `SourceBuffer.timestampOffset` out of that number, so everything
downstream believed the picture stood a few seconds further along than it did.
That was a wrong label while only the scrubber read it — it is recorded as
such, low priority, in the design document. Subtitles made it a wrong answer:
their cues carry the source's own absolute timestamps, so the mismatch put
every line on screen before it was spoken. Reported from real use on an H264
title, where seeking to 600 s, 2650 s and 5000 s lands on keyframes 0.82 s,
1.56 s and 4.64 s earlier.
The copy path now resolves the request to that keyframe, seeks to it, and
reports it. The bytes delivered do not change — ffmpeg lands on the same frame
either way — only the number that names them. The look-up reads the thirty
seconds before the request and cost 0.12–0.51 s on that title, which is the
price already costed in §15.3 and never paid. Re-encoded video is untouched:
it can begin exactly where it is asked to, and does.
Two details worth their lines. The keyframe is passed back to ffmpeg at six
decimals, because rounding a keyframe's own timestamp down puts it before the
frame it names and selects the previous one — the same fault again, smaller.
And the ffprobe window ends past the request, since an interval whose end is
the request never emits a keyframe sitting exactly on it: the resume position
is the one place a viewer asks for the same instant twice, and it would have
been answered a whole GOP early.
The test decodes the first frame served and matches it against the source
frame at the position announced, rather than comparing `start` to an expected
number — both sides of that comparison would be reading the same ffprobe and
would agree by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reported as "I click a subtitle and nothing appears", on three films. Nothing
was broken. The track selected was the container's forced track, which carries
signage and foreign dialogue only: measured on the film in question, 30 cues
and 77 seconds of text across 2h32 — 0.8% of the running time, against 1559
cues and 41.8% for the full track sitting beside it under the same language
tag. At all three positions tested there was genuinely no cue to show; the
full track would have shown one at two of them.
So the defect is that the menu could not say which was which. The label used
the container's title tag, which said "Forced" on that film and says nothing
at all on most, and no other field was carried. The disposition is the half
that is always there: `probe_video` now reads `forced` and `hearing_impaired`,
`stream_init` carries them, and the label states them in the reader's own
language rather than repeating an English word a muxer happened to type.
The node fixture grows a forced track with no title, because a title would
let the old code pass. The label harness's `t` stub took a parameters object
unconditionally and threw on a key that has none — a fixture narrower than
production, fixed here rather than worked around.
Also removes the activeCues probe that found this. It answered its question:
mode showing, cues 30, active 0, none due at that instant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two faults, both found while a hang could not be diagnosed from the node's
own journal.
`except Exception` does not catch `CancelledError`, which derives from
BaseException. A cancelled extraction therefore sent no reply and logged
nothing at all, leaving the client waiting on something nothing would ever
answer — the one shape that cannot be reported by whoever hits it. It now
answers, logs, and re-raises so the cancellation still propagates.
And the handler logged nothing on any path, so the journal could not even say
whether a request had arrived. It now names the file, the track, the file size
and the free slot count on entry, the moment the transcode slot is taken, and
the duration and byte count on the way out — the size and duration because the
extraction is a whole-file demux and its cost is set by the file, not by the
subtitle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MSE decodes no in-band text track, so a subtitle cannot ride inside the
fragmented MP4 the player is fed. The node extracts one track whole, converts
it to WebVTT and caches it under its own hash; the client pulls that blob
through the ordinary file_req/chunk path and hangs a <track> on the video
element — the same indirection as a TMDB poster or an audio transcode, which
is what makes a film's subtitles extracted once in the life of the file rather
than once per viewing. Whole-file also makes the cues absolute, so a seek and
an audio-language change both leave the track untouched.
**The ordinal counts every subtitle stream, including the ones never listed.**
Only text codecs are offered: a bitmap track (PGS, VOBSUB — about a fifth of a
real library) has no path to WebVTT without OCR, and one extracted anyway
yields a header with no cues, which is a menu entry that shows nothing and
reports no error. Numbering the survivors of that filter would give a
PGS/SRT/SRT file the ordinals 0 and 1 for its text tracks and `-map 0:s:0`
would then extract the PGS — the same trap `AudioTrack.ordinal` exists for,
one level deeper. A fixture whose first subtitle stream cannot be decoded pins
it, and the handler checks membership of the probed list, never a range.
Additive and MINOR: the selector is drawn from `subtitle_tracks` in the node's
own `stream_init` and from no version number, so `subtitle_req` is never sent
to a peer that would not answer it. The floor stays at 3.0.
Also here: a failed extraction never touches playback, a superseded reply
cannot install its blob over a newer choice, and `_languageName` is shared
with the audio labels — lifted by both label harnesses, since a lift that
names one function stops covering the rule the moment logic moves out of it.
Tests: 9 node (tracks told apart by the words in the extracted cues, not by
tags), 10 client. Full suite green: 1545 node/common, 1252 hub.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGY17EPph5LsLzePPXhUVc
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
-ss before -i cannot trim copied video, which must begin on a keyframe, but
accurate_seek did trim the re-encoded audio to the exact request. Every seek
on a copied stream therefore opened with a GOP-wide hole in the audio and ran
a GOP out of sync afterwards — 9.979s on a real film with a 10s keyframe
interval. Accurate seeking is now off wherever video is copied, and stays on
where it is re-encoded, which is the only path that could already begin where
it was asked to.
Every timestamp was correct throughout, which is why nothing caught it; the
tests assert on decoded audio and on frames compared against the source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The streaming path mapped 0:a:0 unconditionally, so a dubbed film played in
whichever language was muxed first and the others were unreachable. The node
now enumerates the tracks in stream_init and honours audio_track in
stream_req; switching is the seek path, since one ffmpeg carries one track.
MNP 3.2, additive: the player draws its selector from the node's own list and
never from a version number, so an older node is never asked for a track it
would ignore. MNP_MIN_SUPPORTED does not move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reported from a phone: signing in with the same account showed no
playlists. syncWith was called from exactly one place in the interface,
so creating a playlist, deleting one, removing a track and saving the
queue all wrote to IndexedDB and stopped there. The store pushes itself
now, coalesced, so a new mutation cannot forget to.
Silence was the real defect. The node audited only successes, so a
refusal left no trace and user_blob_list none at all; the background
push swallowed its reason; the interface said nothing. All three report
now, and "Sync now" says what happened either way.
An unreadable blob on a node was treated as a fetch failure and returned
before the push — permanent, once the node held anything. It is an
absence: the client is the authority, and it gets overwritten.
A sign-in reconciles whatever this browser already holds, a pending push
is flushed when the page goes away, and a push that did not land is
retried once.
no_key is spelled out: a client that signs in with its remembered device
key only ever has a bundle key persisted before the playlist subkey
existed, and an AES handle is non-extractable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
One row per playlist plus a manifest, so starring a track rewrites that
playlist rather than the whole collection. blob_enc is a BLOB, not
base64 TEXT: these run to hundreds of kilobytes.
user_id comes from the session and never from the message; kind is
validated against a pattern; every cap refuses with a stated reason
rather than truncating.
Additive, so MNP_MIN_SUPPORTED does not move — a 3.0 node answers
"unknown message type" and the client writes to the next one it reaches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
| |
AuditStore.cleanup was never called, so audit.db kept every entry.
The daemon now runs it at start and daily.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XuNrwLf5EFWCMHzfoEvnpm
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`search_listed` is a per-group setting on the node, changed by a signed
operator op and carried in the sealed handshake ack. Search reads it after
the handshake and stops there: no index is fetched, cached or merged, in any
of the four views, and the page says how many groups it left out. The switch
is a "Search" section in the group's settings, shown to the operator.
Absent means listed, at every layer: roster default, ack default, and the
client only drops a group on an explicit `false` — so an upgrade or an older
node removes nothing from anyone's Search.
It is a listing preference and protects nothing: the node serves the same
index to Search and to the group page and cannot tell them apart, every
member lists the group by opening it, and a client that ignores the flag
lists it in Search too. Design §9.11 says so, so it is never described as
private. The cost is one handshake per unlisted group, because only the node
knows the setting.
Search also took `nodes[0]` twice — for the index and for the pooled
connection — the defect 4cce50f fixed on the group page only. One
`connectToGroup` now walks the list the same way: a refusal about this
browser stops, `not_hosted` or a failed connection moves on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XuNrwLf5EFWCMHzfoEvnpm
|
| |
|
|
|
| |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`IndexProgress` said "scanning, this many bytes of that many" and nothing
more. A group's roots are walked one after another, so a second directory
added during a large scan showed as the bar jumping back to 0 %. It now also
carries the root being walked and its position in the roots table, the kind
of walk (scan, rescan, reconcile, watch), file counts, and the roots
waiting for the scan lock in order: queued by the initial scan, by a
retarget, and by a plug; dropped when a root is removed.
`GET /api/index-status` answers for every group at once, including a group
still in its initial scan, so a client can show indexing on any page. It
names roots: loopback only, like `current_dir`.
`index_progress` and the handshake ack gain the same counters, still naming
nothing (decision D3): the root is a position in the roots table the member
already opened from the sealed index, and the queue is a count. The pusher
keeps speaking while a root only waits for the lock.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A watchdog burst wrote the indexer's single `progress` directly. A file
dropped into a folder during a large scan added its size to the scan's
total, then cleared `scanning` when its own hash finished, so the progress
went blank with hours of hashing left. Bursts now keep their own counters,
shown only while no whole-root walk runs.
A plug rescan took no scan lock and walked its root beside an added root's
scan, both resetting the same counters and reading the drive in turn. It
now waits for the lock, and skips the rescan if the root was ejected or
removed while it waited, since the rescan drops the entries before it walks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A root added from the client arrives over MNP, and _retarget_indexer
started the daemon's reload with the session's own _spawn. When that
session closed - a client reconnecting 47 s into the scan of a 900 GB
root - shutdown_tasks() cancelled the reload mid-scan, and the reload
queued behind it, without a line in the log. The new root was in
node.toml and in the indexer's set but never in the group's context; the
lock was free and nothing retried, so the node served the old roots table
for hours while reconcile hashed the whole drive as missed events. One
loopback reload fixed the live node in 9 ms.
_reload_config now runs the work in a node-owned task and awaits it
through asyncio.shield, so a caller that goes away only stops waiting; a
cancelled reload is logged. plug_root does the same for its rescan, which
drops the root's entries before walking the disk and so left the root
empty when its admin op's session closed.
The existing MNP test replaced _spawn with a list and could not cancel
anything; the new tests close the session for real and fail without this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adding a large directory to a running group made the reload await the
scan of the new root before putting the new RootSet in the group's
context, holding _reload_lock the whole time. For the hours a large drive
takes to hash, the node served the old set:
- a file request under the new root got None from entry_abs_path and the
handler died on None.exists() without replying;
- a writable/removable toggle answered with the live table, still the
old one, so the directory vanished from the operator's settings;
- reconcile saw every file the scan had not reached as a missed event and
hashed it again on the same executor, rewriting progress under the scan.
retarget now applies the set, the roots table and the watcher first, and
with wait=False scans the added roots in the background; the daemon swaps
ctx["roots"] before calling it. A scan lock shared by the initial scan,
added-root scans and reconcile makes the reconcile loop sit out a running
scan without backing off. Every transport site that resolves an entry
answers ROOT_NOT_SERVED instead of crashing, and a delete keeps the entry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A chat message is the plainest member-supplied write there is: the node stores
it in `chat.db`, where nothing expires it — retention is a manual command
(§6.6) — relays it to every other connected member, and has the hub write a
notification for every member of the group. Nothing bounded any of it. The only
ceiling was the DataChannel frame, 64 MB once the handshake is done, so one
member in a loop filled the operator's disk and saturated everyone else's
connection, and the node's answer to each message was `ack`.
Uploads, the other member-supplied write, have carried a filename allowlist,
strict chunk ordering, a no-overwrite rule and a 4 GB cap since C5a — because
somebody asked what one member costs the others on that path. Nobody had asked
it on this one.
Two bounds, for the two halves of the question: **64 KB of ciphertext** for what
one message may cost, and **60 a minute per account per group** for how often
one member may impose it. Both are checked before anything is stored or relayed;
a refusal names itself and is audited, so "why is my disk full" has an answer.
The rate is keyed by account, not by connection: a second tab does not make
anyone type faster, and keying on the session would hand a script one budget per
socket it opens.
No node-wide ceiling beside it, deliberately. The link-preview limiter has one
because a preview spends the *node's* egress and its third-party quota, which is
one shared thing; a chat message spends the sender's own group, and a node-wide
ceiling would let a busy group silence a quiet one — this same defect one level
up. The last test in the new file is that property: a member at their limit has
not spent anybody else's.
Two things stay open on purpose and are named rather than quietly done:
retention still keeps everything, because a default that deletes people's
history is not a review's call; and the composer still offers to send an
oversized message, so this is §6.4's pattern with only the node half built.
§6.6 gains the rule, §13.5b the label — AV20, with AV21–AV23 registering the
three fixes this week that closed the same kind of gap elsewhere.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`_is_node_admin()` is `self._user_id == node_user_id`, and `_user_id` is the
`sub` of a JWT the hub issued. Six node-wide controls were gated on that alone:
`node_status` — which lists every group on the machine with each root's
**absolute path** — plus `node_settings_set`, `roster_read`, `denylist_read`,
`denylist_clear` and `node_reload`.
So the answer to "are you the operator of this node" was "the hub says so",
which NS4 and M3 rule out in as many words: operator authority comes from the
node's roster and from nowhere else, and asking the hub is how the hub installs
itself as node administrator. The reach is bounded — a completed handshake also
needs the group key — but an active hub obtains one legitimately in an
open-join group, which §3.5 concedes, and from there it could read the
operator's directory layout or clear the denylist, which is the persisted
revocation H4 exists to keep.
`_operator_device()` requires both halves now: the account is the one the node
belongs to, *and* the device on this connection has proved a key the roster
holds as an operator. `device_hello` is signed over a transcript naming the
node, the group and this connection's nonce, and `operator_pks()` is rebuilt
from the roster on each call, so an unpinned browser and a revoked one are both
refused at once. The hub holds no user keys and cannot countersign a device.
Keeping the account check as well is deliberate: dropping it would widen these
node-wide controls to any paired operator of any group on the machine, which is
a separate decision. `_is_node_admin()` stays as what it is in the handshake
ack — a hint telling a client whether to offer the Node page — and says so.
Nothing changes for a paired operator: `device_hello` runs unconditionally
after the ack, and anyone using the Node page's controls is already paired,
since `root_add` and every other signed op has always verified against
`operator_pks()`. A browser that never paired now reads nothing there, which is
the state in which it could already write nothing.
test_node_status.py's fixture set the account and not the device, which is how
it went on passing; it now wires the device the way `device_hello` leaves it.
The adversary itself is in test_security_regressions.py — a token naming the
owner's account with no proved device, which the previous source answered with
`node_status_ack`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`_do_file_request` read `tr` as a boolean. Present meant "this is a leased
transfer, skip the leaseless ceiling", and nothing asked whether this node had
ever granted such a lease — `slots.touch(tr)` was called beside it and its
answer, `False` if it is not granted, was discarded. So any non-empty string
bought the whole library with no ceiling of any kind: not the per-member cap,
not the node-wide one, not the leaseless bound that exists to bound a client
claiming to be browsing. The queue held only the clients that chose to wait.
`_lease_of` decides it now, and the three answers differ on purpose:
- **granted**, and of *this* session — served, and touched so the sweeper
does not reclaim a transfer that is plainly moving. The session is checked
as well as the id, because touching another connection's lease refreshed
its idle timer.
- **queued** — refused with `lease_not_granted`, on the upload path too,
before anything reaches the operator's disk. A member reading while queued
is the cap not applying.
- **unknown** — bounded by the leaseless ceiling rather than refused. That is
also what a reconnect looks like from here, where the session's leases died
with the old connection and the client is re-opening them, and it leaves
the residual §5.5 already states: a client that lies gets that bound's
worth of files at a time, not the group. Noted once per connection so the
residual is visible rather than merely documented.
Nothing changes for the shipped client: the transfer store awaits
`lease.acquire()` before it reads a byte, so the refused case is one it never
enters. §5.5 gains a paragraph saying the node decides which of the two a
request is — the document described the accounting without ever saying it was
enforced, which is how it came not to be.
`test_lease_enforcement.py` drives the real handlers over a real index; six of
its nine cases fail against the previous source, each on the property.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
asyncio keeps only a weak reference to a task, so a coroutine started with
`asyncio.ensure_future(...)` whose result is discarded can be collected while
it is still running: the loop logs "Task was destroyed but it is pending!" and
the work simply does not happen. No error reaches the caller, and what is lost
is whatever that coroutine was in the middle of.
The node already had a guard for this, written after an abandoned stream task
lost a transcode slot for good — and it read one file, `webrtc_server.py`,
because that is where the defect was found. Outside that file there were
nineteen sites: the hub's `chat_notify` (a notification for every member of a
group), the indexer's debounce (every real-time index update), eleven in
`daemon.py` including the SIGHUP reload and each enrichment pass, two in
`ops.py`, and five in the loopback API.
`meshbay_common.background.spawn()` is the one door. It holds the task, drops
it when it finishes, and logs what it raised under the coroutine's own name —
an exception in a task nobody awaits was otherwise reported by asyncio at
collection time, out of context or not at all. A peer session's `_spawn` stays
as it is: that one can also *cancel* what it holds, which a module-level holder
cannot, because a session ends and a process does not.
`test_background_tasks.py` walks every package's source and refuses a discarded
handle. It parses rather than greps, so an assignment, a comprehension or an
await is not mistaken for one, and it was checked against a deliberate
reintroduction. A guard that stops at the edge of the file where the bug was
found is a guard against that bug, not against its class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`_register_uploader` walked the index for the entry it had just written, at a
moment when no such entry can exist: the file was a `.part` until the rename on
the line above, which is not indexable, and the watchdog that will index it
debounces for two seconds and then hashes. The walk matched nothing, silently,
so every uploaded file in every group was owned by nobody — and `file_delete`
refuses a caller with no admin authority when the entry records no uploader, so
a member could not delete what they had just sent. MESHBAY_DESIGN.md §5.4
grants that to any non-revoked device of the uploading account.
The record is now written when the last chunk lands (`indexer.record_upload`)
and the entry is stamped from it in `_hash_or_cached`, the one funnel every
entry passes through — initial scan, watchdog, reconcile and replug alike. It
lives in the index cache rather than on the entry alone, because the index is
rebuilt from disk at every start and an owner the node forgets on restart is a
right quietly taken away. It is validated against a live `stat()`, so whatever
later occupies that path inherits nothing; and `_rescan_root`'s carry-over no
longer copies over it, or memory would beat the durable record.
§5.4 also claimed ownership was *provable* — a transcript the uploader signs,
stored with the entry. No such signature has ever existed; `meshbay:upload:v1`
in the code is the groupbox purpose that seals the envelope. The section now
states what the code does, and the transcript is an open item in §15.3.
`test_upload_attribution.py` drives the real handler and a real indexer across
that seam. Against the previous source its two positive cases fail on the
property, not on a missing method — an upload, then a rebuild from disk, then
a different file at the same path inheriting nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An availability review, prompted by the group claim above: a participant
supplies input — who else bears the cost? Six answers where the cost fell on
someone other than the sender, and none of them needs an attacker.
AV3 `chat_notify` carried a `group_id` the hub believed, so any connected
node could write a notification to every member of any group on the
hub, carrying a display string of its choosing, with its account
having no relation to that group. This is the group claim again, two
hundred lines further down the same socket. Gated on what the node is
registered for, and metered: the fan-out is one write per member. The
budget expires by time rather than on disconnect, or reconnecting
would refill it and a node token is good for an hour.
AV4 A swarm source named its own `endpoint` as free text documented as
"ip:port", so an account could publish a third party's address — H6's
`peer_ip` defect, never applied here. Nothing dials a swarm source
today, which is the only reason it was not already a reflection
primitive. It is a transport and a port now, never a host, and the
number of hashes one account may claim is bounded: rows were keyed
(hash, account) with no cap at all.
AV5 `handle_webrtc_answer` resolved any pending `peer_id` from any node's
socket. The answer is the SDP a browser then connects to. That this
had not happened rested on a uuid4 being unguessable.
AV6 `relay_register` had no authentication of any kind: it compared
`pk_relay` against the approved value, which is a *public* key, so
anyone who could read it could rewrite where the hub tells nodes to
send relayed traffic. The module docstring promised signed JWTs and
`jwt` was imported and never used.
AV7 The node held unlimited peer connections and kept one that never
completed a handshake for the life of the daemon. H6 bounded what one
unauthenticated peer costs; the hub's cap is three offers in flight
per *account*, a limit on each caller and not on the machine, so an
operator's exposure grew with the size of their groups.
AV8 `invite-notify` put a request-supplied `group_name` into the subject
of an email the hub sends under its own domain, to any account, with
no rate limit. The name comes from the group row now.
The tests are two accounts each, in one file that says why: a one-member test
proves a one-member property, and every finding here needed a second person
to exist at all. Each was checked against the unfixed code. Two did not
survive that check and were rewritten — one re-enacted the disconnect path
instead of running it (hence `forget_node`), the other called the reaper
itself and would have passed with the call removed from `handle_offer`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4YmK41VsEURWFdop4EEeT
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A node that hosts no groups sends no `group_ids` on its hub socket, and the
hub resolved the claim with `set(claimed_groups or authorized)` — so "I host
nothing" arrived as "I host every group this account belongs to", other
members' included. Such a node can serve none of them: it holds no GEK, and
its own handshake refuses them with "Group not hosted on this node".
`/v1/groups/{id}/nodes` answers in registration order and `_node_groups` is
in-memory, so which node a client was sent to depended on who reconnected
first after a hub restart. GroupPage took `nodes[0]` with no fallback. On
2026-09-11 a hub deploy at 20:14 reshuffled the registry, a second member's
unconfigured node won the race, and a group stopped opening for everyone in
it with its only real host online throughout. Any member could take one of
their groups down, by accident, by leaving an empty node running.
Four changes, because no one of them is sufficient:
- the hub never widens an absent claim, and `update_groups` goes through
the same ceiling as registration — it assigned its list verbatim, so the
bound that makes C2 hold at authentication was one message wide
- the node states the empty set rather than omitting the field
- the refusal carries `not_hosted`, so a client can tell "try the next
node" from "you, here, must do something first"
- GroupPage walks the list instead of indexing into it
The three lines involved date from 13, 20 and 23 August and each is
defensible alone. The defect is in the seam, which is where the last two
also were: a falsy empty collection must never mean "unspecified".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4YmK41VsEURWFdop4EEeT
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The ack was assembled from its own tuple of application names, a copy of the
daemon's `APP_DIR_KEYS`, and the two had drifted: the copy was missing
`helloworld`. So the reference application — the one that exists to prove a new
application needs no special-casing — was the single application whose
configured folders never reached a client, which made the plugin claim false
exactly where it is demonstrated.
Fixed by removing the copy rather than syncing it. The ack now emits whatever
`<app>_directories` the group context carries, and `_app_directories_ctx` is the
only thing that puts one there, so the two cannot disagree again. The transport
names an application in one place, `ALLOWED_APPS`, which is enforcement rather
than a directory list.
The client had the same fault one layer up: `group-page.js` read three names by
hand from the ack while the live-update path beside it was already generic. It
derives the map from the ack's own keys now, so the fix reaches the settings
pane instead of stopping at the wire.
A first attempt moved the list to `roster.py`, where directory *storage* lives,
and `test_helloworld_proves_the_plugin_claim.py` refused it: the roster, the ops,
the config and the root set must name no application at all. That test is the
architecture's own guard and it was right — the list belongs on the daemon, which
is what wires a group's context, and everything downstream is derived from it.
Two new tests, both verified to fail against the previous shape: the ack carries
an application the node names nowhere else, and the ack keeps no list of its own.
`test_the_lists_are_read_under_one_name_each` now asserts the shell names no
application rather than that it names exactly three.
Two stale comments went with it — the ack's, which described scalars removed in
07ff8b4, and the client's, which said those scalars still rode the wire for
MNP 1.0 peers that can no longer connect.
Full suite: 2258 passed, 4 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YVoHVCcfBqud6ZjG4db3y7
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The handshake ack's app-directories entry still said "the same three answers in
one shape" and "the scalars above are derived from these and kept for MNP 1.0
clients". Neither is true since the per-app ops were folded into one: there are
no scalars above, and a 1.0 client cannot reach this code at all — the floor
moved to 3.0 with the lease flag day.
A comment that contradicts the code beside it is worse than no comment, because
one of them is wrong and the reader cannot tell which. This is the same fault
the leaseless-bound comment had, one commit earlier.
What it says instead is what is actually load-bearing: `<app>_directories` is
the only form on the wire, and `chat_directory` below is safe as a second name
for one of them because `_app_directories_ctx` derives it on every build rather
than storing it alongside — which is precisely what the removed scalars did not
do.
daemon.py had the same stale reference three lines from the code that produces
these, pointing at `video_root` for the shape a per-group signed setting takes.
Comments only; no behaviour change. Node suite green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YVoHVCcfBqud6ZjG4db3y7
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`video_root`, `audio_root` and `photo_roots` are gone — the messages, the
signed operations, the handlers, the `ops` wrappers, the three scalars on the
handshake ack, and the client's handlers for their acks. `app_directories`
does the same thing for every application, keyed by the app's own registry
name, and it is what the SPA has been sending.
The three were the same instruction three times, differing only in the key they
wrote and whether they carried a string or a list. That shape is what made
adding an application mean adding a message type, an op, a handler and a widget;
it also meant three validation paths, and the older ones validated nothing —
a typo was stored and then quietly matched no entry, an app showing an empty tab
with no way to tell "misconfigured" from "no files yet".
**What stays, and why.** `Roster.LEGACY_DIR_KEYS` still reads `video_root` and
friends out of `group_settings`: that is a key on an operator's disk, not on the
wire, and a node upgraded into this must find its own configuration. The Search
page still reads its own older cache keys, for the same reason — the cache
outlives a deploy. `CTX_ALIASES` keeps only `chat`, which is the one app whose
second name something still reads.
The two per-app policy test files go with the messages. What only they held —
the real challenge/response path from message to database, which no other test
exercises — is retargeted at `app_directories` in
`test_app_directories_signed.py`, and the handler's own refusals (unknown app,
malformed `directories`, nobody to authorize it) join `test_app_directories.py`.
Node and common suites 1368 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Seven comments pointed at sections of `~/next/improve-downloads.md`, which is
not in the tree and not anywhere a reader of this repository can follow. Each
now states the thing it was citing: why a paused transfer holds nothing, why the
lease is taken after the save target and not before, why a chunk request marks
a lease alive, where the leaseless bound's number comes from.
The leaseless comment also said "two files at a time" three paragraphs under
`MAX_LEASELESS_IN_FLIGHT = 12`, left behind when the bound was raised. A comment
that contradicts the constant beside it is worse than no comment: one of them is
wrong and the reader cannot tell which.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`device_add_request` and `device_hello` were dispatched behind
`and self._nonce_node`, which reads as "pre-proof, once the challenge has gone
out" — and is not what happens: both branches sit after the
`self._user_id is None` guard, so the nonce is always set by the time either is
reached, and a peer that has not finished its handshake gets "Handshake
required" instead.
The guard is removed rather than the branches moved. Filing a device is not
something a peer needs *in order to* prove possession of the group key, which is
the only reason anything is served pre-proof: the request is countersigned later
by a device already pinned, so requiring the caller to finish its own handshake
first costs nothing and keeps the pre-proof surface at three messages.
A test drives all six device messages through the real dispatcher on an
unauthenticated session, because this is a property of the order of its branches
and of nothing else.
Node suite 1216 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`senderkeys.py` and its 13 tests implemented Signal-style sender keys, and
production has never called them: chat is a key per group, per epoch, per
device, derived by name. The reasoning that ruled the ratchet out stays where it
belongs — in `chatbox.py`, at the top of the module that replaced it — because
the argument is the useful part, and it now stands on its own instead of
pointing at a file to compare against.
Kept code that nothing calls is worse than absent code: it reads as an
alternative somebody may reach for, and it has to be maintained past every
refactor to stay compiling, which is maintenance spent on a decision already
made.
The three comments naming `GroupSenderKeyStore` are rewritten to say the thing
they were illustrating.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The QUIC handler stored `payload` as it arrived and broadcast it: no envelope,
no signature check, no `device_hello` to check one against. A message reaching
a group's archive that way is a plaintext row in an encrypted history, and it
would be indistinguishable from one somebody actually wrote.
Removed rather than gated. The transport implements neither the per-device
sealing nor the device identification the WebRTC path requires, so refusing
here would mean maintaining a second, weaker set of rules for a transport with
no client; an unimplemented type is logged and dropped, like every other message
this transport does not have.
The comment on the peer registry loses its chat fan-out aside for the same
reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The group-wide `member_upload` switch is gone: the message, the signed
operation, the field on the handshake ack, the `upload` alias on every root in
the index payload, and the client's fallback path to it.
Whether a member may write has been a property of each root for a while, and
that is the model that survives: a single flag over the group cannot express
"this library is published read-only and that folder is a drop box", which is
the ordinary arrangement. What was left of the switch was a handler that logged
a deprecation and acted on nothing, and a client that read `ack.member_upload`
whenever the roots carried no `writable` — a second source for one question,
with whichever the code consulted first deciding it.
`roots.describe()` drops `upload` for the same reason: it was `writable` under
an older name, and two names for one boolean is one too many.
The paperclip now says "nowhere to write" rather than picking a root, in a group
that has none writable. That is the honest answer; the fallback picked whatever
came first and failed at send time.
Node suite 1215 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`transfer_state` read `slots.per_member` — the node-wide default — while
`_has_room` decides with `member_cap()`, which prefers the group's own signed
limit, and the handshake ack announces that same `member_cap()`. Three readings
of one number, and one of them was the odd one out.
In a group where the operator signed a higher limit, every lease update told the
client "cap: 2" while the node would grant five: the transfers widget draws
`used >= cap` as saturated, so a member with two transfers running saw the rest
of their slots disappear. Lowered the other way it is worse in the other
direction — the interface offers slots the node will queue.
Nothing was ever granted or refused wrongly; the enforcement was right on both
paths. It is the number beside it that contradicted them.
Two tests, one override above the default and one below, because a bug that
reads the node-wide value passes the first whenever the default happens to be
the larger number.
Node suite 1215 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Streaming an Xvid/MP3 .avi answered "Unsupported video codec" — a refusal, on
a file ffmpeg re-encodes at about six times playback speed on the machine that
reported it. Nothing about the source was wrong. The node simply never reached
its own re-encode path.
`probe_video` maps a source codec to an MSE codec string and knows four:
h264, hevc, vp9, av1. Everything else returns None, because there is no
MediaSource decoder in any mainstream browser to give a string to — MPEG-4
Part 2 (Xvid, DivX), MPEG-2, VC-1, WMV, Theora. `_stream_video_inner` read
that None as a verdict on the file and refused, while the re-encode sitting
twenty lines below it was gated on `raw_video_codec in
BROWSER_INCOMPATIBLE_VIDEO_CODECS` — a set containing "hevc" and nothing else.
So the whole ffmpeg fallback existed, worked, and was unreachable for every
codec that most needed it.
The setting that governs the fallback has documented the intended behaviour
since it was introduced: draft-v6 §2.11 says `transcode_incompatible_video`
covers "HEVC *and other browser-incompatible video codecs*". Only HEVC was
ever wired up.
Two questions were being answered by one value, and they are separated now.
"Is there a video stream at all" is the only thing this path genuinely cannot
serve, and the only refusal left. "Can it be copied" needs both an MSE string
to put in `stream_init` and a codec browsers decode; a source failing either
is re-encoded.
The operator's opt-out keeps meaning what it says, and it no longer means the
same thing for every source, because it cannot: HEVC has a codec string, so
`transcode_incompatible_video = false` falls back to a copy and the viewer's
own decoder decides (unchanged). MPEG-4 Part 2 has none, so there is nothing
to fall back to — a `stream_init` with no codec string is one the client
refuses before the first byte — and the stream is refused naming the setting.
"Unsupported video codec" is what sent this report to the file, and the file
was fine.
Verified against the reported file end to end: ffprobe reports mpeg4/mp3
720x404, the decision comes out `can_copy=False`, and the pipeline's exact
argv produces H264 High level 4.1 plus stereo AAC-LC — matching the
`avc1.640029,mp4a.40.2` that `stream_init` advertises and that the client puts
through MediaSource.isTypeSupported byte for byte.
test_stream_hevc_transcode.py becomes test_stream_video_transcode.py: it was
always about the policy rather than about one codec, and it now carries both
halves of it, with a synthetic Xvid/MP3 .avi built the same way as the HEVC
clip. Its module-level skip on libx265 went with it — an ffmpeg without x265
still encodes MPEG-4 Part 2, so that marker was skipping the reported defect
entirely on any box without it; it now gates the HEVC cases alone. Three
cases added, checked against the unfixed source. Hub and node suites 2269
passed, 4 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GXmScYB1uR29YCt74si9J
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reported the day MNP 3.0 shipped: playing a track answered "Too many files open
at once without a transfer. Download this one instead of previewing it."
§3.4.1's bound of two was reasoned about *viewers* — a photo viewer shows one
photo, a preview modal one document, and the second is for prefetching the next.
It forgot the music player, which warms a read-ahead window: `prefetchDepth()`
returns 5 on Wi-Fi and 3 otherwise, so playing an album has six files in flight
and the fourth was refused. Browsing a group is never subject to a transfer slot
— that is a stated requirement, not a tuning parameter — and a constant nobody
had checked against the client broke it.
Twelve now: six for the music read-ahead at its widest, two for a photo viewer
and its own prefetch in the same session, the rest as headroom. Generosity is
cheap here and refusal is not — this is a fairness control among cooperating
clients, not a security boundary, so a client that lies gets twelve files at a
time instead of its member cap, bounded and audited, while refusing a legitimate
read breaks the requirement outright.
And the number is now derived rather than chosen: a test reads `prefetchDepth()`
out of the shipped player and fails if the node's bound no longer covers it, so
widening the client's read-ahead breaks the build instead of reaching a person.
Checked by widening it: "the music player reads 21 files ahead and the node
admits only 12".
Three cases that hard-coded "two then refuse" now set their own limit — they are
about the mechanism, and the shipped number moves with the client.
Node suite 1210 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stage 4 of ~/next/improve-downloads.md, the flag day. Leases become compulsory
and a 2.x peer is refused at the handshake.
**The bound on leaseless reads (§3.4.1) did not exist, and it is what makes the
rest mean anything.** Browsing a group is never subject to a transfer slot —
that is an operator decision and a requirement: a member must be able to browse
a group at capacity exactly as they browse an idle one. But "not leased" cannot
mean "unbounded", or a client that simply omits `tr` transfers outside every cap
and the caps are decoration. A session may now read two distinct files at once
without a lease: one because a viewer looks at one file, two so that prefetching
the next photo stays possible. A count of files and not a byte budget, because a
RAW photo is 60-80 MB and is browsing while a 40 MB archive is a download, and
no size threshold separates them. Thumbnails, posters and cover art never reach
this check at all — they resolve out of the node's own cache.
It is a fairness control among cooperating clients, in the company of
`max_concurrent_streams`, and is not a defence against a member determined to
saturate a node's disk. That member is a member, and the answer to them is
`member revoke`.
**MNP_VERSION and MNP_MIN_SUPPORTED both move to 3.0**, on both sides. The
messages are additive; the requirement is not. An opt-in switch would leave a
leaseless branch reachable on every node, which is finding C6's lesson — a
transport that accepted a bare JWT — one feature later.
**The desktop client now checks before it connects.** The SPA is served by the
hub and picks up a new client on reload; the application ships its own
interface, so an un-updated one would sign in, list groups, and fail every
connection with `version_too_old` — a refusal in a protocol vocabulary with
nothing anyone can act on. It asks `/v1/hub/version` for `client.minimum` and
says so plainly instead. An unreachable hub is deliberately *not* "too old": a
captive portal or a closed laptop must not make starting the application
impossible.
**Every package is aligned on 0.13.0.** `meshbay-client/package.json` had
drifted to 1.0.0 while the Python packages were on 0.12.0 — invisible until
something compared those numbers, and then load-bearing: an installed client
announcing 1.0.0 sorts above a 0.13.0 minimum and walks through the gate meant
to stop it. That is stated in the code rather than left to be rediscovered; it
is acceptable exactly once, because the operator is updating every client, node
and hub by hand for this flag day. A new test fails if two packages ever
disagree again, and another fails if the hub would refuse the client the tree
builds.
Node suite 1209 passed, hub suite 861 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stage 8 of ~/next/improve-downloads.md, second half, plus the gap it exposed in
stage 7.
**Asking where to resume.** The node identifies an upload by (member,
directory, filename), so a client resuming one has to name the file — and
`transfer_open`, the obvious place to ask, travels in clear. Naming it there
would undo exactly what sealing this path bought in MNP 2.0: before it, the same
file was ciphertext leaving a node and plaintext arriving at one. So the
question is asked inside the seal that already exists, as an ordinary
`file_upload` with no bytes and `chunk_index: -1`. The node writes nothing,
creates no state, reserves no name, and answers with `resume_from` in the sealed
ack. A node that predates it refuses the index, which the client reads as "start
from the beginning" — the behaviour it had anyway — and the wait is bounded so
one that answers neither does not strand an upload.
The probe is answered after every check the write path makes, so it cannot ask
questions about a directory the caller may not write to, and it answers only
about the member who asks: otherwise one member could measure another's
progress on a file they never sent, and worse, resume it.
**Pausing an upload.** Reported: no pause button on an upload, even in the
desktop app. Stage 7 built pause around the download path — a target declares
whether it can be stopped — and an upload has no local target to ask. It was
also refused by design, since a transfer handed a lease it cannot re-create must
not be offered a button that would drop its slot for good. Uploads now ask for
their slot rather than being handed one, and say they are pausable outright: a
File is seekable and the node keeps the position. Resuming re-probes rather than
trusting the client's own memory, so it works across a reconnect too.
**And the slot they hold.** `_do_file_upload` never called `slots.touch(tr)`.
Chunks are not gated by the lease, so the file arrived — but the node reclaimed
a grant nobody appeared to be using after thirty seconds, twice, then abandoned
it, and the widget follows the lease. Measured from the journal: a 3.5 GB upload
read "waiting, 0 ahead" for a minute and a half while it was transferring. The
download twin of this was fixed on 2026-09-08; the same omission was still here,
invisible until uploads took a real lease.
`test_the_upload_itself_is_sealed` now checks every message `uploadFile` sends
rather than the first. Adding the probe put a second one in front of the one it
was written for, and it would have kept passing while guarding nothing.
Node suite 1202 passed, hub suite 850 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stage 8 of ~/next/improve-downloads.md, first half. Two defects that are the
same defect seen from two sides.
An upload's progress lived on the session, keyed by `rel_dir/filename`. A
dropped connection threw it away and the client's next chunk was refused with
`not_started`: an upload interrupted at 99% could only be started again from
zero, on a link flaky enough to have interrupted it once. It now lives in the
group context, keyed by member as well -- a shared directory means two people
can be sending IMG_1234.jpg at the same moment and neither may inherit, or
overwrite the position of, the other's.
What the lost state left behind was a `.part` nothing would ever finish, delete
or look at again. It is not an index entry, so it is invisible to every member
and to the operator's own file list: one abandoned film is a gigabyte of their
disk, kept for ever. That leak predates this branch.
A `.part` is deleted only when **both** hold: no upload is writing it, and
nothing has been written to it for 24 hours. Waiting costs disk; being wrong
costs somebody their upload, and is not reversible -- so a read-only root is
never walked (it cannot have received an upload), an unavailable one is never
walked (an unmounted drive reporting "nothing found" is how a careless janitor
deletes a library), and a file whose mtime is in the future is left alone (a
clock that went backwards is not evidence). The reaper matches whole paths and
the state records the path it is writing, rather than both sides rebuilding one
from a root name -- two implementations of one rule whose failure mode is
deleting a live upload.
The rules are in `uploads.py`, pure logic with no asyncio and no transport, the
same shape as `transfers.py` and for the same reason.
23 cases, four of them checked against the unfixed source. Node suite 1195
passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"The node saw no transfer" was concluded twice from a journal that could not
have shown one: the open was logged at DEBUG, and the daemon runs at INFO. Two
diagnoses were built on that non-observation, and both were wrong.
Turning the root logger up to DEBUG is not the answer either — aiortc logs every
SCTP chunk, which on a 2 GB download is both unreadable and slow. One line per
transfer is not a volume problem, and it is the line that answers "did the
client ever ask for a slot, and what was it told".
Node suite 1172 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Raising the per-member cap from 2 to 4 left both waiting transfers on
"waiting". The pool granted them correctly and nobody told the peers:
`ops.set_transfer_limits` computed `granted` and never sent a `transfer_state`,
where the node-wide path (`WebRTCTransport.set_capacity`) does.
That is the first row of §5.2 of ~/next/improve-downloads.md — "node granted a
slot, the push was lost" — reached by writing the decision and forgetting the
send. It is the same omission as the missing `touch()` call one layer up, on the
same day: a mechanism that is right everywhere except at the seam where it has
to reach somebody.
The client recovered after its 60-second watchdog re-asked, which is why this
looked like a slow queue rather than a lost message.
`transfer_probe.py --operator` covers it now, separately from the node-wide
hot-swap it already covered — different door, different code path, and only one
of them was tested. Verified failing with the push removed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reported as "the slots seem hardcoded to 2": `meshbay-node transfers set 8 8`
and still only two downloads at a time. Not hardcoded — that is the *per-member*
cap, which is a group's setting and is checked before the node's, so raising the
machine's total cannot move it. But the diagnosis was right in the way that
matters: nothing could change it.
`OP_TRANSFER_LIMITS` shipped with exactly one front door, the signed MNP
handler, and nothing anywhere opened it — no client call, no CLI verb, no
loopback route. So the cap sat at its default of 2 for ever, which from outside
is indistinguishable from a constant. CLAUDE.md states the rule this missed:
operator operations are one implementation with several front doors.
- `PUT /api/groups/{id}/transfer-limits`, calling the same
`ops.set_transfer_limits` the signed handler calls;
- `meshbay-node transfers per-member <downloads> <uploads> [--group X]`;
- `transfers show` now separates the node-wide pools from the per-group
per-member caps, and marks each `[set]` or `[default]`. It printed "2 per
member" with no indication of where the 2 came from, which is half of why
this looked like a constant.
Zero is refused here as everywhere else: it is not "unlimited", and a member who
may not transfer at all is a member the operator revokes.
Verified on a live node: the cap changes, survives a daemon restart, and
`transfer_probe.py --want 6` measures 4 granted against a cap of 4 where it
measured 2 before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two defects in the lease machinery, both found in the node's own log, neither
reachable from any test on either side.
**`touch()` was never called.** The node ignored `tr` on `file_req` entirely, so
`used` stayed False for every download ever made and the sweeper revoked each
grant thirty seconds in — while the file was transferring at 20 MB/s. The pool
was correct and the handlers were correct; the call between them was missing,
which is why neither side's tests could see it.
**The requeue was a permanent cycle.** A revoked grant went back in the queue,
was granted again a millisecond later because there was room, and was revoked
again thirty seconds on. The node logged the same two reclaims every thirty
seconds for as long as it ran — minutes after the transfers involved had
finished. Three chances now, then the lease is closed and the peer told.
`test_the_counter_never_drifts` could not have caught it: nothing drifted, the
same lease simply never left.
A lease that starts being used forgets its earlier misses: a client that took
two grants to get going is slow, not abandoned.
Also `transfers show` reported the module defaults rather than the operator's
values until something had transferred, so `transfers set 2 2` answered
"applied now" and the next line said 0/8 — indistinguishable, from outside,
from the hot-swap that did nothing for months. The test asserted the defaults
and so agreed with the bug; found by typing the command.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Step 3 of ~/next/improve-downloads.md. Step 2 built the pools with constants;
this gives them to the operator, in the two scopes they belong to.
**The pools are the machine's.** `[node] max_concurrent_downloads` and
`max_concurrent_uploads`, default 8, on the §2.11 pattern: node.toml for a
fresh install, a roster.db override for immediate effect, editable from the
Node page and from `meshbay-node transfers show|set`, applied live through the
one `set_capacity` step 1 fixed.
**The per-member cap is a group's.** How many transfers one member may run at
once here — on the node like every other group setting (not the hub, which
would have authority over someone else's disk; not node.toml, which is
hand-written and needs a restart), changed by a signed operator instruction
(`OP_TRANSFER_LIMITS`, subject "d=2,u=2" so what is signed names the outcome),
broadcast to the group, and read live by the pools.
That was the one thing step 2's shape could not express: `per_member` was a
single node-wide number. `group_limits` and `member_cap(kind, member)` make it
a lookup — the group's own value if it has one, the node's default otherwise —
and it is deliberately the only dimension that is not node-wide.
Three refusals, each with a test:
- **absent means the default (2), never "unlimited".** A group that predates
the setting coming back unlimited would leave the node-wide pool as the only
control, which is the situation slots exist to end;
- **zero is not "unlimited"**, and is not "this member may not transfer"
either: the floor is one everywhere, and the CLI says to revoke the member
instead;
- **an unreadable row reads as unset**, not as zero — the same discipline the
sealed messages follow, where a payload that does not open must never become
a default state on its own.
`handshake_ack` carries this member's own caps for this group, so the interface
can say "2 of your 2 slots are busy" instead of drawing a bare spinner. Absent
reads as "no limit known" and the hint is not drawn — never as "unlimited",
which would have the interface contradicting the node.
1164 node, 793 hub, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Step 2 of ~/next/improve-downloads.md. A download is invisible to the node: it
is a series of independent `file_req` messages, with nothing saying one started
or ended, so there is nothing to count and nothing to cap. The lease is that
missing object.
`meshbay_node/transfers.py` holds the decisions and has no asyncio and no
transport in it, on purpose. The failure modes this has to survive — a slot the
node never gets back, a client waiting on a grant the node has forgotten — are
races through a DataChannel and unprovable there; here the clock is a parameter
and every method returns what changed, so the caller does the I/O and the tests
drive the worst case directly.
What it decides:
- two pools, downloads and uploads, separate from the stream pool: different
resources with different costs, and merging them makes both caps meaningless;
- per-member cap checked *before* the node-wide one, so a member at their own
limit queues behind their own transfers rather than holding a slot a second
member has none of. Per account across their devices, or the cap becomes a
function of how many tabs somebody opens;
- a queue that skips a member at their cap instead of waiting for them —
granting strictly in arrival order lets one member's limit stall everyone;
- `tr` drawn by the client and idempotent, which is what makes a reconnect safe;
- bounded per member, because unbounded queues are how a node runs out of
memory politely.
Every way a slot comes back, with the session teardown as the one that matters
(a closed tab, a quit browser and a dead network all arrive at
`shutdown_tasks`, and none of them needs a timer): explicit close, session
gone, a grant nobody took up in 30 s passed to the next in line, and a granted
transfer silent for 120 s reclaimed with its peer told, so a widget can offer a
resume rather than sit on a lie.
`GET /api/transfers` is the operator's window: when somebody reports a transfer
stuck at waiting, it is the only thing that says whether the node ever had them
in a queue — a log cannot, when the symptom is that nothing is happening. It
carries no filename and no path, which a test pins, because this is exactly
where one would be tempting.
Three things found while writing it, two of them mine:
- the randomised property test rejected `in_use <= cap` at once, and it was
right to: lowering a cap never interrupts a running transfer, so the count
legitimately sits above the new value. The invariant is that a *new* grant
never happens past the cap;
- the sweeper was started with `self._spawn`, which ties a task to one
session's set. It died with whichever peer opened the first transfer, and
every other peer's abandoned lease then stopped being reclaimed — a node that
fills up over days with nothing in the log. It belongs to the node now, with
its strong reference on the transport context;
- the pools are node-wide while `_peer_registry` is per group (finding H1), so
a slot freed in one group can grant one in another and the peer to notify is
not in the notifier's registry. Silently wrong in the first version.
Nothing enforces a lease yet: `file_req` is untouched, no client asks, and the
node grants everything. That is step 4's flag day, and this lands alone.
1148 node, 793 hub, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`ops.set_node_settings` hot-swapped the stream pool by assigning
`webrtc._stream_sem`. That attribute has never existed on WebRTCTransport — the
pool is `ctx["_transcode_sem"]` — so `hasattr(webrtc, '_stream_sem')` was always
False and the branch never ran. The setting was accepted, written to roster.db
and node.toml, and applied only on the next restart, which is exactly what
draft-v6 §2.11 says it does not need. An operator lowering the cap on a
struggling machine, or raising it after "Server busy", saw nothing happen and
had no way to find out why.
`WebRTCTransport.set_capacity()` is the one implementation, on the object that
owns the state, so the download and upload caps the transfer-slots plan adds
next do not each grow their own copy of the mistake.
Resizing has semantics worth stating: the new cap governs new streams and never
interrupts one that is running, because a slot is held for the length of a film
and lowering a number must not take somebody's film away. The replacement pool
is built with the permits that remain (`new - in_flight`, floored at zero) — a
full set would briefly allow more concurrent viewers than either the old cap or
the new one.
That needs a count of slots in use, so `_stream_video` now maintains one instead
of the code reading the semaphore's private `_value`: a number this code keeps
itself survives the semaphore object being replaced underneath it, and the same
counter makes the "N of M in use" log lines mean something.
test_stream_capacity.py drives the real transport and the real `_stream_video`;
`test_ops_calls_the_real_mechanism` fails if the dead attribute comes back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`thumbs` holds every generated thumbnail, every TMDB poster and backdrop, every
Cover Art Archive image and every cached audio transcode. Rows were removed only
when their source file left every group's index (`prune_file`), so a library
that merely changes over years grew this database with nothing to bound it.
Nothing in it is precious — every row is keyed off a value the node can
re-derive — which is what makes eviction the right answer rather than a bigger
disk.
512 MB, evicted on write (a cache only grows when written to; a timer is one
more thing to own and get wrong). `used_at` is marked on every read, including
the lookup by synthetic id that `_fetch_and_cache_poster` makes on every visit
to a poster grid — without that, the images shown most often would be the
coldest rows in the table. A single blob larger than the cap does not empty the
table for nothing.
The migration is the part that touches deployed nodes. `CREATE TABLE IF NOT
EXISTS` adds missing tables and never missing columns, so `used_at` would have
reached a fresh test database and never a real one. `_migrate()` does the
ALTER TABLE and seeds existing rows with "now" rather than 0 — otherwise the
first write after an upgrade evicts the whole cache, a correct-but-hostile
reading of "least recently used" for rows whose age nothing recorded.
The index on that column lives in `_migrate()`, not in `_SCHEMA`: run from the
schema script it executes before the ALTER on an existing database and fails,
which would have been every deployed node refusing to open its cache on the
first start after upgrading. Found by the migration test.
Verified against a real node's database, rebuilt into its pre-migration shape:
rows preserved, column present, seeded, index created, reopening harmless.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST
|
| |
|
|
|
| |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8EDjk6pkYZrCbo63m2x87
|