fix(hub): Search reaches each group with one offer, and a busy hub is not a dead node0.15

The other half of the 4G failure. The page negotiated every group twice: the
sweep opened a connection, read the index and closed it, then the warm-up
opened the same group again. The sweep, the warm-up and the tiles each had a
concurrency ceiling of their own, and together they went past what the hub
admits per account. Whatever the hub refused was then reported as "node
unreachable" and remembered as down, which put that group last next time.

- Every connection goes through ConnectionPool, which holds the page's one
  ceiling (six at once, sized for twenty groups on a phone) and keeps what
  the sweep opened for the tiles. A visit costs one offer per group. A
  refresh costs none for a connection that answers a four-second ping, and
  a connection that died while the phone slept is replaced, not waited on.
- A connection whose index is being read is held against eviction. With
  more groups than the pool keeps, it was otherwise the least recently used
  one.
- Negotiations still under way when the page closes close what they get,
  and a sweep cut short that way remembers nobody as down.
- transport.js sends an offer again on 429, 502 or 503, honouring
  Retry-After, with jittered waits of about twenty seconds at worst. Search
  counts each retry as progress. A 404, 403 or 504 still fails at once, so
  a dead node costs no time.

The fan-out tests assumed a ceiling of three and were re-measured: four dead
groups of twelve now hold nothing back, even on a first visit. The pool and
the retry run as shipped code, lifted as text, against a fake clock. Each
guard was checked by removing it and seeing its test fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

1 files changed, 30 insertions, 29 deletions

diff --git a/packages/meshbay-hub/tests/test_search_fanout.py b/packages/meshbay-hub/tests/test_search_fanout.py
index eda9465..196aa5a 100644
--- a/packages/meshbay-hub/tests/test_search_fanout.py
+++ b/packages/meshbay-hub/tests/test_search_fanout.py
@@ -22,17 +22,21 @@ reader for ten seconds, while two groups of the same batch had answered in two
 hundred milliseconds and nine others had not been dialled at all.
 
 Three things fix it, and each is asserted below: a result is drawn when it
-arrives rather than when its neighbours finish; the ceiling of three is a
+arrives rather than when its neighbours finish; the ceiling is a
 ceiling on *concurrency* rather than a batch, so a dead group holds one place
 instead of stalling a batch and everything queued behind it; and a group that
 did not answer last time is dialled last.
 
 The third is not a refinement of the second, it is what makes the steady state
-correct. A ceiling alone still lets dead groups occupy every place at once —
-with four of twelve down, five live groups appear straight away and the last
-three wait out a deadline. Ordering by what was silent last time puts all eight
-on screen in 600 ms. The cost is that a browser seeing these groups for the
-first time has nothing to order by and pays the first sweep, once.
+correct. A ceiling alone still lets dead groups occupy every place at once — as
+many dead groups as there are places, listed first, and nothing is drawn until
+a deadline expires. Ordering by what was silent last time puts them behind
+every live group. The cost is that a browser seeing these groups for the first
+time has nothing to order by and pays the first sweep, once.
+
+The ceiling is six (search-page.js, `MAX_IN_FLIGHT`). It was three, which was
+also exactly what the hub admitted in flight per account, so the page ran into
+the hub's refusals as soon as anything else connected beside the sweep.
 
 These run the shipped `fetchAllIndexes`, `inFlight`, `lastKnownDown` and
 `rememberDown`, lifted out of search-page.js as text, against a fake clock and a
@@ -93,21 +97,17 @@ def test_several_nodes_down_do_not_delay_the_first_result():
         "dead groups are being waited for one after another")
 
 
-def test_on_a_first_visit_dead_groups_can_still_hold_the_last_few_back():
+def test_on_a_first_visit_four_dead_groups_hold_nothing_back():
     """
-    The limit of a ceiling, stated rather than glossed over.
-
-    Four dead groups and three places: once all three are held by nodes that are
-    not answering, nothing else is dialled until one of them gives up. The first
-    results are immediate and most arrive at once, but the last few do wait — on
-    a browser that has never run this sweep before and so has nothing to order
-    by. The test after this one is the steady state, which is what a reader
-    actually lives in.
+    Four dead groups of twelve on a browser that has never swept before, so
+    nothing orders them. With three places they held every place at once and
+    the last three live groups waited out a deadline; with six, two places stay
+    free and every live group is on screen before any deadline expires.
     """
     out = _sweep(groups=12, dead=[1, 4, 7, 10])
     assert out["firstPaintAt"] == LIVE_MS
     early = [r for r in out["renders"] if r["at"] < DEAD_MS]
-    assert early[-1]["groups"] == 5, (
+    assert early[-1]["groups"] == 8, (
         "the shape of the first visit has changed — re-measure it rather than "
         "adjusting this number")
 
@@ -128,19 +128,20 @@ def test_once_the_silent_groups_are_known_no_live_one_waits_for_them():
 
 def test_a_group_that_did_not_answer_is_dialled_last_next_time():
     """
-    The last bad case, and the reason the browser remembers.
+    The limit of a ceiling, and the reason the browser remembers.
 
-    The three groups the hub lists first are all down, so on a first visit all
-    three places are held at once and there is nothing to draw until one frees
-    up. Told which groups were silent, the sweep puts them behind everything
-    else and the first result arrives on time.
+    The six groups the hub lists first are all down, so on a first visit every
+    place is held at once and there is nothing to draw until one frees up. Told
+    which groups were silent, the sweep puts them behind everything else and
+    the first result arrives on time.
     """
-    first_visit = _sweep(groups=12, dead=[0, 1, 2])
+    down = list(range(6))
+    first_visit = _sweep(groups=12, dead=down)
     assert first_visit["firstPaintAt"] == DEAD_MS + LIVE_MS
-    assert first_visit["remembered"] == ["g-0", "g-1", "g-2"], (
+    assert first_visit["remembered"] == [f"g-{n}" for n in down], (
         "the sweep must record who was silent, or the next visit repeats this")
 
-    again = _sweep(groups=12, dead=[0, 1, 2], knownDown=[0, 1, 2])
+    again = _sweep(groups=12, dead=down, knownDown=down)
     assert again["firstPaintAt"] == LIVE_MS
 
 
@@ -149,7 +150,7 @@ def test_the_order_is_advisory_and_survives_a_browser_that_refuses_storage():
     A private window throws on both halves of `localStorage`. The sweep must
     then behave exactly as a first visit does, not fail.
     """
-    out = _sweep(groups=12, dead=[0, 1, 2], knownDown="refuses")
+    out = _sweep(groups=12, dead=list(range(6)), knownDown="refuses")
     assert out["firstPaintAt"] == DEAD_MS + LIVE_MS
     assert out["remembered"] is None
     assert out["finishedAt"] is not None, "the sweep did not finish"
@@ -166,9 +167,9 @@ def test_a_node_that_comes_back_is_not_punished_for_ever():
 
 def test_no_more_groups_are_dialled_at_once_than_the_ceiling():
     """
-    The ceiling is still a ceiling: twelve groups with three places and every
-    node down cost four deadlines, not one and not twelve.
+    The ceiling is still a ceiling: twelve groups with six places and every
+    node down cost two deadlines, not one and not twelve.
     """
     out = _sweep(groups=12, dead=list(range(12)))
-    assert out["inFlight"] == 3
-    assert out["finishedAt"] == 4 * DEAD_MS
+    assert out["inFlight"] == 6
+    assert out["finishedAt"] == 2 * DEAD_MS