fix(hub): Search reaches each group with one offer, and a busy hub is not a dead node0.15

The other half of the 4G failure. The page negotiated every group twice: the
sweep opened a connection, read the index and closed it, then the warm-up
opened the same group again. The sweep, the warm-up and the tiles each had a
concurrency ceiling of their own, and together they went past what the hub
admits per account. Whatever the hub refused was then reported as "node
unreachable" and remembered as down, which put that group last next time.

- Every connection goes through ConnectionPool, which holds the page's one
  ceiling (six at once, sized for twenty groups on a phone) and keeps what
  the sweep opened for the tiles. A visit costs one offer per group. A
  refresh costs none for a connection that answers a four-second ping, and
  a connection that died while the phone slept is replaced, not waited on.
- A connection whose index is being read is held against eviction. With
  more groups than the pool keeps, it was otherwise the least recently used
  one.
- Negotiations still under way when the page closes close what they get,
  and a sweep cut short that way remembers nobody as down.
- transport.js sends an offer again on 429, 502 or 503, honouring
  Retry-After, with jittered waits of about twenty seconds at worst. Search
  counts each retry as progress. A 404, 403 or 504 still fails at once, so
  a dead node costs no time.

The fan-out tests assumed a ceiling of three and were re-measured: four dead
groups of twelve now hold nothing back, even on a first visit. The pool and
the retry run as shipped code, lifted as text, against a fake clock. Each
guard was checked by removing it and seeing its test fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

1 files changed, 78 insertions, 0 deletions

diff --git a/packages/meshbay-hub/tests/test_offer_retry.py b/packages/meshbay-hub/tests/test_offer_retry.py
new file mode 100644
index 0000000..3c968db
--- /dev/null
+++ b/packages/meshbay-hub/tests/test_offer_retry.py
@@ -0,0 +1,78 @@
+"""
+An offer the hub refuses for load is sent again; one refused for cause is not.
+
+A phone's Search reported groups as unreachable whose node was answering every
+offer that reached it: the hub had refused those offers with 429, one of its
+per-account ceilings, and the browser took that for the node. A 429 — or a 502
+or 503 while the hub restarts behind its proxy — says the hub is busy, so
+`postOffer` waits (the hub's `Retry-After` when it gives one) and sends the same
+offer again. A 404 for a node that is not connected, or a 504 for one that did
+not answer, fails at once: retrying those would make a dead node cost time.
+
+These run the shipped `postOffer`, lifted out of transport.js as text, against
+a fake hub and a fake clock — see harness/offer_retry_harness.mjs.
+"""
+
+import json
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+STATIC = Path(__file__).resolve().parents[1] / "src" / "meshbay_hub" / "static"
+TRANSPORT = STATIC / "transport.js"
+HARNESS = Path(__file__).parent / "harness" / "offer_retry_harness.mjs"
+
+pytestmark = pytest.mark.skipif(
+    shutil.which("node") is None or not TRANSPORT.exists(),
+    reason="node or the SPA sources are not available")
+
+
+def _post(**cfg) -> dict:
+    proc = subprocess.run(
+        ["node", str(HARNESS), str(TRANSPORT), json.dumps(cfg)],
+        capture_output=True, text=True)
+    assert proc.returncode == 0, proc.stderr
+    return json.loads(proc.stdout)
+
+
+def test_a_busy_hub_is_asked_again_when_it_says():
+    out = _post(answers=[[429, 1], 200])
+    assert out["result"] == "answered"
+    assert out["posts"] == [0, 1000], "Retry-After was not honoured"
+
+
+def test_without_retry_after_the_waits_grow():
+    out = _post(answers=[429, 429, 503, 200])
+    assert out["result"] == "answered"
+    assert out["posts"] == [0, 500, 1500, 3500]
+
+
+def test_a_node_that_is_not_there_fails_at_once():
+    for status in (404, 403, 504):
+        out = _post(answers=[status])
+        assert out["result"] == "failed"
+        assert out["status"] == status
+        assert out["posts"] == [0], f"{status} was retried"
+
+
+def test_a_hub_that_stays_busy_is_given_up_on():
+    out = _post(answers=[429])
+    assert out["result"] == "failed"
+    assert out["status"] == 429
+    assert len(out["posts"]) == 6
+    assert out["at"] <= 20000, "a busy hub cost more than the worst case stated"
+
+
+def test_a_retry_after_that_asks_too_much_is_capped():
+    out = _post(answers=[[429, 3600], 200])
+    assert out["posts"] == [0, 10000]
+
+
+def test_a_caller_that_gave_up_sends_nothing_more():
+    """Search's deadline, or a page that went away, closes the transport while
+    it waits; the offer must not go out on its behalf afterwards."""
+    out = _post(answers=[429], closeAt=100)
+    assert out["result"] == "failed"
+    assert out["posts"] == [0]