Skip to content

blog: the actors post with production readings, the turn-stop timer defect and the first silicon run - #1603

Merged
gHashTag merged 1 commit into
mainfrom
blog/actors-production-readings
Oct 11, 2026
Merged

gHashTag merged 1 commit into
mainfrom
blog/actors-production-readings

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Updates the published actors post (trinity#1582, first merged in #1583) with what production and silicon showed on 2026-10-10. The changes cover both languages.

In production (new section):

  • trios#1730, #1754 and #1764 are live, with telemetry on since 19:29Z and turn stop on since 20:38Z.
  • The first hour: 14 reviews per hour, all succeeded. Worker busy time is 9.3% at the median and 24.9% at the peak. Mailbox depth stayed at or below 5. There were no crashes, no drops and no storms. One kill came from a model call that ran over 300 s.
  • That answers the post's open question: "23% busy" was a peak window, and "11 an hour" is close to the 14 measured now.

A real defect found by CI: a SIGTERM-deaf process group could survive turn stop, because Bun fires a 3000 ms timer at 2997–2999 ms and nothing asked the card again. It is fixed in trios#1809 and trios#1811. The stopping benchmark's "before" figure is corrected: it was the benchmark's own cleanup.

FPGA: t27b lowers the hardware card bit-exact (t27#8820). The JTAG self-test top is generated from t27 (#8830), and the clocks are declared in t27 (#8857). Place and route ran on xc7a200tfbg484-2 over 5 seeds. The run on die A (t27#8858) read verdict 0xA5A536FD and signature 0x3DED69B1. One die is not citable, and the post says so.

"What this does not show" and the open questions now list only what is still true.

{
  "version": 1,
  "head_sha": "5625496ae04a87567136edbd559f4b4f5ccb52bc",
  "summary": "The actors post updated with what production and silicon showed on 2026-10-10: three batches live with telemetry and turn stop on, the first hour of production readings, a real turn-stop timer defect found by CI and fixed, and the hardware card placed, routed and run on one die.",
  "changes": [
    "apps/website/src/data/blog/bodies/actors-beat-the-loop-on-recovery-not-everywhere.ts: a new section 'In production: the first readings' in English and Russian; the FPGA section gains the t27b lowering, the 5-seed place-and-route and the die A run; 'What this does not show' is rewritten to what is still true.",
    "apps/website/src/data/blog/index.ts: 7 new receipts (trios#1764, #1809, #1811, t27#7851 reading, #8820, #8857, #8858) and the open questions rewritten in both languages."
  ],
  "tests": [
    {
      "command": "bun build --no-bundle on the body and index.ts",
      "status": "passed",
      "result": "the body and index.ts both parse with no syntax error",
      "evidence": "local run on this head, 2026-10-11 UTC"
    },
    {
      "command": "grep for conflict markers in apps/website/src/data/blog",
      "status": "passed",
      "result": "no conflict markers in any blog data file",
      "evidence": "local run on this head"
    },
    {
      "command": "gitleaks protect --staged",
      "status": "passed",
      "result": "no leaks found",
      "evidence": "local run before the commit"
    },
    {
      "command": "npm run build and the website-checks gate",
      "status": "not_run",
      "result": "the sparse clone has no node_modules; the website-checks workflow on this PR runs them",
      "evidence": "this PR's checks"
    }
  ],
  "limitations": [
    "Production readings cover one hour with telemetry on and 50 minutes with turn stop on, at low load; telemetry's CPU cost in production is not measured.",
    "The silicon result is one die; the run record needs three placements before it can be cited, and the post says so.",
    "The concurrency and dispatcher numbers stay simulations; the post labels them."
  ],
  "tags": [
    "Queen",
    "Actors",
    "Measurement",
    "FPGA"
  ],
  "blog": {
    "title": "Actors beat the loop on recovery, not everywhere",
    "summary": "Update: the actor runtime is in production with telemetry and turn stop on, CI found and fixed a timer defect in turn stop, and the hardware card ran on one die and matched the card on 77,794 decisions.",
    "outline": [
      "Three batches reached production; the first hour answers the old reviewer question: 14 reviews an hour, workers busy 9.3% at the median.",
      "CI under load found a real defect, a Bun timer firing 1-3 ms early, and the fix asks the card again until it says kill.",
      "The hardware card was lowered by t27b, placed over 5 seeds and run on die A; one die is not yet a citable run."
    ],
    "cast": "term/t27c-queen-netlink-fence/session.cast",
    "reproduce": [
      "t27c test-report specs/queen/netlink.t27"
    ]
  }
}

🤖 Generated with Claude Code

…ug and the first silicon run (Refs #1582)

The post said nothing ran in production and the FPGA work was synthesis only.
Both changed on 2026-10-10:
- Three batches are in production, with telemetry and turn stop on. The first
  hour gives 14 reviews/h and worker busy 9.3% median, 24.9% peak, which
  answers the old "23% busy or 11 an hour" question.
- CI found a real defect: Bun timers firing 1-3 ms early left a SIGTERM-deaf
  group unkilled. Fixed in trios#1809 and #1811. A benchmark "before" figure
  is corrected.
- actors_hw was lowered by t27b, placed and routed over 5 seeds, and run on
  die A (t27#8858): verdict 0xA5A536FD, signature 0x3DED69B1. One die is not
  yet citable, and the post says so.
"What this does not show" and the open questions now list what is still
true, in both languages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tri-claw

tri-claw Bot commented Oct 11, 2026

Copy link
Copy Markdown

Summary by t27-bees

This PR updates the "Actors beat the loop on recovery, not everywhere" blog post with production telemetry from 2026-10-10, a CI-discovered turn-stop timer defect in Bun, and the first FPGA silicon run results. Both English and Russian versions are updated, and the post metadata adds seven new receipt references.

Worth a look

  • apps/website/src/data/blog/bodies/actors-beat-the-loop-on-recovery-not-everywhere.ts:211–213 — Claims "Under an 8-thread CPU load, 27 of 100 escalations were lost in one run" for the SIGTERM-deaf defect. This is a single CI run; if the load profile or Bun version isn't pinned, the number may not generalize and could overstate the defect's severity.
  • apps/website/src/data/blog/bodies/actors-beat-the-loop-on-recovery-not-everywhere.ts:217–218 — States "Production was exposed only on that path: a model call hears its abort within milliseconds, and with the flag off the same process was abandoned anyway." This assumes no other code paths ignore SIGTERM; if any actor type or library call masks signals, the exposure is wider.
  • apps/website/src/data/blog/bodies/actors-beat-the-loop-on-recovery-not-everywhere.ts:220–221 — Corrects the stopping benchmark's "before" figure from ~9998–9999 ms to ">10000 ms" because the original measurement was benchmark cleanup. The post doesn't explicitly flag this as a correction to prior published data, which could confuse readers who saw the earlier number.
  • apps/website/src/data/blog/bodies/actors-beat-the-loop-on-recovery-not-everywhere.ts:188–197 — FPGA section claims "the hardware card lowers bit-exact to the reference on all 17 of its tests" and gives exact frequency margins. If the 17 tests don't cover all operational modes (e.g., DSP paths, which are excluded), "bit-exact" may be overstated.
  • apps/website/src/data/blog/index.ts:93–99 — Seven new receipts added (trios#1764, #1809, #1811, t27#7851, #8820, #8857, #8858). If any of these PRs/issues are not yet merged or have different titles, the links will 404 or mislead.

Read: 2 files, +92 -19, the whole diff. Head 5625496. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

@github-actions github-actions Bot added the status:in-progress 🔵 Agent working label Oct 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

@github-actions

Copy link
Copy Markdown
Contributor

🧪 Brain Stress Test Results

Tests: 261/261 passed, 0 leaked
Status: 🟢 PASS
Gate: every test in zig build test-brain-stress passes and none leaks

@gHashTag
gHashTag merged commit df620a2 into main Oct 11, 2026
30 of 31 checks passed
@github-actions github-actions Bot added status:completed Done and removed status:in-progress 🔵 Agent working labels Oct 11, 2026
github-actions Bot added a commit that referenced this pull request Oct 11, 2026
blog: the actors post with production readings, the turn-stop timer bug and the first silicon run (Refs #1582) (#1603)

The post said nothing ran in production and the FPGA work was synthesis only.
Both changed on 2026-10-10:
- Three batches are in production, with telemetry and turn stop on. The first
  hour gives 14 reviews/h and worker busy 9.3% median, 24.9% peak, which
  answers the old "23% busy or 11 an hour" question.
- CI found a real defect: Bun timers firing 1-3 ms early left a SIGTERM-deaf
  group unkilled. Fixed in trios#1809 and #1811. A benchmark "before" figure
  is corrected.
- actors_hw was lowered by t27b, placed and routed over 5 seeds, and run on
  die A (t27#8858): verdict 0xA5A536FD, signature 0x3DED69B1. One die is not
  yet citable, and the post says so.
"What this does not show" and the open questions now list what is still
true, in both languages.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant