Skip to content

ci: the nightly starts again, the playground's two red samples, the ladder deploy's answer, the rig's reference build, name_lookup's sort - #4382

Merged
borisbat merged 12 commits into
masterfrom
bbatkin/nightly-permissions-rig-fixes
Oct 28, 2026
Merged

borisbat merged 12 commits into
masterfrom
bbatkin/nightly-permissions-rig-fixes

Conversation

@borisbat

@borisbat borisbat commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Why. The nightly workflow has failed at startup every night since Oct 24: its extended_checks caller withholds pull-requests: read, which the called commit_messages job asks for, so GitHub refuses the run before any job starts. The playground nightly, dispatched by hand today, fails two samples on production: one wasm module no longer instantiates (a fd_pread import the runner's shim lacks) and the Exp loop sample's guard sits on its own float32 boundary. Its first run on this PR then showed the next break: ubuntu-latest became Ubuntu 26.04 on Oct 25, and the two linux lanes that name apt.llvm.org's noble repo by hand no longer install llvm-22-dev there, and the vulkan lane's lavapipe from the PPA's preliminary 26.04 build segfaults. Two more master regressions surfaced on the same run: the mingw lane compiles dasGlfw against an unpatched GLFW header because msys2's git applies nothing of the IME patch and exits 0, and the ubsan cell dies in the CUDA oracle's CPU reference casting a NaN to uint. Three smaller breaks ride along: the pages deploy reported a ladder service rejecting stores (HTTP 400) as "service down?" for three weeks, the rig's translation reference build died on a gcc 12 AVX-512 false positive on zen4, and name_lookup.cpp called std::stable_sort, fatal under -Werror with clang-19 on libstdc++ 12.

What changes.

  • nightly.yml's extended_checks caller job grants pull-requests: read.
  • extended_checks.yml's linux lane and nightly_daspkg_index.yml read the runner's Ubuntu codename for the apt.llvm.org repo (noble and resolute both exist there).
  • nightly_vulkan.yml pins its lane to ubuntu-24.04, the image it was green on.
  • dasGlfw's ExternalProject runs git apply with GIT_DIR naming a path that does not exist, so no git discovers this checkout above the extracted source (msys2 git 2.54, Git for Windows 2.55, Debian 2.39 and Apple 2.54 all apply the 16 header hunks that way).
  • The CUDA oracle's math whitelist masks the sign and top exponent bit before its float-to-uint cast: a finite non-negative value, the same on both tiers.
  • test_kernel_coverage.das's nolint:STYLE039 on the Russian prompt literals also carries LINT019: the rule fires only on the darwin flavor, whose Metal half compiles them, and the nightly's whole-folder linux lint (the only lint that walks files a PR did not touch) read the marker as stale.
  • Two authored pages carry das-doc: given markers for names their blocks assume (the lint page's STYLE048 example, the translate tutorial's lines); the nightly's darwin doc-verify is the only lane that compiles those blocks.
  • The playground runner's WASI shim answers fd_pread and fd_pwrite with BADF, like its preopen probes.
  • The Exp loop sample's guard allows 32 instead of 10: the exact sum is 15.47 over 1e6 and the float32 sum lands exactly 10 over.
  • pages.yml posts each record store through one helper that prints the service's whole answer as a workflow warning on any non-200, and a refused connection cannot end the deploy under the step's set -eo pipefail.
  • setup_asr_rig.das seeds the translation reference's cmake warnings list with -Wno-error=maybe-uninitialized on a Linux box.
  • name_lookup.cpp orders its buckets through das_stable_sort.

Observable behavior.

  • nightly.yml: startup_failure on every schedule and PR run -> the lanes run.
  • nightly daspkg_index / index_sweep and extended_checks (linux, all): llvm-22-dev : Depends: libllvm22 (= 1:22.1.8...) but 1:22.1.2-1ubuntu1 is to be installed -> the install resolves.
  • nightly vulkan / integration_linux: test_compute_features.das - crashed (exit status 139) inside libvulkan_lvp.so -> the suite runs on the 24.04 lavapipe.
  • nightly build_windows_mingw: use of undeclared identifier 'glfwSetPreeditCallback' -> compiles against the patched header.
  • nightly ubsan tests cell: simulate_nodes.h:2167: -nan is outside the range of representable values of type 'unsigned int' in test_cuda_oracle.das -> the suite runs clean under UBSan.
  • nightly extended_checks (linux, all): LINT019: stale nolint - STYLE039 suppresses nothing on test_kernel_coverage.das -> clean on linux, and darwin's STYLE039 stays suppressed.
  • nightly extended_checks (darwin15, all) doc-verify: [red] lint.rst, [red] dasLLAMA_17_translate.rst -> both green (checked here with utils/internal/doc-verify).
  • Playground, Linq sample in wasm mode: WebAssembly.instantiate(): Import #22 "wasi_snapshot_preview1" "fd_pread" -> runs.
  • Playground, Exp loop sample, both engines: assert failed at main.das:20 -> prints its timings.
  • pages deploy log: ladder import of m5.json failed (service down?) -> ::warning::ladder import of m5.json FAILED: {"error" : "models[0].runs[2].tune: longer than 1024"} HTTP 400.
  • setup_asr_rig --refs on Debian 12 / Zen 4: marian-decoder - did not materialize -> built.
  • clang-19 + libstdc++ 12 -Werror build of name_lookup.cpp: deprecation error -> compiles.

Where to look. The twelve hunks are independent; the permissions line in nightly.yml is the one that unblocks the cron, and the shim stub is the one a visitor sees.

#nightly

Validation, claims, ledger

Validation

  • The GLFW patch step was replayed on this box from inside the checkout: msys2 git under the old GIT_CEILING_DIRECTORIES form applies 0 of 16 header hunks and exits 0 (-v prints Skipped patch for every file); the GIT_DIR form applies 16 with msys2 git, Git for Windows, Debian git on zen4 and Apple git on the M5.
  • The NaN cast was located in the WSL CI mirror with the interpreter's cast node made to throw on a NaN (das stack: _cuda_common.das:305, the math whitelist's CPU run), and the fixed test runs clean under a CI-flag ubsan build there, with the unfixed file as the control.
  • The nightly permission is settled by this PR's own nightly run (#nightly above): a startup_failure conclusion would mean the fix is wrong.
  • Exp loop sample: opened and run by hand on the live playground with the changed source, in both engines - interpreter printed "exp loop", 0.034475000, 10 and no assert; wasm (queued, built by the production buildd, run) printed "exp loop", 0.005999000, 10 and no assert. The unchanged source asserted in the same session first. Natively the sample runs clean in the interpreter and under JIT; numpy's float32 sequential sum of the same terms is 1,000,010, the value the sample computes.
  • Playwright suite: not run locally with runtime artifacts built from this tip - this box has no wasm toolchain staged. The playground-e2e lane runs the suite on this PR without WASM; the wasm-mode evidence for the shim is the nightly's playground job against production after this lands (the Linq sample's wasm import list exists only in the deployed build), which I will dispatch right after the deploy. The shim is copied verbatim into _site/playground/ by the deploy.
  • name_lookup.cpp: the existing tests-cpp/small/test_name_lookup.cpp case controls the sort (comparator flipped: 1 case, 3499 assertions fail; restored: 8/8 cases pass).
  • The rig flag was proven on zen4 by hand before it went into the script: the reference decoder built with it and setup_asr_rig --check returned 0. The script line itself runs only on a fresh rig.
  • The pages helper runs only on a master deploy. Its three arms (200, 400 with a body, refused connection) were exercised out of tree under set -euo pipefail with a stubbed ssh; a bash probe confirmed the command substitution would otherwise exit the step (codex round finding, fixed).
  • Local preflight: the interpreter and JIT suites passed every assertion; their only reds were two files over the 60 s per-file budget while another build shared the box, re-run alone below their budget. The docs gate was not run: this worktree has no generated stdlib RST and the diff touches no docs.
  • Codex round: one P2 (the helper's command substitution under set -e), fixed; the two playground commits landed after the round and were not re-reviewed (a stub pair and one constant).

Claims - stated, not tested

  • The pages helper's arms have no in-repo test: workflow YAML has no harness. A break shows as a deploy log line that reads "imported" for a 400, or a deploy that stops at the first refused post.
  • The rig flag's Linux-only gate: an MSVC configure would refuse a -Wno-error in its /WX list, and the Mac's clang needs none; neither arm has a test.
  • The shim stubs are never called by a sample: no sample opens a file. A sample that did would read BADF.

Not done

  • Why the Exp loop sum moved by an ulp between the Oct 23 and Oct 28 toolchains is not pinned; the new value is the exact float32 sequential sum, so the sample's guard was the defect.
  • The binding checklists were audited and the diff is compliant; the audits raised 27 wording defects in the checklists themselves (.github/workflows/REVIEW.md, REVIEW.md, modules/REVIEW.md, modules/dasLLAMA/REVIEW.md + REVIEW_UPSTREAM.md, modules/dasLLAMA/performance/REVIEW.md, web/REVIEW.md, web/examples/ui/REVIEW.md, site/REVIEW.md), none caused by this diff; they wait for a documentation batch of their own.
  • The ladder service itself is deployed by hand (utils/internal/dasllama-ladder/README.md); no workflow redeploys it after a schema change.
  • The playground reimport nudge in pages.yml still reads a 4xx as "service down?"; it takes no body and has not misreported, so it stays as is.

borisbat and others added 5 commits October 28, 2026 07:06
The nightly workflow has died at startup every night since the
commit_messages job landed in extended_checks.yml: a called workflow's
job may not ask for a permission its caller withholds, and nightly.yml
granted contents and actions only. Every pull_request run of nightly.yml
died the same way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each record store is posted to the ladder service on dasweb-1, and any
non-200 read as "service down?". A stale service refusing newer stores
with HTTP 400 (a tune stamp past its old 1024-byte cap) kept the live
board three weeks old while the workflow stayed green. One helper now
posts each document and prints the whole answer as a workflow warning.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
std::stable_sort allocates through get_temporary_buffer, deprecated in
libstdc++ 12 and fatal under -Werror with clang-19 - the sites PR 3901
removed from the compiler. The perfect-hash builder's bucket ordering
had brought one back.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
setup_asr_rig's translation reference never materialized on a Debian 12
Zen 4 box: gcc 12 reads its bundled GEMM kernels as maybe-uninitialized
under -march=native, and the build's -Werror rides a cmake warnings
list. Seeding that list from the cache puts the specific -Wno-error
ahead of it on a Linux box; the define would break an MSVC configure,
and the Mac's clang needs no relief.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The runtime's fio binding now references pread and pwrite, so a sample
whose link keeps that object imports wasi fd_pread and fd_pwrite, and
WebAssembly.instantiate refused the module ("function import requires a
callable") - the nightly playground's Linq sample in wasm mode. The
shim answers both with BADF like its preopen probes: no sample opens a
file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Several hunks hinge on behavior that can't be confirmed from the repo alone—the marian flag against an external CMake submodule and the wasm fd_pread/fd_pwrite shim plus deploy helper only verifiable against the live production deploy and nightly.

No open issues 🎉

What changed in this PR

This is a maintenance PR that unbreaks the nightly CI cron and repairs several smaller breaks across the playground, pages deploy, the ASR benchmark rig, and a -Werror build failure. The core unblock is a GitHub Actions permissions fix: the nightly's extended_checks caller job must grant every permission any job in the called reusable workflow declares (pull-requests: read), or GitHub rejects the whole run at startup — validation happens before the job's if gate skips commit_messages.

Changes:

  • CI: grant pull-requests: read on the nightly extended_checks caller; refactor the pages-deploy ladder import into a helper that surfaces the service's full response (e.g. an HTTP 400 schema refusal) as a workflow warning instead of a misleading "service down?", while keeping a refused connection from aborting the step under set -euo pipefail.
  • Build/toolchain: migrate name_lookup.cpp from das::stable_sort to the repo's das_stable_sort (avoids a deprecation-fatal std::stable_sort under clang-19 + libstdc++ 12 -Werror); seed the marian reference build's warnings with -Wno-error=maybe-uninitialized on Linux (gcc 12 AVX-512 false positive on Zen 4).
  • Playground: stub WASI fd_pread/fd_pwrite with BADF so the fio-importing wasm modules instantiate, and loosen the Exp-loop sample's assertion tolerance (eps 10 → 32) to the correct float32-sum headroom.
File Description
.github/​workflows/​nightly.yml Adds pull-requests: read so the called extended_checks workflow passes GitHub's startup permission check.
.github/​workflows/​pages.yml Routes ladder store/annotation imports through one ladder_import helper that prints the full non-200 response as a warning and tolerates refused connections under set -euo pipefail.
src/​simulate/​name_lookup.cpp Switches bucket ordering to das_stable_sort (pointer range) to avoid the deprecated std::stable_sort; adds the das_qsort_r.h include.
modules/​dasLLAMA/​performance/​setup_asr_rig.das Seeds the marian cmake warnings list with -Wno-error=maybe-uninitialized on non-Apple/Windows hosts to survive the gcc false positive.
web/​examples/​ui/​src/​main.js Adds fd_pread/fd_pwrite WASI shim stubs returning BADF, consistent with the existing preopen probes.
web/​examples/​ui/​samples/​examples/​exp.das Raises the Exp-loop verification tolerance from 10 to 32 to match the exact float32 sequential sum.

Note: a private human-deferral signal was recorded because a few hunks (the marian external-CMake flag and the wasm shim/deploy helper) can only be fully validated against the external submodule and the live production deploy, which is outside this repo's reach.

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The WASI shim stubs and ASR-rig flag lack in-repo tests and can only be confirmed through the author's post-deploy runtime checks, leaving verification incomplete.

No open issues 🎉

Review effort: Balanced

borisbat and others added 2 commits October 28, 2026 08:43
The sample sums exp(1/k) over a million terms in float32 and asserted
the sum within 10 of 1e6. The exact sum is 15.47 over, and the float32
sequential sum lands exactly 10 over (numpy agrees), so the assert sat
on its own boundary and failed in both engines of the nightly
playground once the terms changed by an ulp. The guard is against a
broken exp, not rounding: it allows 32.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ubuntu-latest became Ubuntu 26.04 (resolute) on Oct 25, and the two
linux lanes that fetch llvm-22-dev named the noble repo by hand, so
their apt install no longer resolved on the runner - the nightly's
daspkg_index and extended_checks linux jobs. apt.llvm.org carries one
repo per codename (noble and resolute both answer), so the lanes read
the codename from lsb_release.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
borisbat force-pushed the bbatkin/nightly-permissions-rig-fixes branch from ee8506b to 412273e Compare October 28, 2026 15:44
Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change spans multiple arms—deploy helper, compiler rig flags, a wasm shim, and apt codename—that hinge on live production services and external toolchains with no in-repo tests, leaving correctness dependent on the author's manual runtime validation that can't be reproduced here.

No open issues 🎉

Review effort: Balanced

@borisbat

Copy link
Copy Markdown
Collaborator Author

playground / verify_samples is red on this PR by construction: that job verifies the deployed daslang.io, not the branch, and the two samples it fails are the ones this PR fixes. Control: the hand-dispatched run on master before this branch existed, https://github.com/GaijinEntertainment/daScript/actions/runs/40696761204, fails the same three cells (Exp loop in both engines, Linq in wasm) with the same text. The fixed Exp loop source was run by hand on the live playground in both engines and printed its timings without the assert. The job goes green on the first nightly after this lands; I will dispatch it right after the deploy.

@borisbat

Copy link
Copy Markdown
Collaborator Author

build_windows_mingw is red on a master regression this PR does not touch: dasGLFW.func_4.cpp and dasImgui/src/native_input.h call the preedit API (glfwSetPreeditCallback, glfwGetPreeditCursorRectangle, ...) that the GLFW 3.5.1 IME patch from #4368 adds, and on this lane the vendored header at modules/dasGlfw/glfw/Release/include/GLFW/glfw3.h does not declare it, although the ExternalProject's patch step reports nothing and the compile's first -I is that directory. The lane last ran green on the Oct 21 nightly; #4368 merged Oct 25 and the nightly has not started since, so this is its first run on that code. The msvc lanes on this tip are the control: they compile the same files against the same patch. Suspect: msys2's git apply under GIT_CEILING_DIRECTORIES given a Windows-style path, so the step applies nothing and says nothing. Not fixed here; it needs a verbose patch step or an msys2 repro.

@borisbat

Copy link
Copy Markdown
Collaborator Author

build / build (linux, 64, Release, ubsan, tests) is red on a master regression this PR does not touch: tests/cuda/test_cuda_oracle.das crashes under UBSan with simulate_nodes.h:2167:37: runtime error: -nan is outside the range of representable values of type 'unsigned int' in SimNode_Cast<unsigned int, float> - the CPU-side oracle of the convert sweep casts its NaN input (uint_bits_to_float, the test's line 13) to uint, which C leaves undefined. The test landed in 3c92f5f on Oct 23 18:51 UTC, after that day's 02:00 nightly, and the sanitizer cells have not run since. The fix is either the test (no NaN through a uint(float) on the CPU reference) or a defined cast in the interpreter; not decided here.

ubuntu-latest became Ubuntu 26.04 on Oct 25; there the kisak-mesa PPA
serves a lavapipe it marks preliminary, and it segfaults inside
libvulkan_lvp.so on dasVulkan's test_compute_features. The lane was
green on 24.04 with the PPA's noble build, so it stays on that image
until the 26.04 lavapipe settles.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 16:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change bundles several independent CI/infra fixes whose correctness depends on runtime validation that remains unperformed, with multiple arms acknowledged as stated but untested.

No open issues 🎉

Review effort: Balanced

borisbat and others added 2 commits October 28, 2026 10:16
The ExternalProject applied patches/glfw-3.5.1-ime.patch with git apply
under GIT_CEILING_DIRECTORIES, meant to keep git from discovering this
checkout above the extracted source. msys2's git takes no notice of a
Windows-style ceiling, finds the checkout, and in repository mode skips
every file of the patch ("Skipped patch 'include/GLFW/glfw3.h'") while
exiting 0 - so the mingw nightly compiled dasGlfw and dasImgui against
an unpatched header and failed on every preedit symbol. GIT_DIR naming
a path that does not exist turns discovery off everywhere: msys2 git
2.54, Git for Windows 2.55, Debian git 2.39 and Apple git 2.54 all
apply the 16 header hunks that way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The whitelist kernel's uint lane cast int_bits_to_float(ia[i]) to
uint, and ia holds negative ints whose bit patterns are NaNs. CUDA
defines that cast as 0; C leaves it undefined, so the CPU reference run
of the kernel is what UBSan's nightly cell reports (SimNode_Cast<uint,
float>: "-nan is outside the range of representable values"). The lane
keeps its float-to-uint cast but masks the sign and the top exponent
bit first - a finite float below 2.0, the same value on both tiers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 17:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The ten independent CI and deploy fixes depend on live CI and production validation that cannot be verified through static review alone.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 28, 2026 18:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Verification hinges on runtime behavior against external infrastructure that cannot be reproduced here, and the author concedes several arms are stated rather than tested.

No open issues 🎉

Review effort: Balanced

The nightly's whole-folder dasLLAMA lint on linux - the only lint that
walks files a PR did not touch - reports LINT019 on
test_kernel_coverage.das: the nolint:STYLE039 on the Russian prompt
literals suppresses nothing there, while the darwin flavor, whose
Metal half compiles those literals, fires STYLE039 on them. That is the
case the LINT019 message names: a rule that fires only in one flavor's
compile keeps its marker and adds LINT019 to the list, as this file's
other markers do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
borisbat force-pushed the bbatkin/nightly-permissions-rig-fixes branch from 8395ee5 to e64fdde Compare October 28, 2026 18:48
Copilot AI balanced review requested due to automatic review settings October 28, 2026 18:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change bundles 11 independent CI/infra fixes whose correctness hinges on external runtime behavior that can't be reproduced in review and must be confirmed via the nightly run.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 28, 2026 20:14
The nightly's darwin doc-verify (the only lane that compiles the
authored pages' das blocks) reds two pages from today's merges: the
lint page's STYLE048 example reads m, Mode, n and r that no block
declares, and the translate tutorial's loop reads lines that only its
companion program declares. Each block now carries the `given` markers
for what its prose assumes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
borisbat force-pushed the bbatkin/nightly-permissions-rig-fixes branch from 5070190 to 1e37bf4 Compare October 28, 2026 20:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change bundles numerous CI/build-infrastructure fixes whose correctness hinges on external behaviors that can't be verified here, compounded by operationally risky pages-deploy and nightly-permission edits.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 28, 2026 20:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change spans broad production CI/deploy surface with several author-flagged untested arms that only validate at runtime after deploy, exceeding what can be confidently verified in review.

No open issues 🎉

Review effort: Balanced

@borisbat
borisbat merged commit 313fc07 into master Oct 28, 2026
67 of 68 checks passed
@borisbat
borisbat deleted the bbatkin/nightly-permissions-rig-fixes branch October 28, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants