Skip to content

build(kernel): 2-mode kernel build recipe + tagged CI job#395

Draft
mani-mathur-arch wants to merge 3 commits into
mani/sea-kernel-backendfrom
mani/sea-kernel-build-recipe
Draft

build(kernel): 2-mode kernel build recipe + tagged CI job#395
mani-mathur-arch wants to merge 3 commits into
mani/sea-kernel-backendfrom
mani/sea-kernel-build-recipe

Conversation

@mani-mathur-arch

@mani-mathur-arch mani-mathur-arch commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

What

Adds a reproducible build recipe for the opt-in SEA-via-kernel backend and a CI job that exercises it, so the databricks_kernel-tagged path is built and unit-tested in CI instead of being dead code. This is the distribution follow-up that lets #393 leave draft.

  • KERNEL_REV pins the kernel commit to build against (currently PR 163 head in the kernel, which carries the statement canceller the driver links; re-pin to the merge SHA once 163 lands).
  • cgo.go drops the machine-specific link line; per-platform cgo_<os>.go files carry ${SRCDIR}-relative LDFLAGS and force static linking (-l:…a). linux/amd64 is real; darwin/windows are documented stubs pending native runners. A cgo_unsupported.go guard fails at compile time (naming the supported targets) on any other OS/arch, instead of an opaque undefined reference at link.
  • build/kernel-lib.sh builds the kernel .a from source at KERNEL_REV (cargo --release --locked --no-default-features --features tls-rustls), with a rev-stamped cache short-circuit, a KERNEL_LOCAL_A prebuilt-archive override, and a fail-loud guard against source cross-builds. The Rust toolchain is pinned via rust-toolchain.toml (it governs the checkout under build/kernel-src/), so the archive is reproducible under a fixed KERNEL_REV. Makefile adds kernel-lib / build-kernel / test-kernel and a kernel-lib-download stub for the future published-.a path.
  • go.yml gains build-and-test-kernel (CGO + rust + cargo-via-JFrog), with timeout-minutes and a per-ref concurrency group so the heavy source build can't park a protected-runner slot or stack redundant runs; the pure-Go CGO_ENABLED=0 jobs are untouched. Built artifacts are gitignored.

Multi-OS support comes via native per-OS runners (host == target) or staging a prebuilt .a, not cross-compiling from source — matching the settled distribution design.

⚠️ Stacked on #393 (mani/sea-kernel-backend) and draft. Contains a temporary push: CI trigger for mani/sea-kernel-build-recipe in go.yml (clearly commented) so CI runs on this stacked branch — must be stripped before merge.

Known items from review (deferred / pre-merge)

Tracked here rather than as code comments, since these are sequencing/scope notes that would go stale in the source.

Must resolve before merge:

  • Strip the temporary push: trigger. go.yml's branches: [main, mani/sea-kernel-build-recipe] exists only to run CI on this stacked branch (its PR base is the feature branch, not main). It must be removed before merge so main doesn't carry a dead personal-branch trigger that would mint an INTEGRATION_TEST_APP token + clone the private kernel on any recreation of that branch.
  • Re-pin KERNEL_REV to a merged commit. It currently points at 163's PR-head SHA (9b90406), reachable only via refs/pull/163/head — GC-able if that PR force-pushes/deletes, which would break cold-cache CI. Re-pin to the 163 merge SHA once it lands, sequenced with the kernel merge. (Merge order overall: feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393 → main first, then this PR, since these files live in the kernel package that only exists in feat(kernel): SEA-via-kernel backend (opt-in, behind build tag) #393.)

Deferred (out of scope for this PR):

  • No live kernel C ABI round-trip in CI yet. The tagged unit tests prove build + link; the real end-to-end test (kernel_e2e_test.go, SELECT 1 → scan) is env-gated and self-skips without DATABRICKS_HOST / DATABRICKS_HTTP_PATH / DATABRICKS_TOKEN, which this job doesn't set. It passes locally against staging (90 pass / 84 skip). Wiring those three secrets onto the job (un-skipping the e2e test) is deferred to the CI-dispatch / creds follow-up.
  • darwin/windows link contracts are unverified. CI is linux/amd64 only, so cgo_darwin.go / cgo_windows.go LDFLAGS are never compiled. M0 is linux; per-OS runners are the multi-OS follow-up.

Testing

  • make kernel-lib builds the .a (reproducible sha256) from a clean clone, a cache-restored tree, and a leftover-source tree.
  • make test-kernel links the archive with CGO and passes the databricks_kernel-tagged unit tests (ok …/internal/backend/kernel).
  • CGO_ENABLED=0 go build ./... still succeeds — default pure-Go build unaffected.
  • Cache short-circuit skips on a matching rev-stamp and rebuilds on a stale one; cross-build (GOOS ≠ host) fails loud.
  • cgo_unsupported.go guard verified: excluded on linux/amd64 + darwin/arm64 (kernel builds normally), compiled on the unsupported combos (clear compile error).
  • Reviewed with Isaac Review (5 passes, converged to a clean pass) and a follow-up Code Review Squad pass (F1/F2/F6 addressed here; F3/F4/F5/F7 tracked above).

Co-authored-by: Isaac

Add a reproducible build recipe for the opt-in SEA-via-kernel backend and a
CI job that exercises it, so the databricks_kernel-tagged path is built and
unit-tested in CI instead of being dead code.

- KERNEL_REV pins the kernel commit (PR #163 head) to build against.
- cgo.go drops the machine-specific link line; per-platform cgo_<os>.go files
  carry ${SRCDIR}-relative LDFLAGS and force static linking (-l:...a).
- build/kernel-lib.sh builds the kernel .a from source at KERNEL_REV
  (cargo --no-default-features --features tls-rustls), with a KERNEL_LOCAL_A
  override; Makefile adds kernel-lib / build-kernel / test-kernel and a
  kernel-lib-download stub for the future published-.a path.
- go.yml gains build-and-test-kernel (CGO + rust + cargo-via-JFrog); the
  pure-Go CGO_ENABLED=0 jobs are untouched. Built artifacts are gitignored.

Co-authored-by: Isaac
- build-and-test-kernel: drop the "no warehouse creds → live e2e/parity
  self-skip" note (goes stale once creds are added to the job).
- kernel-token step: retirement trigger is the repo going public, not a
  published .a — the source clone needs auth regardless of the download
  path, and the source-build option stays alongside the prebuilt .a.

Co-authored-by: Isaac
@mani-mathur-arch mani-mathur-arch force-pushed the mani/sea-kernel-build-recipe branch from 38c283f to 77e0231 Compare July 13, 2026 05:58
…chain, job limits

Addresses three build-hygiene findings from the PR-395 review:

- F1 platform guard: add cgo_unsupported.go, compiled only on OS/arch combos
  with no LDFLAGS file (e.g. linux/arm64, darwin/amd64). The Makefile host==
  target guard passes there and builds a host .a, but the link then fails with
  an opaque "undefined reference to kernel_*". A build-time constant fails
  earlier, at compile, naming the supported targets.
- F2 reproducible .a: pin rustc via rust-toolchain.toml (governs the kernel
  build under build/kernel-src/; kernel repo pins none), match the CI Rust
  step to it, pass cargo --locked (build against the kernel's Cargo.lock), and
  fold rust-toolchain.toml into both kernel cache keys so a toolchain bump
  busts the stale archive/build-tree caches.
- F6 job limits: add timeout-minutes 30 and a per-ref concurrency group with
  cancel-in-progress to build-and-test-kernel, so a hung cargo can't park a
  protected-runner slot and repeated pushes don't stack heavy builds.

Verified: pure-Go CGO_ENABLED=0 build stays clean; cgo_unsupported.go is
excluded on linux/amd64 + darwin/arm64 and compiled on the unsupported combos;
go.yml is valid YAML; the recipe script passes a shell syntax check.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant