Skip to content

Move language-neutral engines out of go-ruby-* into neutral orgs, shared with julia, R and C #918

Open

Description

@tannevaled

Why

go-embedded-ruby, go-embedded-julia, go-embedded-r and go-embedded-c each need many of the same capabilities: regexes, Unicode normalisation, compression, YAML, time zones, digests, random bit generators, linear algebra, and access to files and the network with a sandbox mode. Today the most complete implementations sit in go-ruby-* orgs, and each one describes itself as "Pure-Go MRI-compatible Ruby X". The others would otherwise rewrite them.

The direction, agreed by the maintainer on 2026-11-10: each language should gain the capabilities that used to be intrinsic to one language.

Rule

  • If an engine's reference behaviour is a C library or a standard that several languages share, the engine lives in a neutral org named for that reference. go-ruby-X keeps only the layer faithful to MRI: API shape, messages, output formatting quirks.
  • What Ruby itself defines stays in go-ruby-*.
  • Extraction happens on need, when a second language needs the engine, not as one sweep.

Moving a module costs:

  • a new module path and a release;
  • updating its importers, including rbgo;
  • a // Deprecated: notice on the old path;
  • carrying the CI it had: fuzz targets, the 6 architectures, the documentation's promises.

First inventory of the 178 go-ruby-* modules rbgo binds

engine (today) shared reference other languages that need it
go-ruby-regexp (Onigmo) Onigmo any Go program (the fleet already recommends it for PCRE/Ruby/JS regexes); a natural neighbour of go-regexp/pcre2
go-ruby-unicode-normalize Unicode UAX #15 julia now (Base's normalize calls utf8proc); go-embedded-julia also has its own tables (internal/unitab)
go-ruby-digest, go-ruby-securerandom the standard algorithms; the OS entropy source julia SHA and Random, R digest; entropy belongs to the shared host/sandbox layer
go-ruby-zlib zlib (byte-identical streams) R (memCompress, gzcon), julia
go-ruby-yaml libyaml (Psych, and R's yaml package) R
go-ruby-tzinfo the IANA tz database julia, R
go-ruby-csv, -json, -toml, -msgpack, -protobuf, -arrow, -parquet the formats all of them, each with its own output layer; Arrow also gives notebook cells one in-memory table format across languages
go-ruby-pg, -mysql, -sqlite3, -redis, -http, -net-*, -resolv, -ldap… the protocols all of them, once each VM has a host mode
go-ruby-format, go-ruby-scanf C's printf/scanf core C, R sprintf, julia Printf, each with its own variants on top

Stays Ruby: erb, haml, slim, erubi, marshal, ostruct, zeitwerk, bundler, rubygems, irb, and the Rails, RSpec and Puppet stacks.

Also shared, outside go-ruby-*

  • Linear algebra: go-ndarray/ndarray/linalg (BLAS, LU, QR, Cholesky, eig, SVD, lstsq) becomes a standalone library that julia's LinearAlgebra and R's solve/qr/chol use.
  • Random: the bit generators in go-ndarray/ndarray/random (MT19937, PCG64, Philox) become a shared library, with Xoshiro256++ and dSFMT added for julia. Distributions stay per language, for bit parity.
  • libm: a bit-exact libm (openlibm for julia, the system libm for R and C).
  • Host/sandbox layer: files, sockets, environment, clock, entropy and processes, designed once on WASI's capability model. go-embedded-julia's first version is in progress (M3: julia's seven stdlibs in the image, and a host mode beside the sandbox go-embedded-julia/julia#32).

First extractions (needed by go-embedded-julia now)

  1. unicode-normalize, reconciled with go-embedded-julia's internal/unitab.
  2. digest, for julia's SHA.
  3. securerandom, into the host/sandbox layer.
  4. Onigmo, moved to go-regexp.

Prior art: priorart "cross-language" at 2026-11-10 17:15 CET reported prior art; checked: priorart 'neutral' and 'cross-language' matched only unrelated VM fixes (#108-#121, #366-#367, #674, #870) and spec-shim issues on the words; no issue covers extracting engines or neutral naming

Activity

  1. tannevaled commented on Nov 10, 2026

    @tannevaled
    ContributorAuthor

    go-ndarray's answer for linalg and random (2026-11-10)

    State: neither is mid-change. The only open PRs are go-ndarray/ndarray#106 (Renovate) and #81 (loong64 float32 GEMM in internal/kernels, blocked on hardware access). There is no reason against the extraction, under four constraints:

    1. GEMM first. linalg imports ndarray/internal/kernels (the GEMM micro-kernels). Either the neutral library carries the GEMM too, or the kernels move down first into go-simd/floats, where the K1 kernels already go (K1 remainder: float32 kernels, Dot1D and the dot inside GEMM, Maximum/Minimum move to go-simd/floats go-ndarray/ndarray#98). There is one implementation, in the lowest shared library.
    2. A LAPACK-style API. Today linalg works on *ndarray.Array. The neutral library takes raw slices with strides and leading dimensions, LAPACK-style, and ndarray keeps a thin wrapper. Its oracle is NumPy 2.5.3 on 1,040 cases. That corpus moves with it, as do the CI jobs: fuzz, govulncheck, cross-arch.
    3. Random stays bit-exact. The bit generators (PCG64, MT19937, Philox, SFC64) match NumPy bit for bit for the same seed, including the order in which the samplers consume the stream. Those golden tests move with the new library. Distributions stay per language, and ndarray's samplers rely on NumPy's exact consumption order.
    4. No break for ndarray users. ndarray/linalg and ndarray/random stay as wrappers, so import paths do not change; below v1 this is a MINOR bump.

    The allocation fixes land once, in the new library:

    Also relevant to julia: go-ndarray/ndarray#76 (Wrap, a zero-copy array with strides over a caller's buffer) was filed for julia's column-major arrays. The LAPACK-style API of point 2 serves julia's Matrix directly, with no copy.

  2. tannevaled commented on Nov 15, 2026

    @tannevaled
    ContributorAuthor

    julia needs the linear-algebra extraction now, and it fixes the API's shape

    go-embedded-julia v0.17.0 runs julia's real LinearAlgebra stdlib, but A * x fails: julia sends every dense product, solve and factorisation through libblastrampoline (LBT). Our interpreter has LBT in Go (interp/lbt.go: lbt_forward, lbt_get_config), and it forwards to nothing. go-notebook ranks this first, because it is the first thing a scientific user hits.

    go-ndarray's answer above, and a reread of a v0.41.2 clone today, show that the fleet already has what is needed, but not in importable form:

    • GEMM. A panel-packed, cache-blocked kernel with a go-asmgen SIMD-FMA micro-kernel on 6 architectures (ndarray/internal/kernels: MatMulStridedP, MatVecStridedP, Dot1DP, and float32). It is row-major and internal.
    • LAPACK. getrf/getrs, potrf, geqrf/orgqr, gebrd/bdsqr/bdsdc, hetrd/steqr/stedc, gehrd/lahqr, plus gemm with alpha/beta on strided views, and trsm, herk, nrm2, rot, axpy and dotc (ndarray/linalg). These are unexported, and checked against NumPy 2.5.3 on 1,040 cases.

    So binding julia means doing the extraction this issue describes. No second GEMM gets written, per go-ndarray's constraint 1.

    Proposal: the neutral library's surface is Fortran BLAS and LAPACK

    This is constraint 2 of go-ndarray's answer, made concrete, because julia pins the shape exactly:

    1. Signatures follow the reference BLAS and LAPACK:

      • column-major, with trans/uplo/diag/side flags, alpha/beta, and leading dimensions lda/ldb/ldc;
      • the four types: s, d, c and z;
      • LAPACK's info result convention.

      julia calls exactly that through LBT, in its ILP64 variant (dgemm_64_…), with 64-bit integers. R's solve, qr and chol call the same names, and C programs can call them unchanged. ndarray's numpy-shaped API becomes a thin wrapper over it, as constraint 4 says.

    2. Column-major needs no new kernel. A column-major C = op(A)·op(B) is the row-major product Cᵀ = op(B)ᵀ·op(A)ᵀ over the same buffers, with the operands swapped and the transpose flags flipped. The existing strided micro-kernel serves both layouts. Wrap: an array over a caller's buffer with given strides (zero-copy, for Julia column-major arrays) go-ndarray/ndarray#76 (Wrap) gives julia's Matrix with no copy.

    3. The oracle is the reference netlib BLAS and LAPACK, plus NumPy's 1,040 cases, with a stated error bound per routine (CLAUDE.md, the bit-parity decision of Accuracy contract: a stated error bound against an extended-precision oracle; bit identity becomes an option go-fft/fft#121). The kernel's FMA and blocking stay. Bit identity with OpenBLAS is not the contract.

    4. What julia needs first:

      • level 1: dot, axpy, scal, nrm2, asum, iamax;
      • level 2: gemv, trsv, ger;
      • level 3: gemm, trsm, syrk/herk;
      • LAPACK: getrf/getrs (for \ and lu), potrf/potrs (cholesky), geqrf/ormqr (qr), syevd/heevd (eigen of a symmetric matrix), gesdd (svd).

      That is close to what ndarray/linalg already holds.

    Placement

    This issue's rule names the org for the shared reference, so something like go-blas, or go-lapack for both. The GEMM micro-kernel goes where go-ndarray's constraint 1 puts it, either in go-simd/floats or in that library. go-ndarray holds the kernels and decides that part; I would rather not guess.

    Who does what

    • go-ndarray, if it agrees: the extraction, meaning the move, the wrappers, the corpus and the CI.
    • go-embedded-julia: the LBT side, which forwards julia's dgemm_64_ and friends to the new library, plus julia's own LinearAlgebra test files as a second oracle.

    The julia side waits for the library's first release.

    The same goes for unicode-normalize (first extraction 1 above): julia's uppercase and normalize call utf8proc, and fail today (go-embedded-julia/julia#65).

  3. tannevaled commented on Nov 15, 2026

    @tannevaled
    ContributorAuthor

    go-ndarray agrees: the Fortran BLAS/LAPACK surface, and where GEMM goes (2026-11-15)

    Prior art: this layer is already designed

    priorart turned up go-ndarray/ndarray#92 (docs/design-linalg.md, merged). Its §9 describes exactly the layer proposed above, as linalg/internal/lapack:

    • column-major ports of Reference-LAPACK v3.12.1, generic over s/d/c/z;
    • refblas.go, level 1–3 in netlib loop order (dot, axpy, nrm2, gemv, ger, trmv, trsv, gemm, trmm, trsm, syrk);
    • getrf/potrf/trtrs, geqp3/geqrf/orgqr, gebal/gehrd/hseqr, the stedc family, gges/tgsen.

    None of it has started: a v0.41.2 tree has no linalg/internal/lapack. So the extraction does not move existing column-major code. It builds that design in the neutral module instead of under linalg/internal, and ndarray consumes it from there.

    One amendment to that design

    go-ndarray/ndarray#92 makes bit identity with netlib the contract: a refgemm without FMA (§7.3), and fused or k-reordered GEMM only as an exception (§7.4). It was written before the 2026-11-11 decision on go-fft/fft#121. Under that decision, the netlib-exact port is a porting step and the oracle, not the default. The library ships:

    • default: the packed, FMA, cache-blocked GEMM, and LAPACK blocked over it, with a stated error bound per routine, asserted against an extended-precision oracle;
    • exact mode: netlib loop order with no fusing, kept as an option and as the equality oracle while porting.

    Placement (constraint 1)

    The whole GEMM moves into the new library: packing, blocking and the go-asmgen micro-kernels of all 6 architectures. It does not go into go-simd/floats.

    Shape of the module

    • One module with packages blas and lapack. Reference-LAPACK ships its reference BLAS in the same repository, julia's LBT needs both, and a single version avoids a lockstep pair.
    • The org name needs a decision. go-lapack exists but is not ours: it was created 2018-08-27, has 0 public repositories, and we have no readable membership. go-blas does not exist. Only the maintainer can create an org; I am asking.
    • Dimensions are Go int. That is already ILP64 on 64-bit GOARCH. Mapping dgemm_64_ and friends stays in julia's LBT layer.
    • Bad arguments are reported, not panicked on. They return LAPACK's info < 0 (the xerbla convention). Slice lengths are checked against lda·(n−1)+m with overflow-safe arithmetic, for 386 and arm.
    • The oracles move with the code:
      • reference netlib, plus NumPy 2.5.3's 1,040 cases;
      • fuzz targets per routine;
      • govulncheck;
      • the 6 architectures in CI.
    • Performance is measured against single-threaded OpenBLAS on one core.

    Order (one PR each, once the org exists)

    1. The repository, with CI copied from ndarray.
    2. GEMM moved in, with the column-major API over the existing strided kernel, plus level-1/2/3 BLAS. ndarray switches to it.
    3. getrf/getrs, potrf/potrs, and geqrf/ormqr/orgqr: the first things julia needs. ndarray's Solve/Inv/Det/Cholesky/QR move onto them, and the 1,040 cases still pass. The allocation fixes of perf: linalg SVD, Pinv, Lstsq, Eigh, QR, Eig allocate 90-276 times per 64x64 call go-ndarray/ndarray#104 land here.
    4. syevd/heevd, then gesdd.
    5. The scipy.linalg side of Bind tzinfo, chronic, money, addressable into rbgo as native modules #92 (§10 tasks 4–17), now in the library.

    go-ndarray does the extraction. go-embedded-julia/julia#67 can forward to the library from step 2 for A * x, and from step 3 for \.

  4. added a commit that references this issue on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions