Skip to content

Latest commit

 

History

History
811 lines (728 loc) · 48.2 KB

plan-rbgo.md

File metadata and controls

811 lines (728 loc) · 48.2 KB

Implementation plan — Ruby in pure Go (go-embedded-ruby, CLI rbgo)

Goal: an implementation of Ruby written in pure Go (no cgo) that produces a single static binary embedding compiled code while preserving Ruby's full dynamism (dynamic dispatch, monkey-patching, eval, runtime require, metaprogramming).

This document is the reference roadmap. It is amended at the end of every phase (exit criteria met, decisions settled, risks updated). A standalone critique of the original draft is folded into the relevant sections and summarised in §15 Risk register and §16 Decision journal.

1. Vision & constraints

  • No cgo. Everything is pure Go → trivial cross-compilation (GOOS/GOARCH), static binary by default.
  • Bytecode VM (mruby model), not transpilation to Go. The front-end (lexer + parser + compiler) is embedded in the binary so dynamic eval/require work.
  • Dynamism = a live object model. Dispatch goes through mutable method tables (our objc_msgSend).
  • Reuse the GC. Ruby objects are Go heap objects; Go's GC collects them. We do not write a GC.
  • Build-time selection. Embed only the subset of the stdlib that is needed (require-graph scan + config + build tags), not the whole distribution.

Target & scope

  • Reference semantics: Ruby 4.0 (pattern matching, endless methods, it, numbered params, beginless/endless ranges, safe navigation).
  • In scope: the language core, embedded pure-Ruby stdlib, extension leaves reimplemented in Go.
  • Out of scope (assumed): third-party C extensions (gems with .so), 100% MRI compatibility. We target a well-defined subset that grows (mruby philosophy), validated by conformance tests.

2. Overall architecture

source.rb
   │  (lexer: stateful)
   ▼ tokens
   │  (parser: recursive descent + Pratt, locals table)
   ▼ AST
   │  (compiler: local resolution, catch tables)
   ▼ bytecode (ISeq)
   ▼
┌──────────────────────────────────────────────┐
│ VM (interpretation loop)                      │
│  • object model (RClass/RModule/RObject)      │
│  • mutable method tables → dispatch           │
│  • frames, env (closures), catch tables       │
│  • Fiber ↔ goroutine                          │
│  • Go core + embedded pure-Ruby stdlib        │
└──────────────────────────────────────────────┘

Two execution modes:

  • rbgo run app.rb — compile in memory and interpret (development).
  • rbgo build app.rb -o app — produce the single static binary (see §12).

3. Repository layout

ruby/
  cmd/rbgo/            CLI: run | build | compile | repl
  internal/
    token/             token kinds (rich: tUMINUS vs tMINUS, tLABEL, …)
    lexer/             stateful lexer: lexState, spaceSeen, heredoc queue, literal stack
    ast/               AST nodes
    parser/            recursive descent + Pratt; scope stack (locals)
    compiler/          AST → ISeq; local resolution; catch tables
    bytecode/          ISA, ISeq, (de)serialization
    vm/                interpretation loop, frames, exceptions, super
    object/            object model: Value, RClass, RModule, RObject, Method
      core/            Go core types: integer, float, string, array, hash, symbol, proc, range
    builtin/           Kernel/Object/Module/BasicObject + metaprogramming floor
    fiber/             Fiber ↔ goroutine bridge
    encoding/          String encodings (UTF-8, ASCII-8BIT, transcoding)
    regexp/            adapter onto the standalone go-ruby-regexp/regexp module
    loader/            require resolution over embedded FS + bytecode cache
    stdlib/
      ruby/            upstream *.rb embedded (comparable.rb, enumerable.rb, set.rb, …) via //go:embed
      native/          Go leaves: json, digest, zlib, securerandom, base64, …
  oracle/              test harness: fixtures generated from MRI/Prism (dev only)
  testdata/
  spec/                imported subset of ruby/spec

4. Value representation

Starting decision: Value is an interface with concrete types. Go's GC manages everything.

type Value interface {
    RClass(*VM) *RClass   // dynamic class → used by dispatch
}

Critique — this is the project's central tension, not a Phase 8 footnote. An Integer behind an interface is a heap allocation plus an interface method call and a map lookup on every dispatch. The two ways MRI avoids this — tagged / NaN-boxed immediates (fixnums, flonums, nil/true/false) — fight Go's GC, which must statically know which words are pointers. So you largely cannot do MRI-style immediates in safe Go. Consequences to accept up front:

  • Position the project as deployment/embedding convenience, not raw speed (numeric loops will be several× MRI).
  • Even so, split Fixnum and Bignum so the common integer does not carry a nil *big.Int word, and add a small-int cache (−128..255, like CRuby) early.

Principle: immutable → value types; mutable → pointers. object_id is stable via a weak-keyed table (pointer → id); equal? is pointer identity for reference types. Note ObjectSpace._id2ref / each_object are effectively incompatible with reusing Go's GC (they need a registry that pins everything) — stub them, do not promise them.

✅ EMPIRICALLY CONFIRMED (2026-07-03) — the interface Value stays. After landing the interpreter allocation-floor levers (lazy returnTarget, hashKey no-rebox + symbol intern, int-boxing funnel through IntValue, proc arg elision), we prototyped the full tagged-struct alternative (struct{tag uint8; i int64; obj RubyObj}, immediates inline, tagNil==0) as a complete, correct flip: ~4,700 sites converted by four type-aware go/types rewriters, compiles + -race-green + 100% coverage on all 6 arches incl s390x-BE (see PR #121, parked as a documented spike, not merged). It did eliminate immediate boxing — LoopSum (s+=i ×300k) went 598,931 → 0 allocs/op. But it is a net wall-clock regression on copy-heavy paths: Fib +36%, Wordcount +68% (72→121 ms — would undo the whole allocation-floor arc), HashStringChurn +62%, because a 24–32-byte struct Value costs ~2× the 16-byte interface to copy through args, slices, and Hash entries. A tight alloc-bound int-loop microbench projected ~3.95× — misleading, because it did not model the VM's copy-heavy paths. Conclusion: in single-threaded safe Go, the interface Value's smaller copy footprint beats tagged-struct alloc-elimination; the struct only wins under GC pressure / concurrency. The right levers for the interpreter floor are the ones already merged (targeted alloc elimination on the interface representation) + AOT for compute-bound code (AOT-L3 already crushes int loops via native int64 locals). Revisit tagged Value only if a concurrent/GC-bound production workload justifies it.

Phase 0 ships a minimal Value (Integer int64, Float, String, Bool, Nil, Main) without RClass; the object model lands in Phase 1.

5. Object model (the heart of dynamism)

type RClass struct {
    name      string
    super     *RClass
    methods   map[Symbol]*Method   // MUTABLE → monkey-patching, define_method
    consts    map[Symbol]Value
    ancestors []*RClass            // cached linearization (include/prepend)
    isSingleton bool
}
type Method struct {
    name    Symbol
    iseq    *ISeq      // Ruby method…
    native  NativeFn   // …or implemented in Go
    owner   *RClass
    visibility Visibility
}

Dispatch (our objc_msgSend) walks the ancestor chain, falling back to method_missing. From this fall out for free: monkey-patching (mutate methods), define_method (insert), method_missing (fallback), send (computed selector), reflection (read the tables), singleton classes (RClass{isSingleton:true} spliced into the chain).

6. Bytecode & VM

Stack ISA (YARV-style). Literal pushes, locals (getlocal/setlocal with depth+index), ivars/consts/cvars, calls (send{mid, argc, flags, blockiseq}, invokesuper, invokeblock), compound literals, control (jump, branchif, branchunless, branchnil), definition (definemethod, defineclass), stack ops, and non-local exit (leave, throw{type}).

type ISeq struct {
    Insns  []Instr
    Consts []Value
    Locals []Symbol
    Catch  []CatchEntry  // rescue / ensure / break / next / redo / retry
    Arity  Arity
    Name   string
    Source SourceMap     // for backtraces
}

The catch table is the single mechanism for rescue/ensure/retry and non-local break/next/redo/return from blocks — exactly like YARV.

Critique — native-frame unwinding. A clean "status return, never panic" for control flow (see §8) has a hole: when a Go-implemented method (e.g. Array#each) calls back a Ruby block that does break or raise, the native Go frame must unwind — you cannot return a status through a Go for loop. Plan for a hybrid: status returns for VM-internal flow, but panic/recover at the native↔Ruby callback boundary. Design that boundary in Phase 1.

Phase 0 recurses on the Go stack (so fib(20) just works); explicit Frame/Env objects, catch tables, and Fiber arrive in Phases 1–3.

7. Closures, blocks, Fiber

  • Blocks/Procs compile to child ISeqs capturing env + self; yield is invokeblock. lambda vs proc is a flag (return/arity semantics).
  • Fiber ↔ goroutine: each Fiber.new runs a goroutine on a VM loop, synchronised by two channels (resume/yield) in strict cooperative handoff.
  • Enumerator#next (external iteration) is built on Fiber — load-bearing for loop, lazy, and each without a block.

Critique — Fiber cost (raise to Medium-High). Idiomatic Ruby spawns enumerators constantly. Each is a goroutine (≥8 KB stack) with ~hundreds-of-ns channel handoff per #next, exactly where Ruby leans hardest; and an unfinished enumerator is a leaked goroutine blocked on its channel forever — needs finalizer-based teardown.

8. Exceptions & control flow

  • Exception hierarchy partly definable in Ruby; raise/rescue/ensure/ retry/backtrace/cause are primitive.
  • Unwinding mechanism: return a status (ThrowState{type,value,target}) from the dispatch loop rather than Go panic/recover for normal flow; reserve panic for internal VM bugs — modulo the native-frame boundary in §6.
  • break/next/redo/return from a block = throw targeting the right frame via the catch table. StopIteration is load-bearing for Enumerator/loop.

Phase 0 has no rescue yet, so runtime errors are fatal and travel as a panic(RubyError) recovered at the Run boundary; this converges to the status-return design when exceptions land in Phase 3.

9. GC & memory semantics

  • Reuse Go's GC (Ruby objects = Go heap objects). No GC to write.
  • object_id: a weak-keyed pointer→id table (Go ≥1.24 weak package + finalizers) so it does not pin objects.
  • Finalizers: runtime.SetFinalizer with care; ObjectSpace largely stubbed (_id2ref/each_object permanently out — see §4).
  • WeakRef: via Go's weak package.

10. Front-end (the biggest piece)

Prism itself is a hand-written recursive-descent parser. We reimplement the same family. The "no cgo" constraint applies to the shipped binary; MRI/Ripper/Prism are an offline test oracle (§13).

Lexer — where the difficulty lives: lexState (mirrors MRI EXPR_*), spaceSeen (foo -1 ≠ foo - 1), a literal stack for re-entrant "#{…}" interpolation, and a heredoc queue (<<~/<<-/<<"…", several per line).

Parser — recursive descent + Pratt for expressions; a scope stack resolves variable-vs-method-call: each ident = … registers a local, each bare identifier consults scope.isLocal(name).

Incremental grammar growth: subset first, then interpolation → heredocs → %-literals → multiple assignment/splat → keyword args → pattern matching (case/in) → endless methods → beginless/endless ranges → safe navigation → numbered params/it.

Critique — this is risk #1, ahead of regexp, and there is an escape hatch. The grammar is the largest cumulative effort and is currently sequenced as a single late phase (Phase 5); that is the schedule trap. Consider Prism compiled to WASM, run under a pure-Go WASM runtime (wazero) — a production-grade, upstream-tracking parser that stays inside the no-cgo constraint, deleting most front-end risk (cost: a ~MB WASM blob; loses the hand-written-parser learning goal). Tracked as an open decision in §16.

11. Go core vs embedded Ruby (the minimal floor)

In Go (incompressible): object model, dispatch, exceptions/unwinding, Fiber, eval/require, the metaprogramming floor (send, respond_to?, define_method, instance_variable_*, const_*, hooks), and a small kernel per type: Integer/Float (arith + math/big promotion), String ([]byte+enc, pack/unpack), Array (each+indexing), ordered Hash keyed by hash/eql? (not a Go map), Symbol (interning + to_proc), Proc/Method (call), Range (each via succ).

In embedded Ruby (big win): Comparable (from <=>) and Enumerable (from each) — written once, inherited by every conforming type. Then upstream pure-Ruby stdlib: set, ostruct, forwardable, optparse, logger, delegate, singleton…

Go leaves (C extensions in MRI): json (encoding/json), digest (crypto/*), zlib (compress/zlib), securerandom/base64, sockets/IO (net/os), time, math/big.

Note (Symbol): a monotonically-growing global intern table is a memory-exhaustion vector under "x#{i}".to_sym; Ruby 2.2+ GCs dynamic symbols — track as a known limitation.

12. Build chain & single binary

rbgo build app.rb -o app:

  1. Scan the require graph (literal strings) + explicit include config for dynamic requires.
  2. Stdlib selection: only reached libs are kept (.rb embedded + Go leaves via build tags, so the Go linker drops the rest).
  3. Compile app + selected stdlib to bytecode.
  4. Emit a Go file embedding the bytecode (//go:embed) + registration of the chosen libs.
  5. go build → one static binary.

The binary contains: VM + core + selected stdlib + app bytecode + the front-end (for eval). Closed-world mode (opt-in, no eval/dynamic-require/unknown const_get) drops the front-end and enables aggressive DCE → smaller binary. Precedent: mruby's mrbgems, selected at build time via build_config.rb.

13. Test & oracle strategy

  • Unit tests per package.
  • Golden AST: expected ASTs generated offline from MRI (Ripper/Prism), frozen as fixtures.
  • Behavioural conformance: an oracle/ harness runs snippets in MRI and rbgo and compares stdout/result. (Phase 0 already does this against MRI.)
  • Upstream suites: embed a stdlib lib and run its own test suite → each red test points at a missing primitive.
  • ruby/spec: import subsets progressively (the conformance gold standard).
  • Add differential fuzzing of the parser against Prism (random valid-ish Ruby → compare ASTs); fixed goldens miss lexer-state bugs.
  • Performance is tested too, systematically — see §13a.

13a. Performance, go-asmgen acceleration & multi-architecture validation

Performance is a first-class, continuous concern, not a Phase 8 afterthought.

  • Systematic benchmarks against the reference Ruby (MRI/CRuby). The same corpus that validates behaviour in the oracle/ harness is reused as a perf corpus: each snippet is timed in MRI and in rbgo, and the ratio is tracked over time. Microbenchmarks (go test -bench) cover hot primitives (dispatch, integer/float arithmetic, String/Array ops, Hash probing); macrobenchmarks cover representative programs. Notable regressions are gated in CI.
  • Use go-asmgen to accelerate the hot computations. go-asmgen generates Plan 9 SIMD assembly in pure Go with CGO_ENABLED=0, so it stays inside the no-cgo constraint. Candidate fast paths: String/byte scanning, comparison and pack/unpack, UTF-8 validation and transcoding, the ordered Hash's hashing, bignum kernels, and bulk Array operations. Every go-asmgen-backed path sits behind the same byte-identical tests as its scalar fallback (prototype, then measure before/after).
  • Validate on all six supported 64-bit architectures: amd64, arm64, riscv64, loong64, ppc64le, s390x — natively on amd64/arm64 and under qemu for the rest (note: amd64-AVX2 needs a full x86_64 VM, not Rosetta or docker-qemu-user). Both correctness and the perf matrix run across all six, the same target set go-asmgen itself covers.
  • Every arch is validated on real hardware, not just qemu. qemu is the CI gate; real silicon is the correctness-and-performance oracle. amd64 and arm64 run natively (an x86_64 VM with AVX2; Apple-Silicon arm64). The other arches run on the GCC Compile Farm (cfarm) over SSH (user delavennat): riscv64 (RVV) on cfarm95 (a SiFive-class RVV host) for vector/SIMD work, ppc64le on cfarm112 (POWER8E) and cfarm433 (POWER9) — POWER8 is the ISA floor we gate VSX on — and loong64 on cfarm401 (LoongArch, air-gapped, so artifacts are staged in). s390x has no cfarm node, so it is validated on the IBM LinuxONE Community Cloud (L1CC) (big-endian, vector facility). This real- hardware access matters specifically for the SIMD-accelerated modules (base64/securerandom/hex via go-simd) and the performance benchmarks: the numbers reported are measured on real silicon, not llvm-mca estimates.

14. Phasing

Principle: vertical slice first (de-risk the whole chain thin), then deepen.

Phase 0 — Skeleton & vertical slice — ✅ DONE

Lexer+parser+compiler+VM for a tiny subset: integers, arithmetic, locals, if/while, def/call, puts, basic String. Exit (met): puts 1 + 2 and a recursive fib(n) run end to end; the whole chain exists (thin). Also shipped: floats, unless/until, statement modifiers, floor division, print/p, ~92% coverage (object 100%), MRI differential tests.

Phase 1 — Object model & dispatch — ✅ DONE (core)

Live object model: RClass/RObject/Method with mutable per-class method tables, dynamic dispatch (send) walking the ancestor chain, classes with inheritance (class … < …), @ivars, new/initialize, constants, and user-overridable method_missing (default raises NoMethodError). Calls are now receiver-aware OpSend; top-level def defines on Object. Base hierarchy: BasicObject→Object→{Module→Class}, plus Integer/Float/String/True/False/Nil. Kernel on Object: puts/print/p/class/to_s/inspect/nil?. 100% coverage, CI green on 6 arches. Done since: modules (module … end), include (mixins, via the ancestor walk over each class's included modules), super (bare/forwarding, super(), super(args), super arg), and blocks & yield — real closures via an Env parent chain with depth-addressed locals, { |params| … } blocks, yield (with args), block_given?, lenient block arity, and Integer#times as the first block-driven iterator. Done since (in later phases): Proc/lambda and &block reifying a block as a first-class value (Phase 2), singleton classes (def obj.x / define_singleton_method), respond_to? (Phase 4), and the comparison operators routing through send for non-built-in receivers. (do…end block syntax landed in Phase 2; method_missing receives a Symbol as of Phase 2.) Exit (met for core): define classes, subclass, instantiate, call methods, use ivars, override method_missing.

Phase 2 — Go core types + Ruby mixins — ✅ DONE (core)

Integer (bignum), Float, String (bytes+encoding), Symbol, Array, ordered Hash, Range, pack/unpack. Comparable & Enumerable in Ruby. Started: Symbol literals (:name, :name?/:name!), the Symbol class (to_s/to_sym/==), and method_missing now receives a Symbol (matching MRI, closing the Phase 1 String compromise). Array — literals […], indexing a[i]/a[i]=v (negative indices), length/size/push/first/last/ empty?/include?/each/map, element-wise ==, and puts array-flattening. ordered Hash — literals {k => v} (insertion-ordered), h[k]/h[k]=v, size/length/empty?/key?/keys/values/each/select/reject, ==, the Ruby 4.0 Hash#inspect form ({a: 1, "b" => 2} — label form for symbol keys, spaced => otherwise), plus merge/fetch/dig/values_at/ transform_values/transform_keys/invert/to_h/store/delete/has_value? /each_pair, verified against a local Ruby 4.0.5 oracle. Range — literals 1..5/1...5, the Range class with begin/end/first/last/exclude_end?, comparison-based include?/cover?/member? (incomparable members return false rather than raising, and cover? works on numeric and string ranges), min/max, size/count, to_a/each/map, and ==. Integer ranges iterate; Float ranges raise TypeError ("can't iterate from Float") as in MRI. Comparable & Enumerable — written in embedded Ruby via a go:embed'd prelude that VM.New runs after the native bootstrap (the org's USP: each is built on a single primitive — <=> and each). Enabling machinery: the <=> and << operators; operator method definitions (def <=>, def ==, def [] /[]=, …); comparison operators dispatch as methods when the receiver isn't a built-in ordered type (numeric/string keep the inline fast path); native <=> on Integer/Float/String/Object and a default identity Object#==; and blocks made transparent to yield (a block reaches its enclosing method's block) so each { … yield … } works. Comparable gives </<=/>/>=/==/between? /clamp; Enumerable gives map/select/reject/find/include?/to_a/count/sum/min /max/reduce/any?/all?/none?/each_with_index/flat_map/partition/group_by/tally/zip. Short-circuit && / || with Ruby value semantics (yield the deciding operand; the right side runs only when the left doesn't decide), compiled with a conditional branch over a duplicated operand. do…end blocks alongside { … }, with the do after a while/until condition correctly bound to the loop, not to a call in the condition. String methods: length/size/bytesize /empty?, upcase/downcase/capitalize/swapcase/reverse, strip/lstrip/rstrip/chomp /chop, chars/bytes/split, include?/start_with?/end_with?/index, sub/gsub (string patterns), to_i/to_f/to_s/to_str/to_sym, and [] slicing (index, start+len, Range) — rune-aware where it matters. String is mutable (a reference type): <</concat/replace/prepend/insert/[]=/slice!/clear, the bang transforms (upcase!…gsub!), and freeze/FrozenError. String interpolation "…#{expr}…": the lexer emits STRBEG/STRMID/STREND so the embedded expression lexes in the outer scope (a #{x} reads the local x), with per-interpolation brace tracking for nested {} and nesting; parts are coerced with to_s and concatenated. Ternary cond ? a : b (looser than ranges/binary, tighter than assignment, right-associative) desugars to an If expression. case/when (subject and condition forms) matching via ===: Object#=== is ==, Module/Class#=== is is_a?, Range#=== is membership — so when Integer, when 80..89, when 2 all dispatch correctly (subject evaluated once; === lexed via a new EQQ token). Pattern matching case/in (Ruby 4.0): the subject is deconstructed and bound rather than tested with ===. in is a contextual keyword (the clause opener; no for…in in the grammar to conflict). Implemented forms: value patterns (literals/ranges, matched with ===), variable binding (in x, the wildcard _), class/constant patterns (in Integer, via is_a?), the pat => name binding suffix, and array patterns [a, b] / [1, x] / [a, *rest] / [*init, last] / nested — using the deconstruct protocol (respond_to?(:deconstruct), Array#deconstruct returns self, Struct#deconstruct its values; length + element checks). A leading constant gives a const array pattern Point[x, y] (adds an is_a? guard). Guards (in pat if cond / unless cond) gate a matched clause. With no matching clause and no else, OpRaiseNoMatch raises NoMatchingPatternError (a StandardError subclass). Compiled by compileCaseIn/compilePattern: the subject is evaluated once into a hidden slot, each pattern leaves a boolean and performs its bindings as a side effect, OpTruthy normalizes ===/is_a? results. Hash patterns use the deconstruct_keys protocol (respond_to?, then deconstruct_keys(nil); Hash and Struct both return their member hash): in {name:, age:} (shorthand binding), {status: "active"} (value sub-pattern), {a: x} (renaming), {a:, **rest} (except-captures the unnamed keys), {a:, **nil} (forbids extras), and the empty {} (matches only an empty hash). A key absent from the deconstructed hash fails the match. The brace-less top-level form in a:, b: is accepted too. compileHashPattern emits the per-key key? + [] checks. A leading constant gives a const hash pattern Point(x:, y:) (the paren form; adds the is_a? guard, via deconstruct_keys). Pin ^name / ^(expr) matches against an existing value with === instead of binding; alternative p1 | p2 | … matches if any branch does (short-circuiting). One-line patterns: subject => pattern (rightward assignment — binds, or raises NoMatchingPatternError) and subject in pattern (a boolean test), parsed at statement level or in parentheses (r = (v in P)), with = binding tighter. The find pattern [*pre, mid…, *post] scans for the first matching window (compileFindPattern's scanning loop). Pattern matching is complete. More String methods: ljust/rjust/center (rune-width padding) and tr/count/delete/squeeze (with a-z range expansion). ** exponentiation (right-associative, tighter than *//) + Integer /Float **/pow. Beginless/endless ranges (..5, 1.., arr[2..]; nil endpoints, open in ===/cover?/include?), Array range + start,len slicing (shared sliceRange). Optional/default method parameters (def f(a, b = expr); default compiled into the prologue via OpArgGiven, may reference earlier params; ISeq.NumRequired

  • range arity error). Splat parameter def f(a, *rest) (also *all, and combined with optionals) — collects the remaining args into an array (ISeq.SplatIndex; expected N+ arity error). Splat arguments f(*arr) / [*a, *b] — splice an array into a call's args or an array literal (runtime arg-array build via OpSplatToArray/OpConcatArray + OpSendArray). Keyword arguments def f(a:, b: 2) and f(a: 1) — keyword params (required and optional, defaults may reference earlier params) plus the call-site last-hash sugar; the VM validates unknown/missing keywords (ArgumentError, singular/plural). Array sum/each_slice/rotate + flatten(depth); Hash slice/except/merge!. Blocks/Procs/lambdas: &block params reify the method block as a first-class Proc (call/[]/yield/arity/lambda?); call-site &proc block-pass + Symbol#to_proc (the &:sym shorthand, via native-bodied Procs); proc/lambda constructors; and the stabby lambda ->(x){…}. String#%/format/sprintf — Ruby's format engine (d/i/s/f/e/g/x/X/o/b/c + flags/width/precision), Go-fmt backed. Integer lcm/bit_length/digits(base). String lines/each_line/each_char/each_byte. Parser: negative-numeric-literal precedence (-2.abs==2, -2**2==-4). Class methods def self.foo (singleton-method chain, inherited). Constant assignment NAME = v and attribute assignment recv.attr = v + setter defs def name=. Struct Struct.new(:a,:b) (accessors, to_a/to_h/members/[]/==/size). Integer/Float numeric methods: Integer abs/even?/odd?/zero?/ positive?/negative?/succ/next/pred/to_i/to_int/to_f/to_s(base) /gcd/divmod/digits/chr/upto/downto; Float abs/sign predicates/ to_f/to_i/to_int/ceil/floor/round/nan?/finite?/infinite?. The prelude mixes Comparable into Integer, Float, and String, so between? /clamp work on them, and Enumerable into Array and Range, so both inherit select/reject/find/reduce/sum/any?/all?/none?/each_with_index on top of their native each (native methods win where both exist). Block auto-splat: a multi-parameter block called with a single Array destructures it ([[1,2]].each { |a,b| }), which also makes Hash Enumerable — Hash#each yields a [k, v] pair, so map/find/count/any?/all? /none?/to_a operate on pairs (a one-param block sees the pair); select/reject stay native since they return a Hash. Label hash literals: { name: value } lexes name: as a LABEL and is sugar for { :name => value }, mixable with the hashrocket form. break / next: a compiler context stack resolves them to the innermost block or loop. next [v] returns from a block frame or continues a loop; break [v] exits a loop or terminates the iterator a block was passed to — and unwinds to the correct call site even through a Ruby-level iterator (map { break } returns from map), via an OpBreak panic tagged with the executing block's identity. Compound assignment: += -= *= /= %= <<= ||= &&= desugar to lhs = lhs OP rhs for locals (a fresh-slot-aware OpAssign node so x ||= v defines x), ivars, and recv[i] OP= v. Top-level @ivars: main is now a real object with its own ivar table, shared with top-level method bodies (closing the Phase 1 no-op-set quirk). Array methods: sort/sort_by/reverse/uniq/flatten/compact/join/index/take /drop/min_by/max_by/each_with_object (on top of the inherited Enumerable set); ordering goes through <=>. Plus, since: user subclassing of String/Array/Hash (a subclass instance wraps the value, keeps its class identity, ivars and user methods while the value type's own methods and operators apply), Object#eql?, the set operators (uniq/&/|/-) comparing with eql?, and a broad MRI-differential conformance sweep (Symbol/Module reflection, Numeric rounding, Class.new, Integer() parsing, …). Exit: most "ordinary" Ruby runs. Done since: Array#pack/String#unpack/#unpack1 (the common directives C/c S/s L/l Q/q n/N v/V a/A/Z H/h U), and String encodings — an encoding tag on strings, the Encoding class (UTF_8/ASCII_8BIT/BINARY/US_ASCII), #encoding /#force_encoding/#b/#ascii_only?/#valid_encoding?, length counting bytes for a binary string, and the tag propagating through +/*/slicing/<</dup. Still ahead: the multi-encoding transcoding rules (#encode, incompatible- concat Encoding::CompatibilityError) — a Phase 8 refinement.

Phase 3 — Control flow & exceptions — ✅ DONE (core)

Exception hierarchy, raise/rescue/ensure/retry, non-local break/next/redo/return via catch tables, StopIteration, Fiber + Enumerator + loop + lazy. Landed since: Fiber (Fiber.new/resume/Fiber.yield), Enumerator (external iteration .next/.peek on each-based enumerators, with_index), Kernel#loop, and lazy enumerators ((1..Float::INFINITY).lazy.map{}.first(n)). Still ahead: an infinite Enumerator.new generator driven by first(n) / external next (needs fiber-based suspension; finite generators, the each-based form and lazy already work). Started: the exception hierarchy (Exception → StandardError → RuntimeError + the built-in error classes with correct superclasses), Exception #initialize/#message/#to_s, Object#is_a?/#kind_of?/#instance_of?, and Kernel#raise (string → RuntimeError, class → instantiated, instance → re-raised, class + message). begin/rescue/ensure/else via a re-entrant exec loop with a per-frame handler stack (OpPushHandler/ OpPopHandler, a deferred recover that resumes at the rescue dispatch and lets non-exception panics through); rescue matches by class with is_a? (bare = StandardError), class lists, => var binding, and OpReThrow on no match; else on the clean path, ensure on both normal and propagating paths; internal raises (1/0, NoMethodError, …) are rescuable; bare raise re-raises; retry re-enters the begin body from a rescue (ensure still runs once); method-level rescue/ensure — a def body carries rescue/else/ensure clauses without an explicit begin (shared parseRescueTail); and Kernel#loop (block-driven infinite loop, terminated by break).

Phase 4 — Full metaprogramming — ✅ DONE (core)

define_method, instance_eval/instance_exec, class_eval, constant machinery, hooks (included/inherited/method_added/…), define_singleton_method. (Refinements: deferred.) Landed since: define_method, instance_eval/instance_exec, class_eval, define_singleton_method, extend/prepend, string eval + Binding, the included/inherited/method_added hooks, Module#name/instance_methods /const_get/const_set/const_defined?, and Object#methods. Still ahead: extended/prepended hooks, ObjectSpace, and refinements (deferred). Started: reflection/dispatch — send/public_send (forwarding the block), respond_to?, itself, tap, then/yield_self; Kernel conversion methods Integer/Float/String/Array (capitalized Name(...) now parses as a call). Landed: string eval — Kernel#eval(str) compiles and runs Ruby at runtime against the caller's self (sees self/ivars/methods/constants; def lands in the caller's context; fresh local scope). Parse/compile errors raise SyntaxError (now under ScriptError→Exception, not StandardError); non-String arg → TypeError. Binding — binding captures the live frame (OpBinding: env/self/definee + the ISeq's local names); Binding#eval(str) and eval(str, binding) compile the string as a child block scope seeded with the binding's locals, so eval'd code reads and writes them (locals it newly introduces are scratch). Plus local_variable_get/set/defined?, local_variables (MRI's injected-first ordering) and receiver. Class/module hooks — inherited (on subclassing), included (on include) and method_added (per instance-method def), each looked up as a singleton method so it fires only when defined. (extended/prepended wait on extend/prepend.)

Phase 5 — Complete front-end — ✅ DONE (core)

Full grammar: heredocs, interpolation, %-literals, multiple assignment, keyword args, pattern matching, endless methods, beginless/endless ranges, safe navigation, numbered params/it. (Or adopt Prism-on-wazero — see §16.) Landed: interpolation, keyword args, multiple assignment / destructuring, full pattern matching case/in (value/bind/class/array/hash/find/pin/ alternative patterns, guards, one-line =>/in, deconstruct/ deconstruct_keys), endless methods, beginless/endless ranges, safe navigation &., numbered params (_1/_2) + it, heredocs (<</<<-/<<~, interpolating and literal), the %w/%i/%q/%Q/%W/%I percent-literals (in statement and command-argument position), optional (name = default) and *splat params in block and stabby-lambda lists, the proc.() call shorthand, and paren-less command calls on a method-call result (Fiber.yield 1), if/unless and while/until as expressions, leading-dot method chains across newlines, and a &block param in block lists, and begin…end/case…end as a paren-less command argument. The front-end now parses essentially all idiomatic Ruby — a real-world ActiveSupport core_ext idiom sweep that was 9/15 is now 15/15. Still ahead: only nicher literal/interpolation edge cases.

Phase 6 — Standard library — 🚧 substantially landed

IO/File/Dir, Time, Random, Thread/Mutex/Queue, Regexp (§16), Marshal; embed pure-Ruby libs; implement Go leaves (json, digest, zlib, securerandom, base64). Run upstream suites. Landed: IO/File/Dir/StringIO, Time/Date, Random, Thread/Mutex/Queue (emulated GVL), Marshal (MRI byte-exact), Set, the Regexp bridge (below), and the Go leaves json/digest/zlib/base64. Scientific

  • container bindings (NDArray/FFT/Image/BigDecimal) are wired in too. Still ahead: running the full upstream stdlib suites against real-world corpora (see the conformance-ladder note below).

Regexp bridge — 🚧 landed (6a–6d): the go-ruby-regexp/regexp engine is wired in (CGO=0).

  • 6a — /re/imx literals (lexer disambiguates / via the value/operand state), the =~ operator, a vm-level Regexp value + cRegexp (source/to_s/inspect/match/match?/=~/===), a MatchData value + cMatchData ([] by index or name, pre_match/post_match/to_a/captures/ begin/end/named_captures/size), and String#=~/match/match?. Ruby flags are translated to an inline (?imx) prefix; RegexpError for bad patterns.
  • 6b — String#scan (whole matches, or capture arrays when groups are present; block form yields each).
  • 6c — String#sub/gsub over a Regexp: replacement templates (\0/\&, \1..\9, \k<name>, \``, '`) or a block.
  • 6d — String#split over a Regexp: interpolated captures, field limit, and the awk-style whitespace mode (no arg / nil / " ").

Byte vs character offsets: go-ruby-regexp reports byte offsets, while Ruby's MatchData#begin/#end and =~ report character offsets; the bridge converts (byteToChar). Matched substrings are representation-independent. Differential-tested against MRI (Ruby 4.0.5): offset-exact on ASCII, substring comparison for multibyte. Since go-ruby-regexp's encoding-aware cursor (engine b6bfa2d), a bare . and the byte-oriented classes advance by a whole UTF-8 character in the default UTF-8 mode, so /./ matches é and "café".scan(/./) yields four characters — matching MRI. (Literal multi-byte class members like [é]/[à-ï] are the remaining engine-side item.)

The $~/$1..$N/$&/$`/$' match globals are now supported (VM-global last match). Still deferred: Regexp.new from a string, the (?'name') named-group spelling (unsupported by the engine), and the sub/gsub enumerator and Hash-replacement forms.

Phase 7 — Build toolchain — ✅ DONE

rbgo build, require-graph scan, selection (build tags), //go:embed, single static binary, closed-world mode.

Landed. AOT method compilation (see aot-compiler.md) plus closed-world builds: rbgo build --closed freezes the program's compiled bytecode to Go literals (internal/aot.FreezeISeq), injects it via go build -overlay, and builds under the rbgo_closed tag. That tag drops the lexer/parser/compiler from the link — the front-end is isolated behind a single parseCompileFn seam (frontend_open.go / frontend_closed.go), eval/require of source raise NotImplementedError, Binding#eval is stubbed, and the prelude loads from its frozen bytecode (embeddedPrelude, kept in sync with prelude.rb by a deep-equal drift test) instead of being parsed at startup. A require-graph scan (internal/aot.FrontendUses) reports any eval/require the program makes so the user knows what would raise. The result: a smaller, self-contained binary that runs with no source file. (Per-class stdlib tree-shaking beyond the front-end was measured and intentionally skipped: the Go-implemented bindings are small pure-Go libraries — go-fft ~7 KiB, go-ndarray ~20 KiB, go-images ~40 KiB, regexp ~57 KiB of symbols — so dropping all of them would save under 2% of a binary dominated by the Go runtime. The front-end drop is where the size comes from.)

Phase 8 — Conformance & performance — 🚧 in progress

ruby/spec subset; optimisations: dispatch inline caches (key = class), fixnum cache, reduced boxing. (JIT out of initial scope — Go does not help with runtime native codegen.) Landed: the performance trio — dispatch inline caches (class + global method serial), small-integer interning (fixnum cache), and Env-struct pooling (capture-tracked, halving per-call allocations) — plus a large, ongoing MRI-differential conformance effort closing stdlib gaps batch by batch against Ruby 4.0. The ruby/spec language + core suites run in CI as a shrink-only ratchet (scripts/conformance/rubyspec/, MSpec-compatible shim): every PR must keep the passing-example total at or above the frozen floor, so conformance is tracked and can only go up. Recent batches added, among others, String#dump / #undump / #hex / #append_as_bytes, the Array#pack / String#unpack u (uuencode) and w (BER) directives, and #numerator / #denominator on Integer / Float / Numeric — each landed at 100 % package coverage with a zero-regression sweep.

Still ahead: the bulk of the remaining fail/error mass is in deep subsystems, not isolated methods — IO (real file descriptors + transcoding), File / Dir / ARGF, Time, Process / spawn, ENV, and Marshal (byte-exact) — plus cross-repo engines (go-ruby-regexp character classes, go-ruby-format) and the front-end carrying source positions (backtraces, __LINE__, const_source_location currently report line 0). The library/ suites (socket, net-*, openssl, zlib, …) are a further, largely networking/OS-bound frontier. Contained single-method gaps (case-mapping special-casing, Range#include?/#cover? edges, string index/chomp/gsub corners) are shrinking and continue to land one PR at a time. The remaining hot-path optimisation (the operand-stack/value-stack redesign) was assessed and deferred as high-churn for its gain.

Conformance & benchmark ladder (real-world corpora). Beyond ruby/spec, validate against progressively harder real Ruby, and benchmark against other implementations:

  1. JRuby as a second oracle — extend the MRI differential harness to a three-way MRI / JRuby / rbgo comparison, and add JRuby as a third bar in the perf benchmarks (rbgo is CGO=0 pure Go; JRuby is JVM-JIT).
  2. Pure-Ruby gem test suites as conformance corpora — e.g. ActiveSupport's core_ext (String/Array/Hash/Numeric/Date extensions, all pure Ruby) exercises the stdlib hard; run such suites to drive the remaining gaps (encodings, the parser tail) by demand rather than guesswork.
  3. Real-world workloads — OpenVox (the community Puppet fork; Puppet's manifest evaluation is Ruby-heavy and aligns with config-management interests) and ultimately Ruby on Rails. Full execution needs C-extension and threading depth that is out of scope, but their pure-Ruby subsets and unit suites are achievable targets and strong signals. A spot-check of ActiveSupport- and Puppet-style idioms (each_with_object, method_missing delegation, Comparable, deep dig, type case/when, define_method loops, squiggly heredocs) already runs; the near-term blockers are %w in argument position and String encodings.

15. Risk register

Risk Severity Mitigation
Parser scope / front-end effort Highest Subset-first + oracle; evaluate Prism→WASM on wazero (§16) to delete most risk; test-driven growth
Regexp (Onigmo vs RE2: no backrefs/lookbehind in Go regexp) High Pure-Go Onigmo reimpl., standalone module (go-ruby-regexp/regexp); ReDoS via memoization + timeout
Native-frame unwinding at the Go↔Ruby callback boundary High Hybrid: status returns internally + panic/recover at native callbacks; design in Phase 1
String/encodings (mutable, multi-encoding) High []byte+encoding tag from Phase 2; incremental transcoding
Value boxing vs Go GC (no MRI-style immediates) High Position as embedding-not-speed; split Fixnum/Bignum + fixnum cache early; inline caches Phase 8
Fiber/Enumerator cost & goroutine leaks Medium-High Finalizer teardown; consider pooling; document the perf cliff
Thread/Ractor model baked late Medium Decide early (§16); single-thread + Fiber to start; Ractor needs early design
Marshal / ObjectSpace Low Stub; _id2ref/each_object permanently out
C-extension gems — Out of scope (pure Ruby + Go leaves only)

16. Decision journal / open questions

  1. Value representation — interface + concrete types (start) → split Fixnum/Bignum + fixnum cache (early) → inline caches (Phase 8). Settled: interface first; accept the perf positioning.
  2. Regexp engine — settled → pure-Go reimplementation of Onigmo (Ruby's engine), faithful backtracking VM, as a standalone reusable Go module in the sibling org go-ruby-regexp (repo regexp; see go-ruby-regexp/regexp/docs/plan-regexp.md). ReDoS handled by memoization + Regexp.timeout (as Ruby ≥3.2).
  3. Front-end strategy — settled → hand-written lexer/parser (learning goal, full control), now extracted to the standalone sibling module go-ruby-parser/parser (which this VM imports). Prism→WASM-on-wazero was considered and not taken. The grammar is feature-complete for the core language (see Phase 5); a few literal/command-call forms remain.
  4. Native↔Ruby unwinding — settled → hybrid (status internally + panic/recover at native callbacks); design the boundary in Phase 1.
  5. Thread model — settled → emulated GVL (MRI semantics: one Ruby thread runs at a time, race-free under Go's detector), shipped with Thread/Mutex/Queue. Real parallelism and Ractor remain out for now (Ractor would need early design to be added later).
  6. Target version — Ruby 4.0 semantics.
  7. Oracle tool — Ripper and/or Prism offline (dev only, not linked).
  8. Performance & multi-arch — settled → benchmark systematically against reference Ruby (MRI) from the start (reuse the oracle corpus as a perf corpus, gate regressions in CI); use go-asmgen to accelerate hot computations (CGO=0 SIMD) behind byte-identical tests; validate correctness and performance on all six 64-bit arches — amd64, arm64, riscv64, loong64, ppc64le, s390x (see §13a).

17. First concrete steps (session 1) — done in Phase 0

go mod init · internal/ tree · token/ · lexer/ (with spaceSeen and a minimal lexState from the start) · oracle-style fixtures · ast/ · parser/ (Pratt + scope stack) · compiler/ · vm/ · cmd/rbgo run. Exit criterion (met): puts 1 + 2 and fib(20) run.

18. Prior art & neighbours

  • goruby/goruby (~600★, MIT) — the notable "Ruby in Go", but a tree-walking interpreter (Thorsten Ball lineage), partial, learning-oriented.
  • towski/goruby — a small pedagogical bytecode interpreter; close to our approach, worth reading.
  • goby-lang/goby — a Ruby-inspired OO language, not a compatible implementation.
  • cgo bindings (out of scope: cgo) — go-mruby, go-oniguruma/rubex.

Positioning of go-embedded-ruby:

Axis goruby/goruby go-embedded-ruby
Execution model AST-walking Bytecode VM + embedded front-end
Single binary / tree-shaking No Yes (build-time selection, build tags)
Dynamic eval/require No Yes (embedded front-end)
Regexp Absent Pure-Go Onigmo (go-ruby-regexp/regexp)
Stdlib strategy — Go core + embedded Ruby
Compatibility target Partial subset Ruby 4.0 semantics, test-driven growth

Reusable as reference (MIT): the lexer/parser/object model of goruby/goruby; the bytecode approach of towski/goruby. To study, not to copy — our VM diverges from the compiler onward.

Naming decisions

  • Org go-embedded-ruby (consistent with the go-* org convention; "embedded" names the zero-cgo embeddability USP better than a bare name). Main repo ruby (module github.com/go-embedded-ruby/ruby), CLI rbgo.
  • Regexp engine in its own org go-ruby-regexp, repo regexp — reusable beyond Ruby.