Skip to content

DEV.md

Latest commit

 

History

History
124 lines (81 loc) · 27 KB

DEV.md

File metadata and controls

124 lines (81 loc) · 27 KB
Raw

Development

Install from source

Requires Python ≥3.10 and Rust ≥1.98. In your Python environment:

git clone https://github.com/AnswerDotAI/basedpl.git
cd basedpl
pip install .

For a standalone executable without Python, run cargo install --path .. Cargo installs it in its bin directory, normally ~/.cargo/bin; put that directory on your PATH.

Commands

cargo test
cargo run -- -e '2×3+4'
cargo fastfmt
maturin develop
pytest -q
ship-rs-build

Rebuild with maturin develop after Rust changes before checking the installed extension. Cargo tests alone do not update the editable Python installation. Use cargo fastfmt, not cargo fmt.

The development profile uses optimization level 1 without LTO. Tests inherit these settings. Debug information, assertions, overflow checks and incremental compilation remain enabled.

CI uses the development profile for Rust and Python tests. Distribution wheels use the dist profile: optimization level 2, no LTO, 16 codegen units, no incremental compilation and stripped symbols.

Structure

  • lib/*.apl: Dyalog dfns adapted from April and the Dyalog dfns workspace. Reference cases load these shared definitions with •LOAD; case-specific setup stays in each test. See lib/README.md for usage and provenance.
  • array.rs: ordinary Value atoms and immutable shared arrays; checked construction, recursive prototypes, axis offsets, direct/mapped result frames, cell descriptors, padded cell assembly and row-based display. Arrays retain function handles and cache lexical dependencies without owning lexical frames or Python objects.
  • number.rs: Integer/Exact/Float/Complex values, normalization, checked arithmetic, promotion, comparison and structural conversion. i64 and BigRational share one exact domain. Checked integer overflow falls back to BigRational; integral rational results return to i64 when they fit. Representation is private. Complex values use num_complex::Complex64, normalizing exactly zero imaginary parts to Float.
  • agreement.rs: leading/unit-axis broadcast shapes and index maps, shared by scalar functions, Each, rank frames and explicit scalar axes. Equal-shape and repeated-block mappings avoid general coordinate calculations. No expanded input copies.
  • keyed.rs: keyed arrays. Keys holds element names in ravel order with a name-to-position hash, stored in an optional field of the array. Every ordinary constructor leaves that field unset, so a primitive drops keys unless it reattaches them. construct and merge build keyed arrays, align puts two keyed arguments in one layout, and retain keeps keys on a positional result of the keyed argument's shape. Structural functions carry keys through carry_keys in primitive.rs. Dot access is a binder rewrite to Pick in eval.rs. The design and its decision log are in meta/keyed.md.
  • primitive.rs: primitive identities/glyphs, valences and array-level implementations. Replicate, products and assignment retain their own agreement rules. Allocation caps are separate from numeric-to-integer conversion.
  • number_theory.rs: segmented prime enumeration, Miller–Rabin primality and Brent/Pollard–rho factorisation; exact integer results and scalar-cell assembly.
  • system.rs: case-insensitive •Name table for constant arrays and native functions. System functions use ordinary function nodes and application. primitive.rs contains single-character primitives.
  • polynomial.rs: coefficient/factored/exponent-table forms, Horner evaluation, companion-matrix roots through faer, and analytic polynomial gradients/VJPs.
  • selection.rs: temporary labels for selective assignment. The binder marks their data flow and permits only selection functions; masks still read real user bindings. Option<SelectionKind> distinguishes ordinary evaluation, whole-item selection and element selection. Nested labels retain their storage and paths. Prototype labels preserve empty-cell structure.
  • display.rs: boxed-array diagrams, function trees and session display settings. Returned values and ⍕ remain independent of these settings.
  • syntax.rs: byte-spanned lexer and structural parser, character/numeric literals, definition kind/full span, structural guards and default arguments, and distinct array-literal/index-bracket nodes. Newlines are Dyalog 20 literal separators within parentheses/brackets. parse returns complete syntax, incomplete input, or a structural error without evaluating expressions.
  • eval.rs: persistent session, explicit right-to-left category-reduction stack, and shared primitive/operator/dfn/train calls. Structural resolution consumes one item. One category table selects binding actions, priority and waiting states. Grammatical reduction returns application requests to the evaluator; it never calls functions itself. APL and Python share operand normalization and validation in Function::new. Reduced entities rebind against their right context when their category changes. Functions have immutable shared nodes; unfinished strands/trains/bound-left arguments exist only in the binder. No per-glyph arithmetic precedence. Output and the final result are separate from errors.
  • error.rs: retained source text, byte spans, inspectable error kinds, and readable Unicode-width diagnostics with separate call-site context. Tabs use four-column stops; other control characters are escaped.
  • cli.rs / main.rs: native command and REPL. The REPL evaluates the parsed result once, not during completeness checking.
  • kernel.rs: native kernmini adapter. ThreadWorker owns the interpreter on its usual large-stack thread; async transport and interrupts remain responsive. Implicit display becomes Jupyter results, explicit output becomes stdout, and completion reuses the REPL glyph matcher plus session names. The wheel installs its kernelspec from wheel/data/share/jupyter/kernels/basedpl/. Inspect/history use kernmini's defaults; subshells are not advertised.
  • editor.rs: Rustyline terminal adapter using the shared naming table. Only typed backtick names auto-expand; Tab is an explicit completion request. Bracketed paste/history/navigation cancel automatic expansion. Rustyline owns terminal modes, editing, in-memory history and the final newline on Ctrl-D. No history file or input rewriting in the interpreter/frontends.
  • symbols.rs: shared glyph, canonical name, monadic name, dyadic name and extra completion aliases. Used by editor.rs and exposed as basedpl.symbols for Python exports and notebook JavaScript completion. Add or change names here, not in individual consumers.
  • python/basedpl/keyboard.json: shared Alt-chord map, embedded by editor.rs and packaged for editor adapters. Keys are US characters after Shift, before Alt. Chords insert literals even in strings/comments.
  • protocol.rs: JSON-lines encoding over ordinary Rust values and sessions. No protocol types enter evaluation or arrays.
  • execution.rs: evaluation deadlines and a thread-safe interrupt flag. Context lends primitive code the current execution control and source span. Cancellation unwinds through ordinary errors but bypasses APL guards.
  • worker.rs: sequential evaluations with a separate stdin reader for control messages. python/basedpl/worker.py owns process lifetime, deadlines and hard-kill fallback. No APL execution occurs on the reader thread.
  • reference.rs: independent fixture decoding and structural comparison, shared by Rust tests, the worker's case request and Python's private _check_reference. Every reference case gets a fresh session.
  • python.rs: optional PyO3 boundary. _Array and _Function hold native values; _Session owns a channel to a Rust worker thread. That thread creates and destroys its own evaluator. Requests and replies carry shared native values and retained diagnostics, never Python objects. python/basedpl/__init__.py provides arrays, conversions, thread-backed sessions and errors. functions.py constructs functions and exports word names from symbols.rs. _cli.py forwards arguments to the Rust CLI.
  • tests/core.rs: storage, ownership, parser diagnostics, API behaviour and cross-call recovery checks. Use equiv_in! { &mut session; code => expected_apl, ... } for session workflows and fails_in(&mut session, kind, &[code, ...]) for errors in the same session. Expected expressions run in fresh sessions. Keep Rust constructors for foundational and representation checks. Self-contained language cases belong in tests/reference/core.apl. tests/cli.rs exercises the actual native process. Documentation examples share one session per APL code block. Unannotated code supplies setup; ⍝ introduces an independently evaluated expectation.
  • python/basedpl/reference.py: Source importers, Dyalog expectation capture, scan/review/activation, and Corpus inventory access. Corpus searches and patches tests/reference/inventory/*.jsonl from a kernel. Default views omit large expectations; request fields explicitly. scripts/reference.py is a thin scan/review/activation CLI. python/basedpl/apltests.py reads/writes .apl records and appends reviewed inventory cases.
  • tests/reference.rs and tests/reference/*.apl: bAsedPL semantic cases plus acceptance tests from ngn, April, APLcart and Dyalog documentation. .apl files determine active coverage. Use ⍝⍝ section headings and short case descriptions for non-obvious checks. Optional ⍝ ⎕: lines assert explicit output. core.apl retains exact Rust array equality, including numeric domains. Implementation gaps belong only in pending JSONL records, never passing error tests. See tests/reference/README.md for format and workflow.

Value is a number, character, function or shared ArrayData. Array storage is Integer(Vec<i64>), Float(Vec<f64>) or Mixed(Vec<Value>). Atoms, unit arrays and singleton vectors remain distinct. Checked constructors preserve elements and select compact storage without numeric promotion. as_integers() and as_floats() expose borrowed slices. at() and elements() yield owned values without materializing the array. Shape, prototype, numeric-domain summary and lexical dependencies are cached in immutable Arc storage. Shape and tally return exact integers directly from dimensions. Nonempty arrays derive their prototype from the first item; empty construction requires a prototype. Prototype filling memoizes shared nested arrays and retains function handles. Function equality is identity; ordering is unsupported. Values and shared function nodes are Send + Sync.

Float arithmetic/comparison dispatches outside slice loops. Primitive numeric folds bypass scalar-array allocation and interpreter calls. Homogeneous float sum/product reductions use Rust 1.98 algebraic operations, including axis reductions; grouping and bitwise reproducibility are not promised. All scans use successive left accumulation. Seeded and unseeded forms share lane traversal. Reduction starts from its whole seed on the right; scan starts from its whole seed on the left. Each axis lane uses the same seed. Rank supplies separate seeds per cell. Generic reductions remain right-associated at every rank. Generic functions keep their call order and side effects. Exact arithmetic is unchanged. Finite-result checks remain at construction boundaries; division retains 0÷0=1. No fast-math flags, custom SIMD intrinsics, CPU-specific wheel flags, compensated summation or strict/fast modes are used.

Inverse dispatch carries an optional fixed argument: the pair's Boolean is true for a fixed left argument and false for a fixed right argument. Commute switches sides. Composition, binding and dyadic Behind propagate that information. Inverse scans apply the operand's dyadic inverse to adjacent accumulators, using the seed for the first pair. They share forward scan's axis/seed validation. General forks, monadic Behind and arbitrary dfns require rules beyond this propagation and remain unsupported.

The numeric policy and current limits are documented in README.md. serde_json is a frontend dependency; PyO3 remains optional. JSON requests decode directly to String, one per line, with no object envelope or custom escaping. Responses remain structured objects. Keep JSON and Python conversions separate: they implement different external contracts.

Lexical frames are a stack with non-owning parent indices, separate from dynamic call/handler state. Nested functions see live lexical bindings, not snapshots. Plain dfn name assignment is local; modified, indexed and selective array assignment update the nearest existing lexical binding. Each update retains its resolved owner and original array across modifier calls. The write does not look up the name again. Arrays may contain functions, including active local captures. Returns and outer updates reject functions or arrays whose lexical dependencies would not survive. This includes empty prototypes. Tail calls retain lexical dependencies in argument arrays as well as the called function. Dfns can return functions directly. Public export walks the shared array/function graph once and rejects active frame references. Operator derivation retains operand values without running the body or creating an invocation frame. Frames pop on success and error; no collector is needed. Execute uses the same lexical capture rule for newly created definitions. Python associates exported dfns and arrays containing them with their originating session for global lookup; primitive-built functions are free. Rust callers supply the session when calling a retained function. Recheck the lifetime argument before adding namespaces, nonlocal function assignment or escaping lexical closures. Dyalog reference runs and the scope boundary are recorded in meta/PRD.md gate L.

Explicit stranding ˘ is structural syntax: maximal chains bind before implicit stranding, functions and operators. Each item contributes one element, including array and function values. Compound items require parentheses. Items evaluate right-to-left through ordinary binding; named strand targets use the existing destructuring rules. First ↑, Pick and complete atomic bracket indices return stored values through the shared call-result path, including functions. Array indices supply result frames; partial coordinates retain trailing cell axes. Empty coordinates preserve the argument. Array consumers and cell assembly store function results as elements. Python can construct Array([plus, times]) and retrieve callables with first, pick, .py or .np. Session association follows structural array operations and function operands. JSON rejects functions and function-containing arrays rather than exporting lossy source strings.

Agenda selector◶cases stores its operands in an ordinary composed-function node. Construction validates the nonempty function vector and constant selectors. Function selectors run once per call. Scalar indices use the shared 1-origin conversion. The selected branch receives the original arguments through shared function dispatch.

Catch-all and numbered guards checkpoint the installing frame's local binding map after the condition executes. Restoring it removes later introduced locals and restores previous local values, including modified assignments. Outer/global writes and output are not rolled back. This deliberately omits Dyalog's distinction between rebinding a local and modifying its existing binding. Assignments within the condition survive rollback. Guard handlers are popped before execution and unwind dynamically through ordinary calls. Cancellation and unsupported-feature errors are not caught. Ordinary and error guards may have an empty result; selecting one returns no value.

Empty Each calls its operand once, replacing only empty arguments with their prototypes. Function::call_prototype scopes prototype mode around the ordinary call path. Compositions, dfns, dops and helpers inherit the mode. Pick uses structural prototype selection throughout that call. Other errors and explicit output remain observable. The caller's mode is restored on success and error.

eval_with and call_with install per-evaluation EvalOptions. echo defaults to true; false suppresses implicit display at top level and inside execute, without suppressing explicit output or display commands. An optional output sink receives explicit/display events as they occur instead of collecting Evaluation.output; ordinary callers retain the capture interface. Clone the InterruptHandle to another thread or supply a timeout. Checks run at binding/call boundaries and inside long interpreted/primitive loops. Tight bounded float kernels keep their existing slice paths; native-library calls and individual BigInt operations are not preempted. Python Session uses cooperative cancellation only. The separate process Worker.eval allows a grace period before killing an unresponsive process. Killing loses the session; requests are never replayed.

Session::call resolves a function expression and invokes it with one or two existing arrays through the ordinary APL call path. call_function_with accepts a retained node. Neither binds temporary argument names. Evaluation.function exposes an unshy exportable function result; set_function checks and binds a function. PyO3 _Session.request accepts code or a native function, native arguments/bindings, timeout and echo. The process worker separately decodes bindings and args in protocol.rs; exact JSON integers must not pass through f64. The process protocol does not export native function handles.

Public Python Session uses a worker thread. Both calls and .eval() request echo=False: calls print explicit output and return an Array or an unshy Function; .eval() returns Result without printing. Errors follow the same print/capture rule. Keywords bind APL names, including native functions; timeouts are session attributes. Finalizers close abandoned sessions; context managers close them promptly. Array retains the native value losslessly. .py and .np make Pythonic copies; NumPy is optional and lazy. Python integers transfer through PyO3's BigInt support without decimal-string conversion. Array indexing uses the shared Rust selector, origin one. Arithmetic and function construction use native nodes, never generated APL source.

.fn() parses once and retains a late-bound expression. Function nodes cache whether they contain late-bound operands. At a call boundary, resolution rebuilds affected nodes with their session's current bindings; unaffected nodes remain shared. A memo preserves shared function graphs. Reduction identities, inverse recognition and primitive fast paths then see ordinary functions. Cyclic name resolution errors at the depth limit. The binder still resolves ordinary APL names at execution time. Python word names select direct-call valence and currying; operators use the underlying APL function. Function association is separate from node ownership: free plus bound uses the bound session, and different sessions are rejected.

Dfn return selection follows statement syntax, not display shyness. A final assignment returns its value silently; a non-assignment call returns immediately even when its result is silent. Empty bodies and exhausted guards return no value. Default ⍺ assignment skips its RHS when supplied and does not itself return a value. Each invocation shadows its caller's ⍺.

The binder returns either a value or a tail application at eligible return positions. Defined calls loop over tail applications and discard frames above the callee's highest lexical dependency. This dependency is cached with immutable function nodes, including function operands and train arms. Tests run 10,000 tail calls with one frame, or two when an outer lexical binding is retained. Installed handlers disable tail reuse. Non-tail evaluation and retained lexical frames have a 1,024-level limit. CLI and Python execution threads reserve 256 MiB stacks through execution::thread; with_stack provides the same boundary for Rust callers. Cargo config sets RUST_MIN_STACK for test threads. Direct Session calls use the caller's stack. Tail call diagnostics retain the final tail site, not an unbounded history.

Flat binding/operator derivation and assignment chains use explicit vectors. A common evaluation-depth budget covers groups and all function representations; function construction separately limits graph depth, protecting recursive application and destruction. Neither limit is an execution-time sandbox. No stack-growth crate or CPS conversion is needed for the current subset.

An assignment arrow drains its right-hand binding stack, assigns to the structural target suffix and resumes binding the prefix. Its RHS is never evaluated twice. Target recognition uses runtime categories at top level and dfn-local name rules inside definitions. Statement return selection distinguishes an assignment from an expression containing one. Modified strands call their modifier left-to-right; ordinary strands assign right-to-left.

Numeric comparisons use fixed relative tolerance 1e-14 when approximate, exact rational comparison otherwise. Each tolerance-sensitive operation must ship with independent inside/outside-tolerance cases: comparisons, membership/index-of, match, unique/grouping, and floor/ceiling. Reuse the rounded pair 0.3 and 0.1+0.2 across applicable operations, with an outside-tolerance control, zero/negative boundaries, and exact/mixed counterparts. Keep structural Rust assertions exact; do not use the language's comparison as the test oracle. The concrete acceptance matrix is in meta/PRD.md §9.4.1. The reference corpus deliberately retains unsupported cases; enable them as their requirements are met.

Numeric semantics are Rust-owned. Explicit Python conversions copy exact integers through PyO3 and non-real values through PyComplex::from_doubles. Fraction components transfer as native integers. No Python numeric objects enter the core. JSON remains a separate process boundary: exact integers use arbitrary-sized JSON integers with serde_json's arbitrary_precision feature. Fractions retain tagged decimal-string components, complex values a tagged numeric pair. Reference-interpreter cases compare equal numeric values across exact/float domains because Dyalog has no corresponding explicit exact domain. Rust core tests assert the exact representation, prototypes and compact buffers separately.

Complex arithmetic extends the existing scalar/array/operator dispatch, not a second evaluator. The lexer shares one real-component scanner between ordinary and ajb literals. Equality uses magnitude-based tolerance separately from real ordering; counts still require exact integrality and zero imaginary part. Approximate prototypes/identities normalize to Float. Complex division scales its denominator and direction scales its input to avoid avoidable squared-magnitude overflow/underflow. Powers, logs, circle functions and factorial/binomial extend this numeric layer. Complex components/results remain finite. Real Float values admit ±infinity but never NaN. Comparisons handle infinity before tolerance or exact-to-float conversion. The compact float path explicitly checks zero divisors. JSON uses signed infinity tags; Python uses floating infinities. Dyalog 20.0.53963.0 executions supply structured documentation-example expectations, except labelled bAsedPL policy differences.

The private _core._Session is a thread-safe handle, not the evaluator itself. A request mutex serializes callers; waiting releases the GIL and polls Python signals. A separate state mutex protects the sender and active interrupt handle. No lock wraps evaluation or array data. Each request owns a fresh cancellation flag. Ctrl-C requests cancellation, drains that reply and raises with captured output. Closing drops the sender and interrupts active work without joining an uninterruptible operation. The worker owns no Python objects and drops the evaluator on its creating thread. Arrays remain usable after session closure and can be dropped on any thread. No unsafe Send implementation is used.

Brackets without literal separators stay unresolved until binding. A postfix array/function target gives indexing/axes; an expression boundary or dyadic operator awaiting its right operand gives enclosure. Contents evaluate once through the ordinary value path, including functions. Power unwraps an enclosed count/predicate to collect history; Python .history() constructs the same operand.

Workspace and builds

Prefer broad dependency ranges with a required lower bound, such as >=0.24.4, <1, rather than exact pins or Cargo's minor-constrained 0.x caret ranges. Resolve API changes when they arise; do not add compatibility layers preemptively.

The canonical version lives in Cargo.toml; Python uses dynamic = ["version"]. The crate produces an rlib, native executable, and optional basedpl._core extension. python enables PyO3; extension-module also enables PyO3's extension linking mode. Default Cargo builds have no Python dependency.

Rust 1.98 or later is required by the algebraic float methods. CI tests with stable Rust. Keep fastws-generated Cargo patches and .git/fastws-cargo-key under fastws control. Preserve the pyproject source/cache keys; do not commit the workspace-generated Cargo.lock or manually replace workspace configuration. meta/ is ignored planning material, never committed.

uv builds and maturin develop --release use the incremental release profile: LTO off, 16 codegen units, this package incremental. Distributed wheels use dist: full LTO, one codegen unit, incremental off, stripped output.

CI tests the native core/CLI/JSON process before installing the extension and running Python tests. Python tests cover boundary behavior and the real installed command, not a duplicate Rust semantic suite. A dist-profile CPython 3.13 wheel has also passed these checks in a clean temporary environment outside the workspace, with the source checkout and Rust absent from its import/command paths. The existing wheel/sdist and tagged publication flow remains unchanged.

tests/test_repl.py uses a real pseudo-terminal for symbol entry, ambiguity, bracketed paste, Ctrl-C recovery and Ctrl-D's final newline; piped process tests cannot exercise these paths. Rust editor tests cover matching and quoted/comment context. The editor's Enter callback records just the accepted replacement because Rustyline cannot combine replacement and submission in one command; the adapter applies it before history/evaluation and shows the glyph in its submission message. This does not reprocess an entire source string.

Do not repeat isolated wheel installs for routine feature changes. maturin develop plus the normal tests is the development default; reserve clean-install checks for packaging-sensitive changes or release preparation, using a small relevant subset of existing tests.

Release

Development tests and artifact checks precede release approval. Once Jeremy approves a release, confirm the clean tree and Cargo version, then use ship-release with no flags. For this maturin project fastship tags/pushes the current version, leaves publication to CI, then bumps Cargo and refreshes the editable installation. There is no changelog step. First publication requires Jeremy's PyPI trusted-publisher setup. Never commit, push, tag, or publish without approval.