parser-approach.org

The parser approach

Easyconfigs are Python, but eb-stack does not run Python

An EasyBuild easyconfig (*.eb) is, syntactically, a Python source file. EasyBuild’s own framework loads it by executing it and reading the resulting module namespace. That gives EasyBuild full fidelity (conditionals, computed values, imports, templating via %(version)s-style substitution) at the cost of needing a Python interpreter and the EasyBuild framework installed, and of executing arbitrary code from every recipe it reads.

eb_parse (src/eb_parse.rs) does not run a Python interpreter. It evaluates easyconfigs as a restricted Python-like domain-specific language (DSL) with its own small recursive-descent parser: string/int/bool/~None~ literals, lists, tuples, dicts, name lookups (including local_* variables and the SYSTEM constant), + string concatenation, % string formatting, and the full EasyBuild TEMPLATE_CONSTANTS table (SOURCE_TAR_GZ, GITHUB_SOURCE, PYPI_SOURCE, and around fifty others: see src/eb_template_constants.rs). Unmodeled statements (for, def, import, and similar keywords) are skipped rather than rejected, so a recipe that uses control flow for a field eb-stack does not need still yields every field it does need. One if is read: if ARCH == '<this machine>', with the body parsed as easyconfig text. name.append(...) extends a list or tuple that is already assigned, so an architecture-specific dependencies.append is part of the closure on that machine. Any other if is skipped.

What gets resolved

resolve_easyconfig_str (and its file wrapper, resolve_easyconfig_file) evaluates the whole assignment namespace, then extracts and template-expands:

• name, version, versionsuffix
• toolchain (dict form {'name': ..., 'version': ...}, tuple form, or the bare SYSTEM constant)
• dependencies and builddependencies: 2-, 3-, or 4-element tuples (name, version, optional versionsuffix, optional per-dependency toolchain override), plus the legacy filename form ('OpenMPI-4.1.6-foss-2025b.eb') some easyconfigs still use
• exts_list: name/version pairs, string-only entries, and entries with a trailing options dict

A bare dependency version ('4.1.5') maps to an exact solver requirement (==4.1.5), EasyBuild’s own default dependency semantics, while a version already carrying ==, >=, <=, >, <, or != is used as-is.

%(key)s templates are resolved against a per-recipe template table built from name~/~version~/~versionsuffix~/~toolchain (including derived keys like namelower, version_major_minor, github_account) plus the static TEMPLATE_CONSTANTS table, applied recursively (bounded) so a constant that itself expands to another template still resolves. A recipe assignment of github_account, bitbucket_account, or versionprefix replaces the derived value. Without one, the forge accounts stay at namelower and versionprefix stays unset.

This resolved shape feeds recipe check, stack solve, and the candidate universe for package bump through ResolvedEasyconfig::to_candidate(). A bump also resolves its source recipe, materializes an EasyBuild-origin package plan, and asks Resolvo for the target dependency selection. The targeted emitter then locates the toolchain dict, version assignment, checksum entry, and dependency tuples in the original source text and rewrites only those spans. Unselected content stays intact, while the selected versions come from the profile lock rather than a separate text-rewrite decision path. See architecture.

Correctness is checked against EasyBuild’s own parser

fixtures/parser_hardcases/ contains small easyconfigs exercising the constructs above (template substitution, local_* variables, the SYSTEM toolchain, multi-element dependency tuples, exts_list) paired with golden JSON under fixtures/parser_hardcases/resolved/ produced by scripts/resolve_easyconfig_eb.py, which drives EasyBuild’s real EasyConfigParser (requires an EasyBuild install; not run by cargo test). cargo test asserts the Rust parser’s output against those checked-in goldens: a change to eb_parse that diverges from what EasyBuild itself would produce fails a test.

Unparseable files are skipped, not fatal

Real easyconfig trees are large (annual EasyBuild checkouts run to five figures of .eb files) and not every file in a checkout parses: some use constructs outside the restricted DSL (inline if~/~for, multi-statement configopts built from several += lines with embedded control flow, computed values). parse_easyconfig_tree walks the whole directory and records each unparseable file in ParseTreeResult::skipped with its error; it does not abort the walk on the first bad file. ParseTreeResult::coverage() reports the parsed fraction. Tree-consuming commands such as recipe check, stack solve, and package bump --easyconfigs surface coverage and sample skips, so “not in the candidate universe” and “does not exist” stay distinguishable to the operator reading the output.

Coverage against a real, unfiltered easybuilders/easybuild-easyconfigs checkout (tens of thousands of files, every toolchain generation from 2021a onward) stays above 90%: see reproduction fidelity and limitations for the exact number and what the remaining unparsed files look like.

Why targeted evaluation instead of the alternatives

Two more general options exist and were deliberately not taken:

• Evaluate the file with a Python interpreter, optionally driving EasyBuild’s own EasyConfig class for full parity with what EasyBuild itself would parse. This gives ground-truth fidelity, including computed and templated fields no restricted evaluator will ever handle, at the cost of pulling a Python runtime (and, for true parity, an EasyBuild install) into a Rust CLI’s dependency chain, and of executing recipe code eb-stack did not author.
• A full Parsing Expression Grammar (PEG) or hand-written grammar for the easyconfig DSL (the pest~/~chumsky~/~nom~/~lalrpop family). This stays native and dependency-free like the current approach, but is strictly more implementation effort to track EasyBuild’s own syntax evolution, for a grammar that still would not evaluate control flow or computed values any more than the current restricted evaluator does.

The restricted recursive-descent evaluator is a deliberate middle ground: real assignment/expression evaluation (not a text-scraping regex layer) for every construct the solver and the emit path need, without a Python runtime and without evaluating arbitrary code. It does not (and is not meant to) handle a field whose value depends on control flow, a function call, or an import: those recipes fail that one field’s extraction and the file is skipped, not miscounted as present with a wrong value.

Version comparison is EasyBuild-shaped, not semver

src/version.rs tokenizes a version string into an ordered run of numeric and alphabetic parts and compares position by position: numeric parts compare numerically, alphabetic parts compare lexicographically, and a trailing alphabetic part with nothing on the other side sorts as a pre-release (2025a < 2025b, 1.0rc1 < 1.0). This matches how EasyBuild toolchain generations (2025a, 2025b) and application pre-releases actually sort, which semver’s MAJOR.MINOR.PATCH model does not directly express.