Release 0.21.0 - #67
Open
phy0x1a79ed wants to merge 293 commits into
Open
Conversation
AGENTS.md was 1167 lines of metasmith architecture written when metasmith was the repo. An agent landing here read 900 lines about the solver and the GUI and still could not tell you that src/ecspr existed. It is now 113 lines holding the three things a root brief owes a reader: the monorepo's shared layout, the rules for getting a working tree, and one row per src/* module linking to docs/<module>/architecture.md. Six such documents now exist; three of those modules (ecspr, bash_relay, workflow_solver) had no prose anywhere before this. The register split is stated in the root file rather than left to be inferred, because the next restructure will otherwise sweep the exception: architecture -- what a module is and why it is shaped that way -- goes under docs/, while an authoring brief that only works if it is found without being looked for stays at the package root. src/metasmith_libraries/AGENTS.md is that case and stays put. src/fabfos/AGENTS.md was architecture and moved. Content whose evidence lives off this machine was relocated, never cut: the group_by/batch_size agreement across five files, the lineage index rules, the container-coordinates rule for codegen, the apptainer three-rung chain and the segfaulting mksquashfs on micb0 that is why the third rung exists, and the Nextflow pin with its open upstream bug. What was cut is worked examples a description covers, enumerations the tree and --help already carry, and the how-it-was-found narration around invariants that are kept. docs/ was mostly not documentation: 3,946 lines of which most were finished plan records. Four are deleted. lineage-quadrant-audit.md and harmonize-data-instance-arity.md are kept -- live code and tests cite them by path, one with line numbers -- as are ENV_PORT.md and ASPIRE_PORT.md, which back a statically-enforced engine rejection and the ASPIRE correspondence map. consolidation-followups.md is an open punch list and stays. tests/metasmith/unit: 584 passed, 3 skipped -- unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The module docstring still said `rng` "and nothing else, with the search still to come". `CAPABILITIES` is `["rng", "solve"]`, there is a `solve` subcommand, and mcts/search/refine/rectify all ship. Left as written it is worse than merely out of date. Every solve in this worktree currently falls back to the Python search -- not because the Rust search is unimplemented, but because the staged binary is a mode-444 hardlink out of the shared DVC cache and so is never executed (open task hallamlab#16). A reader chasing that would have found a docstring telling them the fallback was by design and stopped. The replacement says what decides, and names the failure mode. Found by the docs-restructure session, which correctly left source alone and noted it instead.
docs/metasmith/plans/ had no charter, so what belonged in it and when a record could be deleted were decided per-session by whoever looked. This pass nearly deleted two files that live code cites -- lineage-quadrant-audit.md from cache_decisions.py and three tests, harmonize-data-instance-arity.md from a test that cites it by line range -- which is what the rule now names: "Finished" is not a property of the document. It is a property of whether anything still points at it. A plan marked COMPLETE whose invariant a test cites is load-bearing; a plan marked in-progress that nothing references is dead. Grep before deleting; a hit from source or tests is a veto. Also names the fragile case, since it is the one that fails silently: a line-numbered citation survives every rename and reflow of its target and then means something else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten algorithm modules move from src/ecspr/ into src/ecspr/model/, leaving the package root holding only what is genuinely package-level: the version block, the CLI and its commands, the env and the build hash. This is the shape the bake lands beside in T2 -- a subpackage next to ten loose files would have read as an afterthought rather than as one of two layers. Every call site is rewritten rather than shimmed. A re-export in ecspr/__init__.py would execute on every `import ecspr.bake.*` and drag scipy, networkx and cobra into the AAM image, which is precisely the isolation the staged package has to preserve; and a lazy __getattr__ would not cover `import ecspr.build as EB`, which three research drivers use. model/__init__.py stays empty of imports for the same reason. pyproject.toml names the new package explicitly -- it lists packages rather than discovering them, so a forgotten entry ships an install that succeeds and contains nothing. Its requires-python drops to 3.11: the direction stack is pinned there and a floor above it makes the package uninstallable exactly where the bake needs it. No behaviour changes. tests/ecspr: 50 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
…mpile against
The monorepo nested fabfos one level deeper than the standalone repo this script
was written in, and its `REPO="$(dirname "$HERE")"` was never updated. It
resolved to src/fabfos, so `--types $MLIB/data_types` and both
`$MLIB/{resources,transforms}/*/` globs pointed at paths that do not exist --
and the script ran anyway, compiling the build-side library against the pruned
type copies already on disk instead of the shipped library's real definitions.
Two things fall out of correcting it, and neither is cosmetic:
* the pruned type snapshots under transforms/*/_metadata/types/ pick up the
types the standard library has gained since -- most visibly the aspire::
namespace, which had no copy here at all.
* aam_ensemble.py and direction_ensemble.py get new instance_ids. Their
committed ids were STALE, not merely different: both files had been edited
since their last successful compile, and
`multihash(blake3(bytes) || relpath)` computed by hand over the current
files reproduces the new ids exactly. A stale id in the index is the
false-cache-hit failure, so this is the bug rather than a side effect of
fixing one.
Two consecutive runs are byte-identical, which is the property the index is
worth having at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
Twenty-one flat modules under resources/buildlib/ become a python package:
eight in ecspr/bake/aam/, nine in ecspr/bake/direction/, and four at the top
(atom_pairs, encoding, metabolism, evidence) because both lanes use them.
mnx_lookups.py and the four bench_*.py stay -- they are fabfos's, not the
bake's.
Three changes land together because none of them is separately runnable.
THE MOVE. Bare-name imports become relative ones. The cross-lane case is the
one that mattered: direction/drive.py importing ..aam.shard is the same edge
that used to force a `buildlib::aam_shard.py` requirement onto two direction
transforms, with a comment on each apologising for it. Fourteen modules with a
command line grow an `argv` parameter and keep their `__main__` block, so
`python3 -m ecspr.bake.aam.worklist build ...` runs what
`python3 .../aam_worklist.py build ...` ran.
Nothing else moved. Verified by reconstructing every pre-move module from git
and diffing its whole argparse tree -- verbs, flags, dests, defaults, types,
choices, required, nargs -- against the same module at its new path:
14/14 identical.
ONE STAGED INPUT. build.sh vendors src/ecspr into resources/buildlib/ecspr/
(generated, gitignored, excluding __pycache__/*.pyc/*.egg-info/build_hash.txt)
immediately before compiling the metadata, so the index always describes the
copy that was staged. Twenty-one type declarations collapse to one directory
entry; thirteen transforms drop between two and ten requirements each and gain
`bakelib`. Their `libdir` line is unchanged -- the staged directory is named
`ecspr`, so `PYTHONPATH={libdir}` keeps working and `-m` resolves.
A HASH THAT MEANS SOMETHING. Leaf identity was content-addressed only for
files; a directory fell through to `uuid4() + time_ns()`, so an unchanged tree
minted a new id on every recompile and invalidated every cached run that had
read it -- for a multi-day bake, days of Sockeye for a tree nobody touched.
`tree_multihash_key` gives it the file arm's promise, memoized per file on
(path, size, mtime) so an unchanged tree re-stages stat-only. Re-vendoring and
recompiling twice mints the same id; the same guard is what kofam's 27,757-file
profile directory has been losing reuse to.
Two call sites outside the transform system move with it: the reference gate,
which reached refs_encoding and dir_canon by sys.path insertion, and the GEM
transform's embedded driver. Both now import from the installed package, which
is how they already reached ecspr.model.
Verified: the whole R6 plan resolves -- 18 steps, 17 distinct transforms, every
expected transform present, MetaCyc still the only given. tests/metasmith cache
+ unit: 651 passed, 3 skipped. tests/ecspr: 50 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
There were no tests over any of this, and that was a cost of the shape rather
than an oversight: code reached only through a staged path and a subprocess is
code nothing imports. The package makes it assertable, so this asserts it --
100 checks, 5 seconds, over the parts the AAM change is about to move.
The budget is held by running FEWER checks, not faster ones. The tier with the
most weight uses the deployed metabolism_bake trio, and it uses SULFUR: same
coverage of the packing, the vocabulary join and the ratio flip, 24,198 pair
rows instead of ~1.5M. Carbon is the version of this test that would blow the
budget, and dropping to sulfur is the first cut to reach for rather than the
last.
What it pins, chosen as the things a second implementation would silently
diverge on: the verdict cascade and its ORDER (atoms counted only under the
character cap, non_molecule outranking every other blocker, no_transfer
outranking blocked); the four additive gates, refusing rather than warning; the
fusion arithmetic including the NaN-confidence division guard; the direction
combiner's independence claims and its no-evidence default; count_element's
None-is-not-zero; forced_pairs' doubly-stochastic completion. Every module
imports at its new path, every CLI carries exactly its frozen verbs and flags,
and a subprocess per lane proves ecspr.bake drags in no ecspr.model.
Two findings recorded rather than fixed, because this phase is required to move
zero numbers:
* `additive_gates` check 3 -- "no element loses reactions" -- cannot fire. It
compares `below` against `concat([below, added])`, which is a superset by
construction, and `stack` only ever passes the accumulated table. Pinned as
inert, so a fix shows up here as a failure.
* running from the vendored tree writes __pycache__ into it, which would move
its instance_id. Safe in the sanctioned flow -- build.sh removes and
re-copies before compiling the index -- and the suite's own subprocesses set
PYTHONDONTWRITEBYTECODE so the gate does not dirty the artifact it checks.
Two envs, because no shipped env carries rdkit and scipy together and that seam
is exactly what one of these tests defends. `ecspr`: 109 passed, 9 skipped.
`rdkit-scratch`: 98 passed, 2 skipped (the two thermo members live in their own
images). A skip here is a real gap, and the conftest says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
`docs/fabfos/architecture.md` said ECSPr was the only method that gets a package, because it was the only transform whose protocol is an algorithm rather than a dispatch into somebody else's tool. That sentence is what this campaign overturned; it is now the general rule -- custom algorithms live in a src module, work that is only a call to an external tool stops at a transform -- with the reason a flat file stops being enough stated beside it. The driver convention goes in the same document, and in this scope's context.md as the convenience copy (context.md is regenerated by `scope heal`, so the architecture doc is the durable one): drivers stay in research/fabfos/examples/ and tests/fabfos/, because moving one into src/fabfos/ is a form of publishing and most of these are genuine research runs. The question before promoting one is not "is it useful?" but "does a study depend on it?". A journal TODO records the follow-on -- publish the benchmark driver(s) a study actually cites, in practice likely just eydallin -- and, while writing it, `build_clone_gpr_denovo.py`'s sys.path insert turned out to point at <repo>/build_references/..., missing the src/fabfos/ segment the monorepo added. The path does not exist, so its `import bench_universe` has not resolved since the migration. It is one line and it is fixed here rather than folded silently into the note; it reaches a module that stays flat, so it was never part of this migration. Also: AGENTS.md now names three build commands rather than two, and says why the third's internal order matters -- build.sh vendors src/ecspr before compiling, so the index it writes describes the tree that was staged. The _deferred/ EZpred README loses half of its shape objection: a directory-typed buildlib:: entry with a tree digest for an identity is exactly the resolution it named as a candidate. Where the patched source should live is still open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
`ATOM_LIMIT` is applied to `count_atoms(rxn_smiles)`, and `rxn_smiles` is a stoichiometric EXPANSION -- a coefficient of 16 writes the metabolite sixteen times. So the cap has been measuring how many times a molecule appears rather than how much distinct chemistry a mapper must attend to, and nitrogenase, which hydrolyses 16 ATP, was refused at 1,236 atoms while the acetylene-reduction proxy for exactly that chemistry sailed through. `worklist.collapse` writes each distinct molecule once PER SIDE -- water on the left and water on the right are two occurrences the equation needs; the same molecule twice on one side is the copy the cap should never have counted. It splits as TEXT and parses nothing, which is not an optimisation: MNXR144749's expanded string is 80.7 MB and RDKit does not return from parsing it. Components are the `.` at bracket-and-paren depth zero, which is SMILES's own separator; a naive split would cut molecules in half and dedupe fragments of them. COVERAGE MAY ONLY GO UP, and it is structural rather than checked. A reaction the expanded measure admits keeps its expanded string byte for byte -- same universe, same map, same pairs, same weights. Only a reaction that would otherwise have been REFUSED is mapped collapsed. Measured over all 83,795 reactions: 57,061 mappable before, 57,515 after, 454 added, 0 lost. THE THRESHOLD IS RE-DERIVED, and lands on 600 again. 600's warrant was a yield curve over the expanded count, and a curve stops describing a measure the moment you change what is counted. Binning every buildable reaction by COLLAPSED count and joining to the reactions the deployed bake banked: 82-94% up to 600, then 26.9% at 600-650, 16.7% at 650-700, 10.0% at 700-800. The knee is as sharp as the expanded one and in the same place. Arriving at the same number by doing the work is the finding; carrying the constant across would have been borrowing a justification. THE NITROGEN CASE, by name rather than as a count. Five reactions carrying N2 move from refused to mappable: MNXR175605 and MNXR109381 (EC 1.18.6.1, nitrogenase, 16 ATP, 1,236 expanded atoms -> 98 collapsed), MNXR192176 (EC 1.19.6.1, flavodoxin nitrogenase, 1,292 -> 136), MNXR163643 and MNXR184138 (EC 1.18.6.2, the 40-ATP variant, previously refused as `too_long` rather than oversize). Three places had to move together or they would disagree about what oversize means. `aam.curation.complete` re-applies the same two cuts to the reactions the rescue lanes CREATE, and now gets the same second chance. The extractor's `stripped` guard compares template count to participant count, and a collapsed reaction has fewer templates than participants -- so strict mode now accepts EITHER the expanded count or the per-side distinct-structure count, computed by the caller from the same metabolite-to-SMILES map the string was built from. A genuinely stripped reaction still matches neither and is still refused, and no mode flag is threaded from the worklist to the extractor. Both counts are recorded on every row (`atoms`/`chars` stay the expanded ones, `atoms_collapsed`/`chars_collapsed` beside them, `collapsed` saying which string `rxn_smiles` holds), so the pre-collapse curve stays recomputable from a post-collapse table and this is a diff rather than a claim. A FINDING THAT IS NOT MINE TO FIX HERE: the deployed bake holds 14 reactions the 600-atom expanded cut refuses, so the chain regressed against its own predecessor before this change. The collapse repays 3 of them; the other 11 carry no stoichiometric repeats at all -- their collapsed count equals their expanded one, between 603 and 961 -- so they need ATOM_LIMIT itself raised, which is a separate decision with its own cost. `results/deployed_bake_debt.tsv` names all 14. research/fabfos/benchmarks/aam_collapse/ reproduces every number above. tests/ecspr/bake: 105 passed, 2 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
Two populations still bank nothing after the collapse, and they fail for
different reasons. The small one is genuinely large distinct chemistry: 78
reactions the size cut still refuses. The large one is not a chemistry verdict
at all -- RXNMapper's transformer accepts at most 512 tokens, and the reactions
it returns nothing for are the long ones, median SMILES 983 against the
universe's 268. That is a context-window limit eating a BIASED sample: the
reactions with the most and largest cofactors.
The bias is what makes the fix sound. FOR A SINGLE ELEMENT most of a long
reaction is irrelevant -- `3 NADPH + 3 NADP+` blows the limit while contributing
no sulfur at all -- so the submission is an ELEMENT REDUCTION. Keep only the
participants whose formula carries X, require X to BALANCE across the kept set,
map the small reaction that results, and read only X's pairs back out. This is
not the `stripped` case wearing a different name: an atom of X cannot come from
a participant carrying no X, so dropping the X-free participants removes no
possible source and no possible destination for an X atom. Every other element's
map in a reduced submission is discarded UNREAD, because for those the reduction
really is a strip.
The exact arm goes first. `forced_pairs` emits what conservation leaves no
choice about -- a unique bijection at full weight, or the doubly-stochastic
completion when n > 1 -- and needs no mapper, so anything it settles never
becomes a submission.
MEASURED, before anything is spent. Over the 78 size-refused reactions: 23
(reaction, element) settled exactly, 3 reduced submissions. Over the closest
local proxy for the 512-token population (mappable, SMILES >= 900 chars, 1,254
reactions): 332 forced (reaction, element) plus 1,617 reduced submissions =
1,949 partial combinations, and 12,567 pair rows from the forced arm alone with
no mapper time at all. The real target set cannot be computed locally -- it is
"admitted and still ended with nothing", a fact about a run -- so the table
sweeps the length threshold rather than picking one.
`results/partial_lane_yield.tsv`.
LABELLED EVERYWHERE IT IS READ, which is the half the goal names explicitly and
the stop line this lane is built around.
* the layer is L4, laid down LAST, and `stack` restricts each layer to what
nothing below it claimed -- so a partial pair contributes NOTHING where a
real member already reached, not a lower weight and not a second row. That
is machinery rather than a promise, and it is pinned by a test.
* every row carries `source=partial` with `partial_forced` or
`partial_reduced` as its method.
* `close` gains `banked_partial` and `partial_declined` in its closed outcome
set. A reaction whose only pairs are partial is neither `banked` (nothing
mapped it whole) nor `mapped_nothing` (it did produce pairs), and folding it
into either overstates coverage or hides it.
The lane is four transforms: `aam_partial` builds the universe, and the same
three members map it. Its target set is computed by SUBTRACTING every member's
pairs table from what the worklist and the rescue admitted -- so an absent
member is refused rather than treated as covering nothing, which would make the
gap the whole universe at full mapper cost and report it as a finding.
The extractor gains `--partial`: it reads the reduced participant lists from the
universe rather than re-deriving them from `reac_prop`, which would hand every
submission the full equation and make the strict guard call all of them
`stripped`.
Verified: the whole R6 plan resolves at 22 steps, 21 distinct transforms, with
the partial lane sequenced after both mapper passes and MetaCyc still the only
given. tests/ecspr/bake: 118 passed, 2 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
The multiplicity question, quantified rather than deferred. `n_atoms` counts atom-index triples, so a collapsed reaction emits one triple per pair where an expanded one emits sixteen. No EXISTING row moves -- collapse only ever touches reactions that were refused -- but the new rows sit on a different footing from a comparable expanded reaction's, and that is a real question about edge weights. Sized: the 454 recovered reactions carry a median stoichiometric multiplicity of 4.7x (IQR 3.4-6.6, max 89x), and at the deployed bake's 37 rows per reaction they are an estimated 17,021 rows, 0.7% of the table. Whether they should be re-expanded by coefficient is the user's call and goes in the report. And the data pins. Every chunk the bake reads resolves and is materialised here: metanetx 1.5G, metacyc 1.6G over 23,557 files, equilibrator 1.4G, chebi, modelseed, and the deployed metabolism_bake trio at 7.5M. `dvc status` reports one unrelated staleness -- src/metasmith/scratch/gui-main, absent from the cache since before this scope -- and nothing under data/fabfos. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
Two forms, because they answer to two readers. The numbers are a committed table
under research/fabfos/benchmarks/aam_collapse/results/ -- reproducible, diffable,
and what the next session re-runs to check nothing drifted. The report is the
half a person reads.
It closes on the two things that are the user's rather than mine, each stated as
a recommendation with its numbers rather than as an open question with the work
left undone:
* the collapsed reactions' pair weights. `n_atoms` counts atom-index triples,
so a collapsed reaction emits one where an expanded one emits sixteen. Median
multiplicity 4.7x, max 89x, an estimated 0.7% of the table. Recommendation:
leave them as mapped -- re-expanding by coefficient makes an edge weight
scale with how many ATP a reaction happens to hydrolyse, which is the same
conflation of stoichiometry with chemistry that put the cap in the wrong
place to begin with.
* whether the rebake is warranted. Recommendation: yes, once, with the partial
lane in the same pass -- it is one extra stage rather than a second campaign,
and its ledger words make a disappointing result legible rather than
invisible.
It also says plainly what it cannot show. The goal's success indicator is that
the Nostoc GPR builds a network whose AAM permits N fixation, and no local
artifact reaches that -- it needs the rebake and then the GPR. What is claimed is
the rung below: the reactions carrying the chemistry are admitted, and their
distinct chemistry is small.
Published: https://claude.ai/code/artifact/ce389e22-3c60-4831-b432-9ebb127a9349
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
Five items found during the migration and deliberately deferred, plus the documentation half of two of them. None was migration work; all five are contained enough to land together. **The solver was never actually running.** In a source checkout the staged binary arrives as a mode-444 hardlink out of the shared DVC cache, so it could not be executed -- and the fallback to the python search said so once, in a warning nobody read, while every solve in the tree paid ~15x. It cannot be fixed in place: chmod would mutate a cache object every worktree shares, and dvc unprotect would dirty the pin over a permission bit that is not content. So GetEngine now resolves through a runnable copy staged under the temp dir, keyed by content digest so a rebuilt engine is never shadowed, published by os.replace so a concurrent reader never sees a half-written file, and skipped entirely when the bit is already set. probe_engine also stops reporting "not executable", "missing" and "version mismatch" as one undifferentiated warning. Measured: dev/libraries.sh -b goes from >16min (killed, incomplete) to 3m14s with no fallback warning, and nothing in the engine directory changes mode. **The differential gate was blind for the same reason.** Its fixtures probed packaged_engine_path() directly rather than resolving as the planner does, so they errored in half a second saying the binary "failed its handshake" -- about a binary that was fine. They now go through GetEngine. This is what makes the two findings in the followups file visible; neither is fixed here, and both are recorded with the evidence to act on. **A directory-valued library item re-minted its id on every compile.** Only is_file() was content-addressable, so the standard library's one directory item fell to the random branch, defeating the bundle guard's staleness arm and denying a cache hit to both transforms that consume it. directory_multihash_key digests the tree in sorted order with type tags and length framing, symlinks by target rather than followed. Cache keys for those two transforms change once, which is correct and is the point. **TargetBuilder.Add accepted a type name where a handle belongs.** That builds a spec equal to nothing already added, so the duplicate check silently passes and the caller believes it asserted something it did not -- which is exactly what the row-37 test had been doing for its whole life. Every real caller in the tree already passes handles, so the check breaks only the one line that was wrong. The test now constructs a genuine duplicate and a second test covers the guard. **The compatibility fixture starved two chains of a parameter.** clustering:: min_identity has no producer and is not meant to have one; it is caller-supplied, so it arrives as a resource library. 36 passed / 0 failed. **The gui-main pin was dead from both ends.** dev/metasmith.sh --gui recreates that directory with mkdir -p and nothing ever reads it, so it never qualified for a pin -- and its objects had already left the shared cache, which is how the checkout failed in all three worktrees. Dropped, with the rule it violated written up in docs/metasmith/architecture.md: a build artifact may be pinned when rebuilding is expensive and the pin is the distribution path, but the consumer must never require the pinned copy to be usable in place. Verified: transform compatibility 36/0; libraries 60 passed (was 58 with the duplicate-target test failing); fabfos 38 passed; metasmith -m fast 1540 passed. The remaining reds are recorded in the followups file and owned elsewhere.
…s censored
Every mapper method in the deployed table stops dead at 600 atoms because the same
cut sat upstream of all three lanes -- localmapper_only max 600, indigo_only 598,
rxnmapper_only 563. The only thing banked above it is `curated`, which never sees a
mapper. So the 12.8% banking rate above the cap is MetaCyc's coverage, not a
mapper's failure: the knee justifying the constant is a wall the constant built.
Tested from the other side, the cap belongs to the NEURAL members. RXNMapper takes
512 tokens and LocalMapper is the lane that was OOM-killed twice; Indigo is a
compiled substructure search with a recorded timeout that maps a 1,666-atom reaction
in under two seconds. `oversize` therefore means "the neural members will not see
this" rather than "nothing will", and which verdicts a member admits is stated once
in `worklist.{NEURAL,INDIGO}_ADMITS` so two readers in two modules cannot drift.
An `oversize` row now carries its collapsed string too, so `rxn_smiles` means one
thing on every row: the string a member would be given.
Its own sweep records a 36% hang rate on that tail, which is why the routing is not
safe until Indigo runs behind a killable subprocess -- built with the single mapper
pass, which needs the containment regardless.
research/fabfos/benchmarks/aam_cap/ is the sweep; aam_precedent.md is where the
remaining gap lives -- 78.8% of reactions, 80.9% of element slots, and the R-group
rule named as the largest unbuilt lever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
L0 -- reactions, metabolites, atom_ranks, xrefs, synonyms -- existed in no worktree and under no .dvc pin, which is why it was being rediscovered rather than read. (`mnxr_lookup.parquet` is R5, the annotation lanes' id->MNXR bridge, a different artifact.) Rebuilt from the four pinned sources; all five row counts reconcile exactly against what REFERENCES.md records: 83,795 / 1,495,668 / 130,160 / 3,923,902 / 4,007,739. `mnx_lookups` was the one compile/ transform still carrying `labels=["local"]`. Every sibling carries an explicit note that the label belongs to acquire/, where a download needs the login node's route, and that copying it elsewhere pins a compute step to a pool slurm.nf declares as 8 cores / 8 GB with errorStrategy='ignore' -- so a step asking for 32 GB is dropped and the workflow goes green with all five tables absent. This one reads four staged TSVs and allocates 32 GB doing it. The parts driver gains a `lookups` part that runs first and alone, because every AAM lane reads these tables and a rebuild that silently differed would surface six lanes later. Six minutes on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
…states it `C70H131N3O9PS*2` says "and two residues of unspecified composition", so `count_element` returns None and every gate downstream refuses -- while the SMILES in the same MetaNetX record says C70 N3 P1 S1 in explicit atoms. Refusing the whole species turned a known count into an unknown one and took the reaction's balance test with it. That is the largest unbuilt lever this campaign's own precedent note names. `aam_recount` reads the structure where there is one and publishes the answer as `lookup::element_counts`: one row per (metabolite, element), the route recorded, the residue count carried alongside. The routes are ordered and the order is the point -- structure first, because `atom_ranks` is keyed on the structure and a count taken from anywhere else can disagree with the number of ranks a pair row indexes into. The step asserts that agreement itself, before anything joins it. A recount is not a balance claim. `n_residue` travels with every row so a balance test must still make the residues cancel before it believes the difference, and an unknown count stays a real NULL through the parquet -- not zero, which is a claim, and not NaN, which sums. ONE COUNTER IN THE TREE. Three byte-identical copies of "atoms of X in a MetaNetX formula" existed -- in the extractor, in curation, and in mnx_lookups -- all of which had to agree and none of which was tested against the others. The tested one survives; the other two are now aliases and imports. `mnx_lookups`'s own argument is that a derivation five consumers make for themselves drifts, so carrying a private copy of exactly such a derivation was that argument failing on its own terms. Measured while writing the polymer guard: MNXref 4.5 holds ZERO `(...)n` formulas and 131 ending in `n`, all of them Mn / Zn / Sn / In / Rn. A guard on a trailing `n` would refuse the manganese and zinc cofactors and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
…plan Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
Two verbs over one alias scan, two transforms, two evidence standards.
`blockers` accepts only an ELEMENT-NEUTRAL twin -- zero C/N/S/P, all four known
from the recount. That single coded predicate is A1 accepted and A2 held: the
acyl-carrier twin carries C14 N3 S1 P1 through the exact thioester an acyl
stand-in must never fabricate, and it is refused on its own numbers with nobody
maintaining a list. `nametwin` accepts real atoms and therefore demands real
evidence -- a shared source accession, or a balance that closes only after the
substitution with the residue slots cancelling.
The name guard runs before the chemistry, because MNXM900 'hexadecenoate'
against its `CO2*` twin BALANCES and only the name knows a C16 fatty acid was
about to be drawn as a formate. Ties break on the accession, numerically; a
set-iteration tie-break is how a lane's yield wanders on identical inputs.
Neither bypasses the arbiter. Both emit a crosswalk in the rescue's own shape
and face `admit`, the body-cancel gate and the per-element balance like every
other proposer. They bracket LANE_PRIORITY: `nametwin` first because MNXref's
own record on hard evidence outranks any name-stem inference, `blockers`
directly above `acceptor` because it makes that lane's claim from a record
rather than from six hand-written spellings -- so the overlap between them is
measurable, and `summary.tsv` reports it as a first-class number.
The A4 lever lands in the arbiter here: `Refs.counts_of` reads
`lookup::element_counts` before the formula columns, and `concrete_balance`
does the same -- with `residue_of` beside it, because an exact count of the
EXPLICIT atoms is not a conservation claim until the unspecified slots cancel.
An unknown slot count is a refusal, never a zero.
Two bugs the canaries caught, both silent:
* `Series.str.split("||")` reads its separator as a REGEX, and `||` matches
between every pair of characters. `Acceptor` came back as {a,c,e,o,p,r,t}.
* a missing name/formula/InChIKey arrives from parquet as NaN, and NaN is
TRUTHY -- so the skeleton comparison ran on a block that was not there and
`nan != nan` refused both canaries at once.
Graph plans: 25 steps, the two lanes feeding the rescue, MetaCyc still the only
given. 165 passed, 2 skipped.
Three arms, two grains, and the split between them is the deliverable. The CONJUGATE arm is exact and is banked. A participant standing on both sides with EQUAL MULTIPLICITY contributes the same unknown amount to each, so it cancels on IDENTITY -- no formula consulted, nothing inferred -- and what is left may leave conservation exactly one possibility. Unequal coefficients do not cancel: one copy of an unknown amount left over is the same unfaithfulness that makes a naive collapse wrong. The SINGLE-UNKNOWN arm and the CARRIER CLASS are species-grain and are not. Conservation fixes a number about a MOLECULE; it fixes no atom correspondence, and a structureless species has no canonical rank to hang one on. They go in their own table, are reported, and are never banked. The measured 99.7% agreement between this algebra and a mapper argues for the first half and not at all for the second, and `claims.tsv` beside `forced_pairs.parquet` is how that stays true for a reader. Stricter than `atom_pairs.forced_pairs` in two places, deliberately. `atom_ranks` holds ONE rank list per (metabolite, element), so a carrier written twice offers two indistinguishable copies and a bijection over them would be a choice; and a count that disagrees with the length of that list is refused rather than emitted, because a pair row is an index into it. The recount is what makes any of it reach -- `count_element` refuses a `*` and these targets are full of residue species. The price rides along: an exact count of the EXPLICIT atoms is not a conservation claim until the unspecified slots cancel across the reaction, and an unknown slot count refuses. Targets are stated by subtracting from `INDIGO_ADMITS` and the rescue's completions rather than by listing verdicts, so a new verdict is handled without being added here. `interm::aam_algebra` has no consumer until T6's `aam_stack`, so it is deliberately absent from the DAG renderer's EXPECTED for now. 179 passed, 2 skipped. Graph unchanged at 25 steps.
The partial lane's own docstring argued this could not be done: its targets are the
reactions that ended with nothing, and there is no way to know which those are without
having run the members. That is correct about KNOWING and was the wrong conclusion,
because the layer stack makes the two errors cost different things. Layers are additive
and ordered and their gates refuse rather than warn, so a reduced submission built for a
reaction that maps fine is never claimed by anything. Over-offer and you pay mapper time;
under-offer and you lose exactly the coverage the lane exists to add.
So `aam_forecast` predicts silence per (reaction, element) from NAMED mechanisms -- two
size caps that are our own constants, RXNMapper's context window as a property of the
string, and the previous bake's own records of what hung, timed out and came back empty.
It may only ever add submissions. `aam_universe` then concatenates the three submission
classes into one table and the three members read it ONCE: nine mapper lanes become three,
and the six `*_rescue` / `*_partial` transforms are deleted with their six types.
512 characters is measured, not converted from the token limit. Against the prior run's
44,547 recorded outcomes it catches 97.2% of RXNMapper's silences for 13.2% of the
universe, and the population that maps has p90 = 497. The two halves catch different
things and that is the argument for having both: the string finds 96.5% of Indigo's
timeouts and only 36.9% of its errors, because an error is a property of the chemistry and
two thirds of those reactions are short. 22.6% of the universe has no prior record at all
and the summary reports that denominator, so a rate over the recorded half cannot read as
a rate over the whole.
The keys cannot collide, and it is a fact about the classes rather than a convention: the
rescue completes exactly what the worklist refused, so `whole` and `completed` are disjoint
on mnxr, and `reduced` keys on MNXR#X. Uniqueness is asserted anyway -- a member reads this
into a dict, so a duplicate does not raise, it drops one of two different molecules.
Three things one universe forced that three passes had hidden:
* Every pair row carries its SUBMISSION CLASS. A reaction submitted whole and again as a
carbon reduction produces (mnxr, C) rows either way, so the layer a row belongs to is
not recoverable afterwards. `layers.fuse --submission-class` is what a layer is now,
and it is a better key than "which pass wrote this file" because it stops layer
membership depending on scheduling.
* The gap is computed at (reaction, element) grain. Per pass the conflation could not
bite; in one pass a reaction whose carbon mapped would mark its own nitrogen reduction
as covered and LocalMapper would skip the submission the forecast built for it.
* Indigo runs in a KILLABLE CHILD PROCESS. `automap` stalls below the Python layer where
no signal reaches it, and a stalled member in a single pass is the whole ensemble's
third vote rather than one pass of three. A hang now costs a budget and a fork, which
is what lets INDIGO_ADMITS keep the oversized tail.
The cache is keyed on the submission string, not the reaction id: one MNXR now has up to
four possible strings and serving the wrong one is not a stale row, it is a map of a
different molecule under the right name. And sidecars re-partition instead of refusing --
crc32(mnxr) % n is recomputable, so the partition of a union is recomputable whatever
wrote it. Refusing across a re-shard turned a resume into a full re-map of a lane that had
been killed two thirds of the way through.
Graph plans at 22 steps, three stages in order, MetaCyc still the only upstream given.
Gate 205 passed / 2 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
`aam_ensemble` fused, stacked, closed the ledger and minted the bake in one protocol, so no artifact existed between "the layers agree" and "this is the reference" -- and a correction has to run over exactly that. Split at that seam: `aam_stack` -> `interm::aam_stack` -> `aam_redox` -> `interm::aam_pairs` -> `aam_reference`. Two new types, one deleted transform. The repair, and it is a repair rather than a filter. NADH and NAD(+) differ by one hydrogen and H is not in the element vocabulary, so an MCS mapper cannot tell the two 21-carbon skeletons apart and routes a substrate's carbon into the cofactor. Every C/N/P arm running between a NAD(P)/FAD/FMN couple and a substrate is refused -- and the source atom's surviving arms are rescaled back to the total it started with, so the refusal CONCENTRATES the claim instead of deleting it. Where nothing survives the couple is removed and conservation is asked about the remainder, which is the surgery `reduce_for_element` already performs. Scope is per family and per reaction: a family counts only when it appears oxidised on one side and reduced on the other, so NAD as a genuine substrate is untouched. Same-family-or-nothing also catches transhydrogenase, which a cofactor-vs-substrate reading misses. C/N/P only; sulfur is untouched because these cofactors carry none, and the run exits non-zero if an S row moves. The cofactor set is resolved, not hardcoded: an exact name map cross-checked against each family's C/N/P signature, shipped as a table beside the refusals -- including the records that carry a signature under a name the map does not know, which is the check that says the map went stale rather than that chemistry did. Two carries discharged with it. `worklist close` reads what the FORECAST offered rather than what the partial lane managed to build, so a declined reduction is `partial_declined` and not `mapped_nothing`; and a reaction the repair empties carries `redox_emptied`, the one new outcome. The partial lane now merges the rescue's structures (deriving their formulas from the SMILES, placeholders excluded) so it can reach a rescue-completed reaction at all -- reported as its own number so T9 does not attribute it to the forecast. Gate 225 passed / 2 skipped. Graph plans at 24 steps, three stages in order, MetaCyc still the only upstream given.
The parts are the graph's three stages now -- prepare ends at the one submission table, map ends at the three member tables, assemble ends at the trio -- so a failed assembly re-maps nothing and a method change in the preparation costs an hour rather than a day. Both drivers declare the same seam PATHS, because they write into one data/temp/_seams/ namespace and a part run on the cluster only lines up with its successor run off it if the two files agree where a seam lives. Six seams out of prepare, not one. The assembly reads the worklist, the forecast, the rescue, the partial lane and the algebra directly, so each has to survive the part rather than be recomputed by whoever needs it next. Four of them are DIRECTORIES, which retrieval could not copy: every seam used to be a single parquet and copy2 on a directory raises. THE REUSE AUDIT IS THE POINT. A plan proves reuse at transform grain -- an import satisfied is a producer absent, and check_plan asserts it. Nothing in a plan says anything about reuse at REACTION grain, which is where the twelve hours are. So the driver decomposes each member's work before a job is placed: universe by submission class, what the staged cache holds as finished, what its sidecars hold as attempted, the todo those leave, and how much of that todo is ground the deployed bake has never covered. It refuses when a cache that holds rows leaves a todo within a few percent of the whole universe -- that is a cache that did not take, and the only thing distinguishing it from an honest first run is that somebody said there was one. It reads the cache off the REMOTE with a stdlib digest script, keyed on (id, hash of the submission string): a path that resolves on this workstation and not on the cluster is exactly the failure it exists to catch, and one reaction now has up to four submissions, so a row keyed on the id alone would serve whichever ran last. Cache-out is the evidence, because cache-in is a staged input a lane cannot write to -- so a finished map part folds its merged tables and sidecars back into data/temp/aam_cache, which is what the next run stages. Two lane lists that had gone stale with the graph: the whole-graph driver's EXPECTED and EVIDENCE_TOOLS still named the six deleted mapper lanes and one `ensemble` where there are now metacyc/stack/redox/reference, and neither it nor the split driver had ever been told about direction_bake -- both would have refused a plan that was correct. REFERENCES.md R6 says three stages and six layers, and says what the redox repair does and why it is a repair. Gate 225 passed / 2 skipped. All nine parts plan to exactly their own lanes; the whole graph plans at 19 steps and the renderer at 24 with no unexpected lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
`buildlib::ecspr`'s instance_id is the tree digest and the committed one was a generation behind -- a late edit in T6 landed after that task's build.sh run. Two consecutive builds reproduce the new id, and the vendored tree is byte-identical to src/ecspr, so this is the index catching up rather than anything moving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
…arry it The members run wrote both tool directories into results/ and reported neither. A run's index records the products that are some target's lineage; evidence has no consumer and no target asks for it, so the directories are published and nothing in _metadata/index.yml mentions them -- and the manifest was the only route retrieval had. The `lookups` part declares no evidence, so this is the first part that could have noticed. The directory NAME is the attribution anyway. That is why each lane copies its evidence ROOT rather than the directory under it, so walking results/ for *_evidence-tool_output/<artifact>/<tool>/ recovers a fact the manifest never held rather than guessing at a lost one. The manifest pass stays first. Re-routed the members run with --retrieve-only, which is what it is for: dgbyg 24 files, equilibrator 4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
`complete` was killed at its two-hour wall twice, at 99% CPU with nothing written since minute three. It is not slow work: the same loop was measured at 33 seconds over the same 24,098 blocked reactions. It counted atoms unconditionally and only then consulted the character cap, so a stoichiometric expansion went to RDKit -- and MNXR144749's is 80.7 MB, which RDKit does not return from. The bound that excludes those reactions is a `len()`, and it was sitting behind the unbounded call it exists to prevent. `worklist.adjudicate` has always ordered these two the right way round, and its own comment says why. `complete` re-applies the same two cuts because it measures a string it has just BUILT rather than one it inherits, and that is where the copy drifted. The two now agree, which is the actual invariant: a completed reaction and a whole one are measured by the same functions against the same constants, in the same order. The test asserts the ORDER by making count_atoms fail on an over-length string, not by timing it -- a timing assertion passes on a fast machine with the order wrong, and this failure is unbounded rather than slow. It fails against the old code with the exact string length it should never have been handed. Two things so the next one is legible rather than silent: the completion loop prints its position and rate every 2,000 reactions, and the lane declares four hours instead of two. The margin is not a measurement -- it is what turns a repeat into something a reader can act on. Gate 227 passed / 2 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
…e no truth value `aam_forecast` died in the queue on its first row. `list(r.substrates or [])` reads as an ordinary null-guard; `substrates` is a list column, parquet hands it back as a numpy array, and `array or []` asks for a truth value numpy refuses to give for anything with more than one element. Every reaction with two substrates -- which is to say the first one -- raises. Twenty forecast tests passed, because a unit test naturally writes python lists and `list([...] or [])` is fine. What was untested is the seam: the CLI reads its participants from lookup::reactions, and nothing in the suite had ever round-tripped that column through a file. So the read is a named function now and the test writes a real parquet -- including the null case, which arrives as a float NaN rather than None and needs the TypeError branch rather than the `is None` one. `worklist.adjudicate` reads the same three columns and has always spelled the guard `if x is not None`. Same fix, same reason, second site. Gate 228 passed / 2 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
One rule, two symptoms. A run's _metadata/index.yml carries every type with an `ext:` and no type without one -- so the evidence directories AND the four seams that ship a table beside the refusals that make it readable are written to results/ and named nowhere. `members` reported neither tool directory; `prepare` reported the rescue, the algebra and the partial lane as absent while all three sat on disk, which would have resurrected nine lanes in the next part's plan. The published layout IS the attribution -- <order>_<namespace>-<type>/<artifact>/ -- which is also why each lane copies its evidence ROOT rather than the directory under it. So the walk recovers a fact the index never held. Two artifacts of one type is a second producer, which the planner drops silently, so that case names both rather than picking one. Re-routed the prepare run with --retrieve-only: seven products, nine evidence directories, nothing left behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
rxnmapper and indigo read the same universe, the same cache and the same image, so metasmith gives their evidence::tool_output products the same artifact id -- and the engine publishes by artifact id. In the map run both wrote results/2_evidence-tool_output/<one id>/ and only indigo/ survived. prepare's nine lanes were safe by accident: they read different inputs. Nothing is lost. Each step's evidence root is intact in its own task work directory -- which is where the transform's success check read it, so the run was green and correct and only the copy to results/ collapsed. So retrieval reaches past the collision rather than re-running a member: any declared tool still missing after the manifest and the staging walk is recovered from <run>/nxf_work/*/*/<artifact>/<tool>/. This is not cosmetic, because cache-out IS the evidence. Without it promote_cache folds indigo's cache home and leaves the neural members with none, and the next run's audit reads two thirds of the universe as todo -- firing the refusal on a cache that was never staged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
LocalMapper's per-reaction timeout was signal.alarm plus a _Timeout raised in the handler. localmapper.get_atom_map catches broadly, so the exception was absorbed and the one-shot alarm was spent without ever having bound anything. Shard 3 of this run spent 86 minutes on one reaction -- a nitrogen reduction of a 1,201-character chlorophyll -- against a declared 240 s budget, at 99% CPU, with the timeout log EMPTY and n_timeout at zero. The lane reported no timeouts while exceeding the budget twenty-one times over, which is worse than a slow lane: a bound nobody can see failing is not a bound. It was never blocked below Python. A second SIGALRM sent by hand freed it in seconds -- the signal always arrived and the library simply ate it. So the parent owns the budget, exactly as indigo_member already had to learn it. The two lanes now share that Mapper: one submission to a forked child, and silence past the budget answered with SIGKILL, which no except can swallow. The model loads in the child rather than the parent, so six shards on one node still hold one copy each. The alarm stays inside the child as the cheap fast path, and timeout vs killed are counted apart because their ratio measures how often the swallow happens. The timeout log now names which of the two it was. An empty log is what let 86 minutes read as a normal run. Gate 230 passed / 2 skipped. Vendored and the instance_id reproduces across two consecutive builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
V1 clears the floor at both grains -- 67,008 reactions (80.0% against the deployed 78.8%) and 171,123 (mnxr, element) keys (82.8% against 80.9%) -- and the ledger closes over exactly 83,795 reactions, which the deployed bake has no way to do. But +957 net is +2,204 and -1,247, and the loss has one cause: 1,088 of the 1,247 are blocked by the NADPH-hemoprotein reductase couple, four ids with no formula and no SMILES. The deployed run banked those reactions by mapping them with the protein absent from the SMILES, both mappers agreeing on a mutilated equation. Refusing that is right, and it is still a loss; reporting only the net would hide it. Those four ids are also element-neutral in exactly A1's sense, so a curated `*` body would unblock 1,179 reactions -- the cheapest lever left. V2: GAPDH exact, arms sum to 1.0, S untouched by assertion, 1,904 refusals between two DIFFERENT cofactor families (transhydrogenase, which a one-sided rule misses), and none of the 240 surviving cofactor-substrate rows is a leak -- every one is FAD/FMN in a single redox state, which the both-sides predicate correctly leaves alone. Keys fell by 29 against a criterion that said they must not; 13 were re-derived, no reaction was emptied, and the plan predicted a residue of 26. V3 is the campaign's own indicator and it flipped. 13 of 21 nitrogenases bank, all injective, 10 carry an N2<->NH4 pair. In Nostoc the deployed bake ABSTAINS -- N2 is not a node -- and r6 converges at conductance 3.2157 reaching 31 of 31 endpoints, with L-glutamine and L-glutamate the two largest draws. GS/GOGAT is the route fixed nitrogen actually takes, which is evidence the network is right rather than merely non-empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sv415zpHuVcV43KqueR8wz
…effect that settles The in-flow `.res-head` pixel-chased onto the diagram's first node was always the redundant copy: a plan tall enough to scroll past that point had already carried the sticky bar up to answer the same question. It goes, and with it `HEAD_H` and the `topCy` measurement it needed. The one remaining header, the guide lines and the rows now read a single `--res-cols` template, so they cannot drift apart. The row is right-justified rather than centred, so the fixed-width columns anchor the right edge and only the diagram pans. `seedFromAgent` reads and writes `runParams`/`seededParams`, writing a fresh object each time, so the `$effect` that called it tracked its own writes and re-fired forever. `untrack` keys the effect to `chosenAgent`/`agentChoice`, which is the actual condition for re-seeding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CitDf2CY1HNbefFwufrkrh
…s, params-executor precedence, docs and tests
Ports uncommitted work from the engine/dev worktree, which was on the
same commit but ahead in working-tree state:
- downloadKofamDB.py: extracts the profiles archive instead of moving it
raw into the slot, fixing kofamscan's "not a directory" guard.
- downloadInterProScanDB.py: runs setup.py's indexing in a subshell so
its exit-trap marker doesn't try to write to a directory the task's
uid can't write to.
- kofamscan.py: updated guard message to describe both producers.
- workflow_ops.py: appends an explicit `executor {}` override from
`params.executor` after the free -b/nproc auto-detect, since the
preset's own `executor { cpus = params.executor.cpus }` block is
evaluated before -params-file is merged and silently never applies.
- dev/metasmith.sh, dev/fabfos.sh, docs, RELEASE_PROTOCOL.md, tests:
carried-over fixes and doc updates from the dev-scope session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ftp.uniprot.org has gone down whole-host before (every path 404s, not just this one) with no warning. Pinning a single replacement mirror just resets the same fragility clock -- fall through EBI, uniprot.org, then Expasy instead of failing the step the moment one host is down. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both /log and /trace resolved through logs.latest whenever run_number was still null (staging/staged/launching) -- a symlink shared across every run of a task_key, so a brand-new run briefly showed the PREVIOUS run's log lines and per-step trace rows, painting steps as already done/failed before any real work happened. api.py: short-circuit both endpoints with an empty/pending result the instant run_number is None, instead of ever resolving through the shared symlink. RunView.svelte: key the log-panel display and the progress bar's failure check off `rec.run_number == null` rather than enumerating state names, so this can't reopen if another pre-run state gets added later. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…en resources column
Resource overrides were keyed by step.order and persisted to
localStorage per workflow name. Regenerating a workflow (e.g. pointing
a given at a different source) can drop or add upstream steps and
renumber everything after them, so a stale numeric key silently
reattached to whichever step now sat at that position instead of the
one it was meant for -- the step that actually needed the override got
none. Key by transform name instead (a stable identifier the backend
already matched via its string-keyed `.*__{tr}` branch), and drop any
override naming no step in the current plan before sending it.
Also: "setup environment" hidden behind `{#if false}` pending real
testing -- untested feature, do not ship to release. Resources column
widened ~5% so memory-column chips don't squish, and its header
centered over the grid it names rather than sitting at its left edge.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Staging re-sent every registered library in full on every click -- 3 of 54 transforms, all 85 env resources -- because PrepTransfer queues self.location and nothing narrows it. engine/gui asked for a prune. Two mechanisms were already here for it and neither reached the staging path. PruneTypes was broken in two ways that cancelled out. It kept a namespace only when a *manifest entry* named one of its types, and every entry of a transform library is `transforms::transform`, so it deleted all but that one; the whitelist built from the transforms' own requires/produces could not rescue a namespace, being consulted only inside the branch for namespaces that had already survived. And _persist never unlinked while Load re-reads the whole types/ directory, so the deletion never reached disk and the over-deletion was invisible. Fixing either alone breaks the library. responseSurface needs four namespaces and shipped all 28: 188 KB of type YAML for 28 KB of types. So survival is a union now, the whitelist matches on properties rather than on a node hash a cloned Dependency does not share, index `parents:` entries count as used, and what the directory holds is what survives -- including files left by an older build with a wider --types, which are namespaces nobody declared. The views were the other half. A mask meant "the subset that matters" to the solver and nothing else, because __getattr__ forwarded PruneTypes and PrepTransfer straight to _original. Both are real on the view now. PrepTransfer materializes an image and queues one transfer of it, so the ssh executor still opens one rsync per library rather than one per file, and WorkflowTask.SaveAs derives the mask from the plan: the steps' transforms, plan.given -- already narrowed to the endpoints the plan consumes -- and the ancestor closure of both, which crosses libraries because a parent entry names the library it lives in. The image is the tree MINUS the unused entries, not a selection of the used ones, and that is load-bearing twice. The manifest is never narrowed because the library key is a hash of it and that key names the staged directory, prefixes every packed DataInstance and appears in every step's transform reference. And `build` keeps `_`-prefixed files out of the manifest, so _extract_noncoding_chunks.py and its two siblings -- which promotech.py and bakta_noncoding.py copy out by __file__ -- are runtime dependencies a selection would drop with nothing to say so. On the annotation trio: 370 files / 824 KiB -> 41 / 104. Same task key, same plan.Pack(), and a workflow.nf identical byte for byte modulo the bundle path. Two things this turned up on the way. DataTypeLibrary.Save was a truncating open(..., "w"), which writes THROUGH a hardlink -- and an image links what it images, so pruning the copy rewrote the original. It goes through write_yaml_atomic now, like the index, and the image copies its own metadata rather than linking it. And narrowing the types exposed Spec._get_endpoint resolving a target by finding the first library holding its *namespace* and then asserting the type was in it. That only ever worked because every library carried every type; seven of eleven templates failed until it resolved by the type instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Killing a run left tools running on the agent. Every kill targeted a single
pid, nothing escalated past SIGTERM, an ssh disconnect half-killed a run, and
nothing could say what survived.
A run now has one identity and two groups. start.sh backgrounds the driver
under `set -m` and exports METASMITH_RUN=<key>.<timestamp>, recording both as
RUN.pgid and RUN.token beside PID.lock; the driver puts nextflow in a second
group so a cancel stops the workload and leaves the driver to snapshot logs
and promote the cache. Docker tool containers carry the token as the msm.run
label, apptainer puts it back after --cleanenv, and it survives a setsid, so
the environ scan catches anything that leaves the group.
The relay job protocol now records who asked (.owner) and for which run
(.run), before the .start rename. Its .pid holds a process group -- the
launcher runs each job under `set -m` -- and every stop path signals the
group, TERM then KILL. A job whose requester died is reclaimed by the watcher,
which is what repairs the ownership inversion the bounce creates. The watcher
setsids out of the login shell's group, so a disconnect no longer stops it,
and a shutdown that cannot kill a job keeps that job's records.
CancelWorkflow is a ladder that returns {stopped, survived, rung}; the GUI
writes `cancelled` only when nothing survived, `cancelling` otherwise.
`metasmith workflow ps|reap` are the recovery surface.
Tests: a new `lifecycle` axis (dev/metasmith.sh -tl, 13 tests) covering group
kills, relay reclaim, kill-run scoping, watcher survival across SIGHUP and the
environ-scan backstop; one fast file pinning the generated scripts; two docker
e2e tests for cancel and reap end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e the files are
Leaf identity was a blake3 digest of the file's bytes, taken on whichever
machine registered the row. For an input living on the agent's host the
client cannot read it, so every registration minted a fresh random id and
every re-submission of the same inputs re-keyed the whole cache.
An id is now multihash("stat" || abspath || mtime_ns), and StageWorkflow
re-derives every one of them before compiling -- on the agent, where the
files are, a local stat rather than anything crossing the wire. The plan
goes back to task.yml because CollectResults joins the trace against it,
and the task key is recorded there too: it names the staged directory, and
WorkflowPlan._update_hash folds the given ids in.
The trade is deliberate. Two hosts holding identical bytes at different
paths no longer agree, and an in-place edit that restores mtime is
invisible. The tests that pinned cross-location collapse now pin the same
property against one shared library, which is what a re-run actually is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lishes only its targets Each of kofamscan, diamond_uniref50 and interproscan now produces two files, both carried through the chunk/merge pair the way predictf already does it. UniRef50's hit table gains a header and loses `stitle`, which moves to sseqid -> description; the other two never carried a description at all and gain one from data they were already parsing and discarding -- interproscan's `signature_desc` attribute, and the definition column of the ko_list that transform already requires. The consumers of the uniref50 table follow: the GPR lanes and fabfos_evidence read the header and join the description table, and each now refuses a header it does not recognise rather than emptying the lane. `_MB, _SB, _CB` in metagenomics_from_paired_reads move with the targets inserted above them -- they are positions in TARGETS, and stale they pinned checkm/gtdbtk to the wrong nodes. `publish_intermediates` defaults to False. The filter already existed in nextflow_codegen and nothing outside one test ever set it, so a results folder carried every per-step output. CollectResults still registers an unpublished file, since that manifest entry is what lets a target name it as an ancestor. The run view's file tree moves into the results card; the side panel becomes the selected result -- its full ancestry above the preview, read off the parent graph the collected library re-expands on load. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWL4GMBLduxY85TUM5t3V5
# Conflicts: # src/metasmith/models/workflow/task.py
# Conflicts: # src/metasmith/agents/runner.py
`_write_input` already skips rewriting an input whose content matches, because a leaf id is the file's path and mtime and several tests here model a re-run by calling a builder twice against one tmp_path. `_build_transform_lib` never got that treatment: it rewrote every transform .py, transforms.yml and index.yml unconditionally, and reached for `shutil.copy` -- not copy2, so mtime is not preserved -- for the types file. This extends the same rule to those five writes. It fixes no currently-failing test, and it is not a cache fix. Measured: bumping the mtime of all nine files of a built transform library leaves the hit set identical to baseline, as does the samples library's own _metadata and the shared types.yml. Transform-library mtimes do not reach a cache key. The change is here because a fixture that rewrites identical bytes to model "nothing changed" is saying something it does not mean, in a suite where mtime is load-bearing elsewhere -- not because anything currently depends on it. What does reach a cache key is a leaf inside a GIVEN data library, and there one file is enough: bump one and the hit set empties, including steps that never read it, since the re-keyed leaf moves its consumer's key and so on down the chain. That is `stat_multihash_key(abs_path, st_mtime_ns)` with no content fallback, and it is a property of the identity scheme rather than of anything in this file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWL4GMBLduxY85TUM5t3V5
… from it The engine declared `metasmith_libraries` as a conda run dependency and nothing had ever published that package, so `-bd`, `-bc`, `-bs` and `-uc` all died on an unresolvable spec -- the Dockerfile's `mamba env create` and `conda mambabuild` read the same base.yml. Removing that one line is what unblocks them. It goes back to being vendored, which a52d497 deleted, but not to what a52d497 deleted. That reversal rested on two things, and the change that replaced vendoring is what dissolves both. `clone_stdlib` now copies the library into the project and compiles it there, so nothing shipped needs compiled metadata -- which removes the compile-before-vendor ordering hazard entirely rather than guarding it. `--no-metadata` makes that explicit on the shared verb: fabfos still ships a compiled bundle and is still refused a hollow one, metasmith ships content and compiles its own copy. The other reason was that `templates/` was never in the `--src` list, so a released GUI offered none; it is in it now, and `_assert_library_bundle` counts them. The bundle lands at `src/metasmith/vendor/` specifically because `_build_hash.py` walks that tree. Library content therefore moves the engine's build hash, the wheel, the image tag and the SIF together -- measured, not assumed: editing a transform took the hash d7898e5 -> 99cdd1f and restoring it took it back. A library and the engine that reads it can no longer disagree about which is which, which is the property the monorepo exists to provide and the reason the library stops being separately versioned and published at all. There were two resolvers, not one. `agents/conda.py` finds the `envs/` recipes a `--runtime mamba` deployment creates its tool envs from, and it had its own copy of the find_spec lookup; fixing only `gui/stdlib.py` would have left mamba resolving nothing. Both call `standard_library_root()` now: vendored bundle first, an importable `metasmith_libraries` second, which is what a source checkout has and an installed package never needs. Also deletes `src/metasmith/std/`. It was generated by a research script, shipped in the wheel, the conda package, the image and the SIF, and read by nothing in `src/` -- its only consumer reads the `research/` copy. Shipping it beside the vendored library would have put two transform libraries in one wheel. Verified with `metasmith_libraries` not importable at all: the bundle resolves, clones, compiles 13 `_metadata` directories, and yields 20 data types, 11 transform libraries and all 11 templates, stamped with the engine's own version. That case -- a wheel, a `pip install`, a mamba or native runtime -- previously got no library by any mechanism, because conda's solve was the only thing that could ever have delivered one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lished The staging guard proves the bundle was in the tree; it cannot prove it reached the artifact. This is the same three-place shape the solver engine already has, for the same reason and with the same failure mode: an image whose bundle is absent or empty installs, imports and runs, and the only symptom is a blank type panel and no templates -- which nobody sees until a user opens the page. Runs at -bs and -ud, beside _assert_engine_in_image, and asks the metasmith inside the image to resolve its own library and count what it found. MSM_SKIP_LIBRARY_CHECK is the shared override, the same way the two solver checks share MSM_SKIP_SOLVER_CHECK -- they are one claim checked in more than one place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and docs that no longer vendor
The executor is sized by appending a literal `executor { }` block to the config
file rather than by `params`: the preset's block is evaluated at parse time,
before the `-params-file` merge, so every caller was heard at runtime and
ignored at submit.
downloadKofamDB unpacks profiles.tar.gz with --strip-components=1 and fails on
an empty profile set instead of moving the archive into the slot;
downloadInterProScanDB runs setup.py in a subshell so the exit-code trap still
fires in a writable cwd; kofamscan's staging error names both producers.
dev/metasmith.sh --idev/--ibase reach envs/metasmith. AGENTS.md drops
--vendor-library, the consolidation followup re-points the unrecoverable-pin
entry at data/fabfos/runs/aska/gpr, and RELEASE_PROTOCOL.md carries a measured
budget for the full sweep.
Tests follow the sandbox short-circuit and the Deploy on_phase signature; the
flow fan-out budget is split from the batching assertion and xfailed; the
setup-environment tests shut off the installed-library recipe root so a staged
envs/tools cannot change the answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scope's own commit (8b6a796) was already contained in release verbatim, so the merge resolves entirely to release's side: the merged tree is byte-identical to release. The one conflict, in workflow_ops.py, was release adding the local executor's free -b/nproc auto-detect ahead of the params-driven executor block both sides already carried.
…ked state Overrides now persist to workflows/<name>/overrides.yml, the same atomic-write style as request.yml/result.yml, instead of living only in browser localStorage. Each workflow also gets its own editable copy of the Nextflow config preset it launches with (workflows/<name>/preset.nf), seeded eagerly at creation and carried through fork -- decoupled from the shared package preset it started from, editable as raw text via a reused ConfigEditor. A pre-existing workflow that predates this self-heals its own preset copy the first time a run is launched against it. The workflow page now splits the solve action, the solved plan/DAG, and the stage-and-run controls (params, agent, preset, launch) into three cards instead of one, dropping the long-dead disabled setup-environment markup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four defects in the task cache, each confident in the silent direction: a run carrying all of them still reported completed. A directory could not be cached at all. `_find_step_outputs` and the staging fallback both filtered `is_file()`, so a directory product was left at the shard root; `cached_files_for_branch`, `_populate_hit_outputs` and `_published_index` mirrored the same predicate on the read side. The walk now treats an entry named in the canonical output spelling as one output, file or directory, and does not descend into it. Nothing branches on the declared extension to decide which: `GetPreferredFileExtension` answers "" for plenty of file types and a directory type can carry one. Shard size sums the tree, since the directory inode's own size would tell `cache gc --max-size` a few kilobytes for a product of any size. A hit that could not serve a declared slot emitted `Channel.empty()` into a group that never completed -- declared, never submitted, no trace row, no error. `compute_cache_decisions` now asks the shard manifest the same question the emitter asks of the shard, per slot, so the fallthrough could become a raise that cannot fire. Demotion is all-or-nothing: half a step from the shard and half from a fresh execution emits a channel whose members came from two different runs of the transform. A hit published nothing into `results/`. Nextflow adds a path to the publish set only when `PublishOp.getTaskDir` resolves it under `session.workDir`, and a cache shard is outside it -- dropped with no log and no error, so nothing the emitter puts on a channel can reach `results/` from a shard. The emitter records the channel-to-directory spelling in `workflow.cache_publish.json` and the driver places the products after nextflow exits. The virtual runtime stops publishing hits itself; publishing them there is what let a green suite coexist with a run that lost every cached product. `WaitForWorkflow` called a healthy run failed. `PID.lock` goes when nextflow exits, but the driver runs on through promotion, results and log gathering, and the driver is what writes the sentinel. The poll gained a liveness probe on `RUN.pgid`; absence of the file still falls back to the old answer rather than waiting forever. Its output is now labelled rather than positional, because `grep -c` prints its 0 *and* exits 1, so the `|| echo 0` fallback emitted the count twice and shifted every line after it. Alongside: `msm data invalidate` moves a leaf's mtime strictly forward and re-mints through the same formula the agent uses, which is the lever stat-only identity was missing for a change below the top node. And `docker_builder` lost `-f docker/metasmith/Dockerfile` in the monorepo reshape, so every test under `tests/metasmith/e2e/docker/` had been erroring at fixture setup; its build timeout was also below a cold conda solve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…riting Staged libraries are pruned per-transform, so two libraries in the same namespace can carry disjoint types. The transform_libraries loop was overwriting tlibs[namespace] wholesale instead of merging like the data_libraries loop already did, so the last-loaded library silently clobbered types an earlier one had -- surfacing as "datatype [gff] not found in [sequences]" once a run's transforms landed in staging order that put the type-bearing library first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
compute_cache_decisions (run from PrepareNextflow) mutates instance_id in place on the plan's produce/require instances, stamping deterministic lineage ids -- the same ids baked into every .nf/.meta file. task.yml was only written before that stamping, so it kept stale pre-stamp ids while execution metadata referenced the new ones. The first step whose input was another step's output (rather than a leaf) then failed to resolve: MissingInstanceError: required instance_id=... missing for dep=.... Writing the plan again after PrepareNextflow keeps task.yml in agreement with what the generated workflow actually references. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every run of a workflow stages into the same task_key workspace on the agent -- one PID.lock, one process group. Nothing stopped launching a new run while an earlier one was still live, so the second run silently took over that shared workspace; cancelling either run's button afterwards killed whatever the agent was actually running, not necessarily the one that was clicked. create_run now rejects launching a new run for a workflow that already has a live one, mirroring the existing guard on delete_workflow. The launch button in WorkflowView disables itself and explains why, instead of surfacing this only as a failed request. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A step whose result comes from metasmith's own content-addressed cache never becomes a Nextflow process -- codegen splices the cached files straight into the channel instead of invoking it -- so it has no row in Nextflow's own trace file. The GUI's run view only ever read that file, so a cached step rendered as if it had never run. Cache hits are already recorded as InvocationEvents in _metasmith/trace.jsonl at staging time. ReadCacheHits (run_control.py) reads it from the agent -- preferring the copy runner.py now makes in the run's own logs.<timestamp> dir once a run finishes (so a collected run keeps its cache-hit rows after the live copy gets truncated by the next stage), falling back to the live workspace copy only while the requested run still owns it. parse_cache_hits/_merge_cache_hits (ops/runtime.py) turn "hit" events into synthetic trace rows shaped exactly like real ones, named via the same NextflowProcessName helper the /steps endpoint already uses, so no frontend change is needed -- RunView's existing per-process grouping picks them up for free. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Opal <noreply@anthropic.com>
Co-Authored-By: Opal <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts 0.21.0 from the fork's
releasebranch.quay.io/hallamlab/metasmith:0.21.0-173bdf4— published, plus the movedlatestand0.21.0tags, all atsha256:c55b1db8.anaconda.org/hallamlab/metasmith0.21.0 — published asmetasmith-0.21.0-py_0.tar.bz2, after a clean-room install into a throwaway env reportedbackend=rust.What is in it
The three engine lines merged into
release—feat/dev,feat/guiandfeat/engine/process-management— plus the vendoring change that unblocked packaging at all.envs/metasmith/base.ymlno longer declaresmetasmith_libraries, which is what made-bd/-bc/-ucsolvable again. The bundle is stagedinto
src/metasmith/vendor/and lives inside the tree the build hash walks, so the library'scontent is part of the engine's version and the two cannot drift.
disk-backed; a three-card workflow page; live docker/apptainer pull progress in the deploy log;
cache-hit steps reported in the run trace; a second run refused while one is live.
7.5s → 1.1s on
metagenomics_from_paired_reads.Behaviour changes worth knowing about
WorkflowPlan.publish_intermediatesnow defaultsFalse. A run that dies before its finalstep leaves an empty results folder where it used to leave per-step outputs. Per-step logs are
the replacement.
makes a 24 GB DIAMOND database or a 27k-file profiles tree cost one stat instead of a full-tree
hash, and the trade was taken deliberately. Two consequences follow from it:
dvc checkoutunder areference tree is enough, and one file will do it. Mitigation: pin the library —
restat_leaf_ids()skips pinned libraries by design.stat_leaf_idstats the top level only, so a change belowthe top level does not move the id and a cache hit can return results computed from the old
input. Reproduced end-to-end against a real
dvc checkout. It is unfixed here: the fix iscontent addressing with stat as a memo key, which changes
CACHE_KEY_VERSIONand invalidatesevery existing shard — a migration rather than a patch.
Verification
Dependency-free tier green before the bump. Every packaging guard ran against the artifacts
themselves rather than the tree they were built from: real relays in all four slots,
backend=rustin both the image and a clean-room conda install, and a non-empty standard library in the image
(20 data types, 11 transform libraries, 11 templates).