diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index c08164c..2dbca0a 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -9,7 +9,7 @@ - **How it affects the project** ## Testing - + - **Commands run and results** - **Unit tests added/updated** - **Integration tests added/updated** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 187cde5..7b350b8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -114,21 +114,21 @@ uv run pre-commit install --install-hooks Hooks are split across two stages so that committing stays cheap while the checks that most often break CI still run before anything leaves your machine. -On every **commit** — fast, auto-fixing: +On every **commit**, fast and auto-fixing: - `ruff`: fixes lint issues where possible. - `ruff-format`: formats Python files. - `cargo-fmt`: runs `cargo fmt --check --manifest-path rust/Cargo.toml`. - `uv-lock-check`: runs `uv lock --check` when `pyproject.toml` or `uv.lock` changes, so a dependency edit that was never relocked fails here instead of as an opaque CI sync error. -On every **push** — the whole-repo gates: +On every **push**, the whole-repo gates: - `pyright`: the same type check CI runs. - `cargo-clippy`: all targets, warnings denied. The ruff hooks cover the **whole tree**, matching CI's `ruff check .`. They used to be scoped to `src/` and `tests/`, which meant `examples/` could only ever fail in CI. -The full pytest suite and `cargo test` are deliberately in neither stage — they rebuild the Rust extension, and CI shards them across four runners far faster than a local serial run. Use `make check` when you want everything locally. +The full pytest suite and `cargo test` are deliberately in neither stage; they rebuild the Rust extension, and CI shards them across four runners far faster than a local serial run. Use `make check` when you want everything locally. ## Running subsets diff --git a/README.md b/README.md index fb54cbe..2bbe366 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,15 @@ [![License](https://img.shields.io/pypi/l/tablassert.svg)](https://github.com/SkyeAv/Tablassert/blob/main/LICENSE) [![Docs](https://img.shields.io/github/deployments/SkyeAv/Tablassert/github-pages?label=docs)](https://skyeav.github.io/Tablassert/) -> Extract knowledge assertions from tabular data into NCATS Translator-compliant KGX NDJSON — +> Extract knowledge assertions from tabular data into NCATS Translator-compliant KGX NDJSON, > declaratively, with entity resolution built in and optional quality control. Tablassert turns biomedical spreadsheets (Excel, CSV, TSV) into knowledge graphs ready for NCATS -Translator. Declare how your columns map to subject–predicate–object statements in YAML; Tablassert +Translator. Declare how your columns map to subject-predicate-object statements in YAML; Tablassert resolves free text to standard CURIEs, attaches provenance and statistical annotations, and emits KGX-compliant nodes and edges. -**[Full Documentation](https://skyeav.github.io/Tablassert/)** — installation guides, tutorial, +**[Full Documentation](https://skyeav.github.io/Tablassert/)**: installation guides, tutorial, configuration reference, and API docs. ## Quick Start @@ -23,7 +23,7 @@ configuration reference, and API docs. pip install tablassert ``` -Given a CSV of gene–disease associations with p-values and sample sizes, declare the mapping in a +Given a CSV of gene-disease associations with p-values and sample sizes, declare the mapping in a table config (`table.yaml`): ```yaml @@ -77,7 +77,7 @@ Build the knowledge graph: tablassert build-kg graph.yaml ``` -Output is one JSON object per line — nodes with Biolink categories, edges with annotations: +Output is one JSON object per line: nodes with Biolink categories, edges with annotations. ```json {"id":"HGNC:11998","name":"TP53","category":["biolink:Gene"],"taxon":"NCBITaxon:9606"} @@ -92,15 +92,15 @@ See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full w ## Key Features -- **Declarative YAML configuration** — define data transformations without writing code -- **Built-in entity resolution** — map free text to genes, diseases, and chemicals with standard +- **Declarative YAML configuration**: define data transformations without writing code +- **Built-in entity resolution**: map free text to genes, diseases, and chemicals with standard CURIEs, taxonomic filtering, and provenance, backed by an embedded redb database -- **Optional quality control** — a four-stage audit (exact → fuzzy → abbreviation → SapBERT embeddings) flags +- **Optional quality control**: a four-stage audit (exact → fuzzy → abbreviation → SapBERT embeddings) flags low-confidence mappings -- **KGX compliance** — emits NCATS Translator-compatible node/edge NDJSON with Biolink categories +- **KGX compliance**: emits NCATS Translator-compatible node/edge NDJSON with Biolink categories and predicates -- **Autonomous agent** — `tablassert agent` derives, builds, and refines configs for whole papers -- **Performance & reproducibility** — lazy Polars pipelines and a deterministic UV-based +- **Autonomous agent**: `tablassert agent` derives, builds, and refines configs for whole papers +- **Performance & reproducibility**: lazy Polars pipelines and a deterministic UV-based development environment ## Installation @@ -142,19 +142,19 @@ results = resolve_many( # [{"original_gene": "TP53", "gene": "HGNC:11998", "gene_name": "TP53", ...}, ...] ``` -Point `resolve_many()` at a fullmap database to resolve any iterable of entity strings to CURIEs — +Point `resolve_many()` at a fullmap database to resolve any iterable of entity strings to CURIEs, no LazyFrame setup or NLP preprocessing required. See the [Batch Resolution API](https://skyeav.github.io/Tablassert/api/lib/) for the full reference. ## Documentation -- **[Installation](https://skyeav.github.io/Tablassert/installation/)** — install methods, extras, and development setup -- **[Tutorial](https://skyeav.github.io/Tablassert/tutorial/)** — step-by-step example with synthetic data -- **[CLI Reference](https://skyeav.github.io/Tablassert/cli/)** — complete command-line flag reference -- **[Use Case Gallery](https://skyeav.github.io/Tablassert/examples/)** — real-world configuration patterns -- **[Configuration](https://skyeav.github.io/Tablassert/configuration/graph/)** — graph and table configuration reference -- **[Agent](https://skyeav.github.io/Tablassert/agent/)** — the autonomous agent pipeline -- **[API Reference](https://skyeav.github.io/Tablassert/api/fullmap/)** — core functions documentation +- **[Installation](https://skyeav.github.io/Tablassert/installation/)**: install methods, extras, and development setup +- **[Tutorial](https://skyeav.github.io/Tablassert/tutorial/)**: step-by-step example with synthetic data +- **[CLI Reference](https://skyeav.github.io/Tablassert/cli/)**: complete command-line flag reference +- **[Use Case Gallery](https://skyeav.github.io/Tablassert/examples/)**: real-world configuration patterns +- **[Configuration](https://skyeav.github.io/Tablassert/configuration/graph/)**: graph and table configuration reference +- **[Agent](https://skyeav.github.io/Tablassert/agent/)**: the autonomous agent pipeline +- **[API Reference](https://skyeav.github.io/Tablassert/api/fullmap/)**: core functions documentation ## Developing @@ -182,6 +182,6 @@ described in: ## Contributors -- [Skye Lane Goetz](mailto:sgoetz@isbscience.org) — Institute for Systems Biology -- [Gwênlyn Glusman](mailto:gglusman@isbscience.org) — Institute for Systems Biology -- Jared C. Roach — Institute for Systems Biology +- [Skye Lane Goetz](mailto:sgoetz@isbscience.org), Institute for Systems Biology +- [Gwênlyn Glusman](mailto:gglusman@isbscience.org), Institute for Systems Biology +- Jared C. Roach, Institute for Systems Biology diff --git a/docs/agent.md b/docs/agent.md index 4930fba..f47d779 100644 --- a/docs/agent.md +++ b/docs/agent.md @@ -1,11 +1,11 @@ # Autonomous Agent (`[agent]` extra) **Why this exists:** hand-authoring a Tablassert config for every PMC supplementary table does not scale. -The optional `[agent]` extra makes it autonomous — point it at **PubMed Central (PMC)** article IDs and it +The optional `[agent]` extra makes it autonomous: point it at **PubMed Central (PMC)** article IDs and it **derives the config for you**, then builds, audits, and iteratively improves the graph until the entity resolution *maps* (coverage threshold). The outcome is an **NCATS Translator-compliant KGX knowledge -graph** per article — a claim the loop verifies rather than asserts, by constructing every emitted -record as its own Biolink class (see [Biolink validity](#biolink-validity)) — with the whole loop +graph** per article, a claim the loop verifies rather than asserts, by constructing every emitted +record as its own Biolink class (see [Biolink validity](#biolink-validity)), with the whole loop scored on **quality / cost / wrong tool calls**. Under the hood it is built on [smolagents](https://github.com/huggingface/smolagents) `CodeAgent` (a @@ -42,7 +42,7 @@ The `[optimize]` extra (only needed for `agent --optimize`) pins: ## PMC-AWS data source -Tables are fetched from the **new** PMC open-access S3 bucket — the sanctioned bulk path. +Tables are fetched from the **new** PMC open-access S3 bucket, the sanctioned bulk path. | | | | --- | --- | @@ -57,9 +57,9 @@ failing fast (cheap checks before any large download and before any model call): 1. Enumerates version prefixes via S3 `list-objects-v2` (`?list-type=2&prefix=PMC.&delimiter=/`) and selects the **latest** version (numeric, so `PMC.10` beats `PMC.2`); older versions are ignored. 2. Checks the latest version's `.json` metadata for open access (`is_pmc_openaccess` / a `CC*` - `license_code`) — **before** any large download (not open access ⇒ `PermissionError` immediately). + `license_code`), **before** any large download (not open access ⇒ `PermissionError` immediately). 3. Enumerates the version's objects (`?list-type=2&prefix=PMC./`) and confirms a data table is - present (a file with extension `.xlsx .xls .csv .tsv`) — **before** any large download (none ⇒ + present (a file with extension `.xlsx .xls .csv .tsv`), **before** any large download (none ⇒ `FileNotFoundError`). 4. Downloads only the **useful** files to `outdir//` and returns their paths: the main text (`.xml`/`.nxml`/`.txt`/`.pdf`), the `.json` metadata, and every data table. Binary media (images, @@ -102,7 +102,7 @@ config to be re-fetchable). A `--local` directory that does not exist fails loud ## Model configuration The agent talks to an **OpenAI-compatible** endpoint (e.g. a Qwen endpoint). Configuration comes from -CLI flags **and** environment variables — **secrets are never hardcoded**, and the command **fails +CLI flags **and** environment variables: **secrets are never hardcoded**, and the command **fails loudly** if any required value is unset. | Flag | Env var | Purpose | @@ -110,12 +110,12 @@ loudly** if any required value is unset. | `--model-id`, `-m` | `TABLASSERT_AGENT_MODEL_ID` | model identifier | | `--api-base`, `-ab` | `TABLASSERT_AGENT_API_BASE` | OpenAI-compatible base URL | | `--api-key`, `-ak` | `TABLASSERT_AGENT_API_KEY` | API key (secret) | -| `--backend`, `-b` | — | `openai` (default) or `litellm` | +| `--backend`, `-b` | n/a | `openai` (default) or `litellm` | ```bash export TABLASSERT_AGENT_MODEL_ID="qwen3-max" export TABLASSERT_AGENT_API_BASE="https://YOUR-ENDPOINT.example.com/v1" # placeholder -export TABLASSERT_AGENT_API_KEY="sk-***" # placeholder — never commit a real key +export TABLASSERT_AGENT_API_KEY="sk-***" # placeholder: never commit a real key ``` If a value is missing, `tablassert agent` prints a message naming the exact flag/env var and exits @@ -139,29 +139,29 @@ artifact metadata, and existing table list. Flags: `--max-steps`/`-ms`, `--map-t `--max-improve-iters`/`-mi`, `--state-dir`/`-sd`, `--backend {openai,litellm}`/`-b`, plus `--local`/`-l`, `--reflexion`, `--judge-model`, `--judge-threshold`, `--biolink-threshold`, and the `--optimize`/`-o` prompt-optimization flags (`--instructions-file`, `--instructions-out`, `--max-metric-calls`, `--dataset`). -The [CLI reference — `agent`](cli.md#agent) is the authoritative flag table; the list here is a compact +The [CLI reference: `agent`](cli.md#agent) is the authoritative flag table; the list here is a compact reminder. ### What the supervisor does -The **outer supervisor is deterministic Python** (not an LLM) — smolagents' #1 practice is deterministic +The **outer supervisor is deterministic Python** (not an LLM); smolagents' #1 practice is deterministic control flow over agentic decisions. For each PMC id it: 1. **Fetches** the latest-version article payload (`fetch_pmc_article`: main text + metadata + all tables; fails fast on not-open-access / no-table) and presents **all** candidate tables to the agent. 2. Runs the **inner `CodeAgent`** to *derive* an initial table config (`pmc_article_context` → `read_table` → `derive_config`, every section gated by the Section JSON schema). The agent maps **each** mappable - table/worksheet as its own section — **one config per paper** (see below). + table/worksheet as its own section, **one config per paper** (see below). 3. **Builds + audits** in one deterministic mega-tool (`build_and_audit`: validate → build → QC → coverage → **Biolink validity**). 4. **Improves** while coverage `< map_threshold` and budget remains: `propose_config_edit` → rebuild → - **accept iff no worse on coverage *or* Biolink validity and strictly better on one** (monotonic — + **accept iff no worse on coverage *or* Biolink validity and strictly better on one** (monotonic: regressions on either axis are rejected, so a coverage win can no longer be bought with invalid KGX). 5. **Records** metrics, **checkpoints**, and moves to the next config. A config that won't map after `--max-improve-iters` is marked `SKIPPED: ` and the supervisor -advances — one difficult article never aborts the batch. A config that **builds** but whose fullmap -coverage **cannot be measured** (an unreproducible source frame) is marked `BUILT_UNMEASURED` — a +advances: one difficult article never aborts the batch. A config that **builds** but whose fullmap +coverage **cannot be measured** (an unreproducible source frame) is marked `BUILT_UNMEASURED`, a terminal **non-failure** that is neither a certified `MAPPED` nor counted as a `SKIPPED`; the best config is still written and is reusable by the full pipeline. Coverage measurement itself is multi-cwd: a relative `source.local` is resolved against the build workdir as well as the current @@ -172,13 +172,13 @@ directory before a config is declared unmeasurable. Two opt-in extensions layer on top of the deterministic improve loop (both reuse the configured endpoint; neither is required): -- **`--reflexion`** — when the deterministic `propose_config_edit` stalls, a tier-2 LLM reflexion +- **`--reflexion`**: when the deterministic `propose_config_edit` stalls, a tier-2 LLM reflexion improver reflects on the coverage feedback and proposes an edit that may change predicate/source (same model config). -- **`--judge-model` / `--judge-threshold`** — a semantic judge scores the built output; when +- **`--judge-model` / `--judge-threshold`**: a semantic judge scores the built output; when `--judge-model` is set, `MAPPED` additionally requires the normalized score to clear `--judge-threshold` (`0.5` when unset). Without `--judge-model` the coverage gate alone decides. -- **`--biolink-threshold`** — `MAPPED` additionally requires the built KGX's Biolink pass rate to +- **`--biolink-threshold`**: `MAPPED` additionally requires the built KGX's Biolink pass rate to clear it. Defaults to `0.0` (report only): the rate is always measured and recorded, and raising the threshold turns that measurement into a terminal gate. See [Biolink validity](#biolink-validity) below. @@ -187,7 +187,7 @@ endpoint; neither is required): Coverage answers *did the terms resolve?* It says nothing about whether the resulting records are consumable. The agent therefore validates **its own output**: after each build, `build_and_audit` -constructs every emitted node and edge as the Biolink Pydantic class named by its own `category` — +constructs every emitted node and edge as the Biolink Pydantic class named by its own `category`, the same check [`tablassert validate-kgx`](cli.md#validate-kgx) runs, and the same classes `translator-ingests` builds. Four fields land in the audit report: @@ -200,7 +200,7 @@ the same check [`tablassert validate-kgx`](cli.md#validate-kgx) runs, and the sa **`demoted_edge_pct` is the predicate signal.** Tablassert derives an edge's association class from the (subject category, object category) pair, then `resolve_association_class` gives up as much of -that class as the predicate requires. A predicate the class forbids is **never an error** — it +that class as the predicate requires. A predicate the class forbids is **never an error**: it silently demotes the edge and discards every qualifier and evidence slot that class declared. So `gene_associated_with_condition` on a gene~disease table builds cleanly, maps perfectly, and produces `biolink:Association` edges. Nothing but this number tells you. @@ -222,7 +222,7 @@ cannot drift from the model the build validates against: these on `Association`, so a strict check rejects edges carrying them. `biolink_valid_pct` exempts them (and the other curated KGX carryovers) so the agent is scored on **its own** decisions. - The exempt set is *derived* — `TABLASERT_EDGE_EXTRAS - ` — so it + The exempt set is *derived* (`TABLASERT_EDGE_EXTRAS - `), so it empties itself when the model catches up, with no code change. Two related silent behaviours the agent's prompt now names, since neither raises: @@ -231,7 +231,7 @@ Two related silent behaviours the agent's prompt now names, since neither raises attached to **no** Pydantic class, so its value is routed onto the inlined `StudyResult` rather than emitted on the edge. Names that are not association slots at all (`q_value`, `fold_change`, …) are folded into `supporting_text`. Authoring either now emits a `BiolinkRelocationWarning` naming where - the value actually went — a warning, not an error: nothing is lost, and every existing config + the value actually went, a warning, not an error: nothing is lost, and every existing config keeps building. - Enum-ranged qualifiers take a literal token (`object_direction_qualifier: increased`), never a CURIE, and are deliberately **not** entity-resolved. `map_coverage` skips them for the same reason @@ -239,10 +239,10 @@ Two related silent behaviours the agent's prompt now names, since neither raises ### Multi-section configs (one per paper) -The agent authors **one table config per paper** that may contain **multiple sections** — one per +The agent authors **one table config per paper** that may contain **multiple sections**, one per mappable supplementary table/worksheet. The config is shaped as `{template, sections}`: -- **`template`** carries the shared per-paper **provenance** (`repo` + `publication`) and nothing else — +- **`template`** carries the shared per-paper **provenance** (`repo` + `publication`) and nothing else: in particular **no `source`**. - **`sections`** is a list with one entry per table; **each section owns its own `source`** (its own `local` path **and** its own `source.url` download link, plus `sheet`/`row_slice`/`delimiter` as @@ -332,27 +332,27 @@ gate can only answer true/false and would otherwise swallow the reason. The agent's `instructions` make the techniques explicit: -- **ReAct + planning** — `CodeAgent` is a ReAct loop; `planning_interval=3` re-plans every few steps. -- **Structured / constrained output** — `derive_config` injects the Section JSON schema; a +- **ReAct + planning**: `CodeAgent` is a ReAct loop; `planning_interval=3` re-plans every few steps. +- **Structured / constrained output**: `derive_config` injects the Section JSON schema; a `final_answer_checks=[validate_table_config]` gate means the agent can only terminate with a config whose **every section** is schema-valid (multi-section configs are validated section-by-section). -- **Few-shot exemplars** — the tutorial gene~disease section, the ALAMV6 organism~chemical section, and a +- **Few-shot exemplars**: the tutorial gene~disease section, the ALAMV6 organism~chemical section, and a multi-section config (one config, two tables, each section its own source/url). -- **Reflexion-style self-critique** — `propose_config_edit` / `reflexion_improve` reflect on failing rows, +- **Reflexion-style self-critique**: `propose_config_edit` / `reflexion_improve` reflect on failing rows, error codes, and unresolved terms, then make a targeted, schema-valid edit. -- **Error-recovery prompting** — tools return rich coded errors; the prompt directs the agent to read the +- **Error-recovery prompting**: tools return rich coded errors; the prompt directs the agent to read the code + message and fix precisely that field, never repeating an unchanged config. -- **Context trimming** — a `step_callback` tallies tokens/steps and failed/wrong/redundant tool calls, and +- **Context trimming**: a `step_callback` tallies tokens/steps and failed/wrong/redundant tool calls, and trims large old observations to save tokens. ### Prompt-injection defenses PMC article text and tables are **untrusted data**. Defenses: -- **Data-fence + spotlighting** — `read_table` wraps content in `<<>>` / +- **Data-fence + spotlighting**: `read_table` wraps content in `<<>>` / `<<>>` preceded by a guardrail; the instructions state that fenced content is DATA, never instructions, and any embedded commands are ignored. -- **Minimal authorized imports** — the executor allowlist is just `["yaml"]`, so a hijacked agent cannot +- **Minimal authorized imports**: the executor allowlist is exactly `["yaml"]`, so a hijacked agent cannot `import os`/`subprocess`. ## Evaluation & optimization loop @@ -361,22 +361,22 @@ The harness scores every run on three objectives and optimizes them as a black b **Deterministic metrics (gate the loop):** -- **Quality** — fullmap mapping coverage (0.40), **Biolink pass rate** (0.25), KG node/edge **F1** vs +- **Quality**: fullmap mapping coverage (0.40), **Biolink pass rate** (0.25), KG node/edge **F1** vs the reference graph (0.15), QC audit pass rate (0.10), and config schema validity (0.10, and a hard gate: an invalid config scores 0). -- **Cost** — `RunResult.token_usage` + step count (the API is free; tokens are the proxy). -- **Reliability** — failed / wrong / redundant tool-call counts from the `ActionStep` logs. +- **Cost**: `RunResult.token_usage` + step count (the API is free; tokens are the proxy). +- **Reliability**: failed / wrong / redundant tool-call counts from the `ActionStep` logs. **LLM-as-judge (semantic dimensions only):** a pointwise **0–3** rubric over schema validity, coverage, **Biolink validity**, QC pass, predicate/category appropriateness, provenance completeness, efficiency, -and tool-call cleanliness — with **position** (both orderings averaged) and **verbosity** bias mitigation. Deterministic +and tool-call cleanliness, with **position** (both orderings averaged) and **verbosity** bias mitigation. Deterministic metrics gate the rest; the judge only scores what a metric cannot. Without a judge model, an offline deterministic heuristic is used. **Optimizers:** -- **Reflexion** — the simple first-increment retry (`reflexion_improve`). -- **GEPA** — `dspy.GEPA(metric=gepa_metric, candidate_selection_strategy="pareto", …)` optimizes the +- **Reflexion**: the simple first-increment retry (`reflexion_improve`). +- **GEPA**: `dspy.GEPA(metric=gepa_metric, candidate_selection_strategy="pareto", …)` optimizes the agent's `instructions` + tool `description`s + exemplars as a **black box** from textual feedback (`gepa_metric` returns `dspy.Prediction(score=weighted_quality, feedback="")`). It is system-agnostic, Pareto-native, and needs few rollouts. @@ -393,7 +393,7 @@ Following GEPA best practice, the optimizer splits the models: a **strong reflec proposes the few instruction edits, and an optional **fast task LM** (`--task-model`) runs the many candidate program evaluations. Pointing `--task-model` at a cheap model (e.g. a flash model) keeps the run fast while the strong model does the thinking; without `--task-model` the reflection LM is used for -both. `--gepa-threads` parallelizes GEPA's candidate **LM forward passes** only — the coverage-scoring +both. `--gepa-threads` parallelizes GEPA's candidate **LM forward passes** only: the coverage-scoring builds stay serialized on the process-wide `_GEPA_BUILD_LOCK` (`agent.py`, since `os.chdir` is process-global), so a higher thread count does not speed up the expensive build/coverage step. @@ -412,12 +412,12 @@ tablassert agent PMC11708054 --configuration-file ./graph.yaml \ `--dataset` is a YAML/JSON list of examples. Each example carries `table_summary` and `coverage_feedback` (the program inputs); it MAY also carry: -- `fullmap` — a fullmap path. When present, the GEPA metric scores each proposed config with **real +- `fullmap`: a fullmap path. When present, the GEPA metric scores each proposed config with **real fullmap coverage** (via a `build_and_audit` head-sample), so GEPA optimizes the genuine objective rather than a validity-only proxy. -- `workdir` — the directory a proposed config's relative `source.local` resolves against (LLMs mimic the +- `workdir`: the directory a proposed config's relative `source.local` resolves against (LLMs mimic the exemplar's `./downloads/...` paths), so coverage is measured on the actual table. -- `head` — default `true`: score a fast 5-row preview; set `false` for full-fidelity coverage builds. +- `head`: defaults to `true`, which scores a fast 5-row preview; set `false` for full-fidelity coverage builds. `--max-metric-calls` bounds the GEPA metric budget. `save_optimized_instructions` / `load_optimized_instructions` persist and reload the prompt (a `{instructions, descriptions}` mapping). @@ -428,21 +428,21 @@ live model; the offline suite exercises this path via an injectable `gepa_cls` s `tests/agent_fixtures/PMC11708054/` is an offline replay pair: the ALAMV6 reference config, a small **synthetic** source table, and a trimmed reference config (CC-BY attribution to PMC11708054; the -reference KGX is computed in-test against a tiny real redb — nothing large is committed). A second +reference KGX is computed in-test against a tiny real redb; nothing large is committed). A second fixture, `tests/agent_fixtures/GENE_DISEASE/`, is a gene~disease config in multi-section -(`{template, sections}`) shape with PMID provenance — used to keep the offline heuristic judge and the +(`{template, sections}`) shape with PMID provenance, used to keep the offline heuristic judge and the W3 multi-section validation honest on a distinct config. ## Edge-count acceptance (agent vs reference) An agent-produced config is only as good as the graph it emits. The acceptance gate is an **edge-count fraction**: built over the SAME payload against the SAME fullmap, the agent config must emit at least -half the KGX edges of a richer hand-curated reference config — `agent_edges >= 0.5 * reference_edges` +half the KGX edges of a richer hand-curated reference config: `agent_edges >= 0.5 * reference_edges` (`REFERENCE_EDGE_FRACTION` in `tests/test_agent_edgecount.py`). A config that reads only one sheet, or that skips `explode_by` on a multi-valued column, silently emits far fewer edges; this gate catches it. **Offline harness (committed fixtures):** `tests/fixtures/edgecount/` ships a synthetic -PMC10766526-shaped disease x system workbook plus three configs — the improved-agent shape +PMC10766526-shaped disease x system workbook plus three configs: the improved-agent shape (multi-section, correct `sheet` + `row_slice`, `explode_by`/`prioritize` breadth, paired `effect_size` + `effect_type` annotations), a strictly richer reference (all three sheets), and an intentionally-impoverished single-section no-`explode_by` negative control that MUST fail the gate: @@ -474,7 +474,7 @@ wc -l _.edges.ndjson _ ``` The same comparison is scriptable via the env-gated test: set `TABLASSERT_PMC_COMPARE` to a JSON -array of four paths — the agent config, the reference config, the payload, and the fullmap redb. +array of four paths: the agent config, the reference config, the payload, and the fullmap redb. A JSON array (not a colon-separated string) keeps POSIX paths containing `:` and Windows drive-letter paths working. The `` and `` may carry relative `source.local` paths; both are rebuilt over ``: @@ -488,7 +488,7 @@ Unset, that test skips with a printed reason; the offline fixture tests run rega ## Testing -The agent suite is **fully offline** — no live LLM or network. It uses a `FakeModel` smolagents stub, +The agent suite is **fully offline**: no live LLM or network. It uses a `FakeModel` smolagents stub, mocked/snapshotted PMC data, a tiny real redb (`rs.build_fullmap_db`), and injectable GEPA stubs. ```bash diff --git a/docs/api/fullmap.md b/docs/api/fullmap.md index 94cf8f9..1350581 100644 --- a/docs/api/fullmap.md +++ b/docs/api/fullmap.md @@ -1,6 +1,6 @@ # Entity Resolution (fullmap) -The `fullmap` module resolves free-text strings to standardized biological CURIEs against the embedded redb database — call `resolve()` for low-level, LazyFrame-based entity resolution inside a pipeline. +The `fullmap` module resolves free-text strings to standardized biological CURIEs against the embedded redb database: call `resolve()` for low-level, LazyFrame-based entity resolution inside a pipeline. ## resolve() @@ -37,7 +37,7 @@ Column name containing text strings to resolve. **`db: Path`** -Path to the fullmap redb file (already resolved — see `fullmap_db_path()` and [Fullmap](../fullmap.md)). +Path to the fullmap redb file (already resolved; see `fullmap_db_path()` and [Fullmap](../fullmap.md)). **`taxon: Optional[str]`** @@ -74,8 +74,8 @@ Controls category-frequency tie-breaking when multiple matches exist for a term. Suffix appended to `col` to locate the `level_two` output column. `resolve()` expects the LazyFrame to already have two NLP columns applied upstream: -- `col` — the `level_one` output (whitespace stripped, lowercased) -- `col + tag` — the `level_two` output (non-word characters removed via `\W+`) +- `col`: the `level_one` output (whitespace stripped, lowercased) +- `col + tag`: the `level_two` output (non-word characters removed via `\W+`) The default `"_two"` matches `level_two`'s default tag. @@ -188,7 +188,7 @@ Rows without a valid CURIE are filtered from the returned frame. ### Provenance Tracking -Every resolved entity carries its source database, source version (snapshot date), and the matched synonym that triggered the match — enabling auditing and quality control. Case is handled by the NLP levels above: `level_one` matches any case variant, `level_two` further strips punctuation for hyphenated or slash-delimited names. +Every resolved entity carries its source database, source version (snapshot date), and the matched synonym that triggered the match, enabling auditing and quality control. Case is handled by the NLP levels above: `level_one` matches any case variant, `level_two` further strips punctuation for hyphenated or slash-delimited names. ## Integration with QC diff --git a/docs/api/lib.md b/docs/api/lib.md index 2039204..c0af06d 100644 --- a/docs/api/lib.md +++ b/docs/api/lib.md @@ -1,6 +1,6 @@ # Batch Resolution (lib) -The `lib` module exposes `resolve_many()`, a high-level convenience function that batch-resolves an iterable of entity strings to CURIEs — use it in scripts and notebooks when you want results without building LazyFrames or running NLP preprocessing yourself. It wraps the lower-level [`resolve()`](fullmap.md) pipeline (normalization, fullmap lookup, resolution, optional QC audit when `qc=True`) and returns a plain Python list of row dictionaries. +The `lib` module exposes `resolve_many()`, a high-level convenience function that batch-resolves an iterable of entity strings to CURIEs; use it in scripts and notebooks when you want results without building LazyFrames or running NLP preprocessing yourself. It wraps the lower-level [`resolve()`](fullmap.md) pipeline (normalization, fullmap lookup, resolution, optional QC audit when `qc=True`) and returns a plain Python list of row dictionaries. ## resolve_many() @@ -31,13 +31,13 @@ For example, if `col="gene"`, each returned row dictionary will contain keys lik **`entities: Iterable[str]`** -An iterable of text strings to resolve. Each string is treated as a candidate entity name that will be normalized and matched against the fullmap synonym database. Accepts any iterable — lists, tuples, generators, sets, etc. +An iterable of text strings to resolve. Each string is treated as a candidate entity name that will be normalized and matched against the fullmap synonym database. Accepts any iterable: lists, tuples, generators, sets, etc. Examples: `["TP53", "BRCA1", "EGFR"]`, `("aspirin", "ibuprofen")`, or a generator expression. **`fullmap: Path`** -Filesystem path to the fullmap redb file, or a base directory containing it (resolved via `fullmap_db_path()` — see [Fullmap](../fullmap.md)). +Filesystem path to the fullmap redb file, or a base directory containing it (resolved via `fullmap_db_path()`; see [Fullmap](../fullmap.md)). **`taxon: Optional[str]` (default: `None`)** @@ -61,7 +61,7 @@ Example: `[Categories.GENE]` prevents gene mappings from appearing in the output Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the deduplication stage adds a category-frequency score (computed in Polars after the SQL query) and prefers the category that appears most frequently across all matched terms in the batch. When `False`, frequency-based tie-breaking is disabled. -This is useful when resolving a column of related entities (e.g., all genes) — the shared context helps disambiguate terms that map to multiple categories. +This is useful when resolving a column of related entities (e.g., all genes): the shared context helps disambiguate terms that map to multiple categories. **`qc: bool` (default: `False`)** @@ -69,7 +69,7 @@ When `True`, runs the QC audit stage after entity resolution. The QC pipeline va ### Return Value -Returns a `list[dict[str, Any]]` — one dictionary per resolved entity. The list is produced by calling `polars.DataFrame.to_dicts()` on the collected resolution output. +Returns a `list[dict[str, Any]]`: one dictionary per resolved entity. The list is produced by calling `polars.DataFrame.to_dicts()` on the collected resolution output. Each dictionary contains the following keys (where `{col}` is the value of the `col` parameter): @@ -88,7 +88,7 @@ Each dictionary contains the following keys (where `{col}` is the value of the ` ### Pipeline Internals -Internally: wrap the iterable in a single-column LazyFrame; snapshot the raw input to `original_{col}` (returned) and `{col}_pre_resolution` (internal, dropped — mirrors edge output); apply `level_one`/`level_two`; resolve the fullmap path via `fullmap_db_path()`; delegate to `fullmap.resolve()`; optionally run `fullmap_audit()` when `qc=True`; collect and `to_dicts()`. +Internally: wrap the iterable in a single-column LazyFrame; snapshot the raw input to `original_{col}` (returned) and `{col}_pre_resolution` (internal, dropped; mirrors edge output); apply `level_one`/`level_two`; resolve the fullmap path via `fullmap_db_path()`; delegate to `fullmap.resolve()`; optionally run `fullmap_audit()` when `qc=True`; collect and `to_dicts()`. ### Example Usage @@ -152,7 +152,7 @@ for row in result: | **Context params** | `column_context` exposed; `section_hash`, `config_file`, `tag` not exposed | Fully configurable | | **Use case** | Standalone batch lookups, scripting, notebooks | Internal pipeline integration | -`resolve_many()` is designed for ad-hoc and programmatic use — scripts, notebooks, and one-off lookups. For pipeline integration where you need full control over logging, context metadata, and lazy evaluation, use `resolve()` directly. +`resolve_many()` is designed for ad-hoc and programmatic use: scripts, notebooks, and one-off lookups. For pipeline integration where you need full control over logging, context metadata, and lazy evaluation, use `resolve()` directly. ### NLP Processing @@ -170,6 +170,6 @@ for row in result: ## Next Steps -- **[Entity Resolution](fullmap.md)** — Lower-level `resolve()` function details -- **[Quality Control](qc.md)** — Multi-stage validation of resolved entities -- **[Configuration](../configuration/table.md)** — YAML-driven entity resolution settings +- **[Entity Resolution](fullmap.md)** - Lower-level `resolve()` function details +- **[Quality Control](qc.md)** - Multi-stage validation of resolved entities +- **[Configuration](../configuration/table.md)** - YAML-driven entity resolution settings diff --git a/docs/api/qc.md b/docs/api/qc.md index 4414d0c..1ec8583 100644 --- a/docs/api/qc.md +++ b/docs/api/qc.md @@ -1,10 +1,10 @@ # Quality Control (qc) -The `qc` module validates entity-resolution mappings through a four-stage pipeline (exact, fuzzy, abbreviation expansion, SapBERT semantic similarity) — it runs behind `build-kg --qc` and `resolve_many(qc=True)` to keep only high-confidence assertions. +The `qc` module validates entity-resolution mappings through a four-stage pipeline (exact, fuzzy, abbreviation expansion, SapBERT semantic similarity); it runs behind `build-kg --qc` and `resolve_many(qc=True)` to keep only high-confidence assertions. -QC runtime support is optional. Install `tablassert[qc]` to enable it — the extra pulls `scikit-learn` and `sentence-transformers` (`torch` and `numpy` arrive transitively); `rapidfuzz` is a core dependency and is always available. +QC runtime support is optional. Install `tablassert[qc]` to enable it: the extra pulls `scikit-learn` and `sentence-transformers` (`torch` and `numpy` arrive transitively); `rapidfuzz` is a core dependency and is always available. -`fullmap_audit()` checks the whole extra before it does any work and raises `QcRuntimeMissingError` naming every absent package and the install command. Checking up front matters because the two packages are needed at different stages — `scikit-learn` from the start, `sentence-transformers` only if Stage 4 is reached — so a half-installed extra would otherwise fail after the audit had already run. `build-kg --qc` performs the same check before the build begins, since the audit does not run until the very end of the build. +`fullmap_audit()` checks the whole extra before it does any work and raises `QcRuntimeMissingError` naming every absent package and the install command. Checking up front matters because the two packages are needed at different stages (`scikit-learn` from the start, `sentence-transformers` only if Stage 4 is reached), so a half-installed extra would otherwise fail after the audit had already run. `build-kg --qc` performs the same check before the build begins, since the audit does not run until the very end of the build. ## fullmap_audit() @@ -61,7 +61,7 @@ Context fields used in QC failure logs for traceability. Returns a Polars LazyFrame containing only the rows whose `col` value (CURIE) has **at least one** passing pre-resolution/preferred-name pair. -QC scores unique `(CURIE, pre_resolution, preferred_name)` pairs, but the result is joined back to the input via a **semi-join on the CURIE column** (`df.join(passed.select(col), on=col, how="semi")`). The retention granularity is therefore the CURIE, not the individual pair: if *any* pair for a CURIE passes any stage, *every* input row sharing that CURIE is kept — including rows that were themselves part of a failed pair. A CURIE (and thus all of its rows) is dropped only when *none* of its pairs pass any stage. Failed pairs are logged with section/config/column context and their fuzzy/SapBERT scores. +QC scores unique `(CURIE, pre_resolution, preferred_name)` pairs, but the result is joined back to the input via a **semi-join on the CURIE column** (`df.join(passed.select(col), on=col, how="semi")`). The retention granularity is therefore the CURIE, not the individual pair: if *any* pair for a CURIE passes any stage, *every* input row sharing that CURIE is kept, including rows that were themselves part of a failed pair. A CURIE (and thus all of its rows) is dropped only when *none* of its pairs pass any stage. Failed pairs are logged with section/config/column context and their fuzzy/SapBERT scores. ### Four-Stage Pipeline @@ -107,7 +107,7 @@ or fuzz.partial_token_sort_ratio(original, preferred) >= 80 _is_abbrev(original, preferred_name) or _is_abbrev(preferred_name, original) ``` -The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss — `AML` ↔ `acute myeloid leukemia`. +The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss: `AML` ↔ `acute myeloid leukemia`. **Performance:** O(n) character scans per row, no model inference. @@ -131,7 +131,7 @@ return similarity >= qc.SIMILARITY_THRESHOLD # 0.5 **Model:** `cambridgeltl/SapBERT-from-PubMedBERT-fulltext` -**Backend:** [sentence-transformers](https://www.sbert.net/) (PyTorch). Embeddings are compared with scikit-learn's `cosine_similarity`. SapBERT's self-alignment pretraining pulls UMLS synonym pairs together in embedding space, which fits this stage's task — deciding whether two names denote the same entity — better than the NLI/STS-trained BioBERT it replaced. The 0.5 threshold was carried over from that BioBERT gate and has not been re-tuned for SapBERT's score distribution. +**Backend:** [sentence-transformers](https://www.sbert.net/) (PyTorch). Embeddings are compared with scikit-learn's `cosine_similarity`. SapBERT's self-alignment pretraining pulls UMLS synonym pairs together in embedding space, which fits this stage's task, deciding whether two names denote the same entity, better than the NLI/STS-trained BioBERT it replaced. The 0.5 threshold was carried over from that BioBERT gate and has not been re-tuned for SapBERT's score distribution. **Lazy-loaded** on the first `fullmap_audit()` call that reaches the embedding stage via `get_sapbert()`, then cached globally for the lifetime of the process. @@ -191,7 +191,7 @@ Output: 990 rows (700 + 250 + 10 + 30) ### Rejection Logging -When `log=True`, each rejected CURIE is logged at INFO level with its context and the scores that caused the rejection: `curie`, `original`, `preferred`, `col`, `fuzz` (partial token sort ratio), `config`, `hash`, and — when the SapBERT stage ran — `sapbert` (cosine similarity). +When `log=True`, each rejected CURIE is logged at INFO level with its context and the scores that caused the rejection: `curie`, `original`, `preferred`, `col`, `fuzz` (partial token sort ratio), `config`, `hash`, and, when the SapBERT stage ran, `sapbert` (cosine similarity). ### Integration with Pipeline diff --git a/docs/api/utils.md b/docs/api/utils.md index 6edc5d7..1eb2918 100644 --- a/docs/api/utils.md +++ b/docs/api/utils.md @@ -1,14 +1,14 @@ # Utilities (utils) -The `tablassert.utils` module provides the shared working-directory constants and a compact hashing helper used throughout the CLI — import it for deterministic section hashes and the `.tablassert/` artifact layout. Deterministic UUID generation for KGX edge identifiers lives in the Rust extension (`tablassert.rs`) and is documented below as well. +The `tablassert.utils` module provides the shared working-directory constants and a compact hashing helper used throughout the CLI; import it for deterministic section hashes and the `.tablassert/` artifact layout. Deterministic UUID generation for KGX edge identifiers lives in the Rust extension (`tablassert.rs`) and is documented below as well. ## Constants -**`BASE: Path`** — `Path("./.tablassert")` +**`BASE: Path`**: `Path("./.tablassert")` The single parent working directory. All runtime artifacts live beneath it. -**`STORE: Path`** — `BASE / "store"` +**`STORE: Path`**: `BASE / "store"` Intermediate parquet storage for compiled subgraphs (`.tablassert/store/`). Created on import. diff --git a/docs/cli.md b/docs/cli.md index 81fa146..1a7c791 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,8 +1,8 @@ # CLI Reference Tablassert extracts knowledge assertions from tabular data into KGX NDJSON. The `tablassert` app -exposes **six subcommands** — `agent`, `build-fullmap`, `build-kg`, `convert-legacy`, `validate`, -and `validate-kgx` — plus an app-level `--version` flag. Run `tablassert --help` (or ` --help`) +exposes **six subcommands**: `agent`, `build-fullmap`, `build-kg`, `convert-legacy`, `validate`, +and `validate-kgx`, plus an app-level `--version` flag. Run `tablassert --help` (or ` --help`) for the live surface. ## Command index @@ -38,7 +38,7 @@ Use this to autonomously turn one or more PMC articles into audited, improved KG (`pip install "tablassert[agent]"`); `--optimize` additionally needs the `[optimize]` extra (`pip install "tablassert[optimize]"`, pulls `dspy`). Both are checked after flag validation and before any model is built or article fetched, so a missing extra is reported with its install -command instead of surfacing mid-run — see [When an extra is missing](installation.md#when-an-extra-is-missing). +command instead of surfacing mid-run; see [When an extra is missing](installation.md#when-an-extra-is-missing). ```bash tablassert agent PMC-IDS... --configuration-file GRAPH.yaml [OPTIONS] @@ -52,8 +52,8 @@ page lists the flags; see | Option | Type | Required | Default | Description | | --- | --- | --- | --- | --- | -| `PMC-IDS` (`--pmc-ids`) | list[str] | Yes | — | One or more PMC article ids (positional) | -| `--configuration-file`, `-f` | Path | Yes | — | Caller-owned Graph YAML; supplies build metadata/fullmap and receives successful absolute table entries | +| `PMC-IDS` (`--pmc-ids`) | list[str] | Yes | n/a | One or more PMC article ids (positional) | +| `--configuration-file`, `-f` | Path | Yes | n/a | Caller-owned Graph YAML; supplies build metadata/fullmap and receives successful absolute table entries | | `--model-id`, `-m` | str | No | `None` | Model id (env `TABLASSERT_AGENT_MODEL_ID`) | | `--api-base`, `-ab` | str | No | `None` | OpenAI-compatible base URL (env `TABLASSERT_AGENT_API_BASE`) | | `--api-key`, `-ak` | str | No | `None` | API key secret (env `TABLASSERT_AGENT_API_KEY`) | @@ -73,7 +73,7 @@ page lists the flags; see | `--max-metric-calls` | int | No | `8` | GEPA metric-call budget for `--optimize` | | `--dataset` | Path | No | `None` | YAML/JSON list of `{table_summary, coverage_feedback}` examples for `--optimize` (an example may also carry `fullmap`, `workdir`, and `head` to score each proposed config with real coverage) | | `--task-model` | str | No | `None` | Fast model id for GEPA's many program evaluations (cheap task LM + strong reflection LM); `--model-id` is the reflection LM. Defaults to the reflection LM | -| `--gepa-threads` | int | No | `None` | Thread count for GEPA's evaluation pool (`--optimize`) — parallelizes candidate LM forward passes only; coverage-scoring builds stay serialized on `_GEPA_BUILD_LOCK` | +| `--gepa-threads` | int | No | `None` | Thread count for GEPA's evaluation pool (`--optimize`): parallelizes candidate LM forward passes only; coverage-scoring builds stay serialized on `_GEPA_BUILD_LOCK` | ```bash tablassert agent PMC11708054 --configuration-file ./graph.yaml @@ -83,7 +83,7 @@ tablassert agent PMC11708054 -f ./graph.yaml !!! warning "Secrets" Model config comes from the flags above **or** the `TABLASSERT_AGENT_*` environment variables - (explicit flags win). Secrets are **never** hardcoded or defaulted — a missing value fails loud + (explicit flags win). Secrets are **never** hardcoded or defaulted: a missing value fails loud (exit 2) **before** any model is built. --- @@ -119,7 +119,7 @@ tablassert build-fullmap --aria2c --output /data/fullmap/fullmap.redb By default `build-fullmap` looks for a prebuilt `fullmap.tar.zst` at `https://stars.renci.org/var/babel_outputs//fullmap//` (the version directory is the **installed Tablassert package version**, never hardcoded), verifies it against the -published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension — streaming zstd → +published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming zstd → tar with the GIL released (the decompressed tar never touches disk), then validating the extracted primary + shards against the force-build contract (exact `v5` schema, a recorded `build_id`, the exact shard set, and per-shard `build_id` equality) before atomically renaming them into place. If no @@ -144,12 +144,12 @@ The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is | Option | Type | Required | Default | Description | | --- | --- | --- | --- | --- | -| `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | — | Graph YAML | +| `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Graph YAML | | `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `biolink:not_significant` edges before resolution) | -| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact → fuzzy → abbreviation → SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON — no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), and no stray whitespace — verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies — and fails the build (non-zero exit) on any violation | +| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact → fuzzy → abbreviation → SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), and no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies) and fails the build (non-zero exit) on any violation | | `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging | | `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: ≤5 random rows/section, cached to `.head.parquet`, never clobbers a full build | -| `--threads`, `-t` | int | No | `None` (auto) | Worker threads for the parallel fullmap reads behind entity resolution. Readers fan out across the 16 record-shard files, and values above the (non-empty) shard count further split the busiest shards' term buckets across more concurrent readers of the same shard — redb readers share-lock, so they never contend with each other. Unset keeps the auto behavior: large batches (≥ 1024 terms) fan out, small ones stay serial. Results are identical at any worker count | +| `--threads`, `-t` | int | No | `None` (auto) | Worker threads for the parallel fullmap reads behind entity resolution. Readers fan out across the 16 record-shard files, and values above the (non-empty) shard count further split the busiest shards' term buckets across more concurrent readers of the same shard; redb readers share-lock, so they never contend with each other. Unset keeps the auto behavior: large batches (≥ 1024 terms) fan out, small ones stay serial. Results are identical at any worker count | ```bash tablassert build-kg graph.yaml --qc --log @@ -157,13 +157,13 @@ tablassert build-kg graph.yaml --qc --log Output is written to `rig.artifact_base_path` (created when missing) as `{name}_{version}.nodes.ndjson`, `{name}_{version}.edges.ndjson`, and `{name}_{version}.RIG.yaml`; intermediate parquet lands in -`.tablassert/store/`. The RIG document is audited in memory before it is written — an invalid +`.tablassert/store/`. The RIG document is audited in memory before it is written: an invalid or incomplete RIG fails the build with `[rig-validation-failed]` and nothing is emitted. See [Graph Configuration](configuration/graph.md). ??? info "Build progress & stages" - The build runs six parallel stages — Loading Tables → Extracting Sections → Building TCode → - Collecting Instructions → Building Subgraphs → Compiling Graph — plus a seventh, Studying Graph, + The build runs six parallel stages (Loading Tables → Extracting Sections → Building TCode → + Collecting Instructions → Building Subgraphs → Compiling Graph) plus a seventh, Studying Graph, only when `--qc` is passed. They run under a three-row live progress block (stage header; section bar with count/elapsed/ETA; in-flight item detail). Each completed stage prints a green `✓ Stage N · NAME · elapsed` line above the live block. During Building @@ -179,7 +179,7 @@ Use this to migrate a legacy table config (or a whole directory of them) into th exact-duplicate entries the template/section overlay produced are dropped and same-key `qualifiers` entries merged into one per key, the removed `relationship_strength` annotation is renamed to `effect_size`, every `source.reindex` entry is validated against the v12 model, -and each `source.local` is resolved onto the real downloaded payload — never left pointing at +and each `source.local` is resolved onto the real downloaded payload, never left pointing at a stale `./DATALAKE` path. ```bash @@ -187,28 +187,28 @@ tablassert convert-legacy LEGACY-PATH [ARGS] ``` `LEGACY-PATH` is a single legacy YAML file or a directory of them (non-recursive). Each converted -config is written as `.v12.yaml` — beside its input by default, or under `--out` when given. +config is written as `.v12.yaml`, beside its input by default, or under `--out` when given. | Option | Type | Required | Default | Description | | --- | --- | --- | --- | --- | -| `LEGACY-PATH` | Path | Yes | — | Legacy YAML file, or a directory of `*.yaml` legacy configs | +| `LEGACY-PATH` | Path | Yes | n/a | Legacy YAML file, or a directory of `*.yaml` legacy configs | | `--downloads`, `-d` | Path | No | `None` | Directory holding the downloaded article payloads (`PMC/PMC./...`) that each `source.local` is resolved against: the `local` basename is tried first, then each `source.url` entry's basename (the urls hold the real payload filenames), recursively, preferring a hit under the section's own publication directory; an unresolved source lists every basename tried | | `--fetch` | Flag | No | `False` | When no local payload match exists, download the article from PMC open access (via `provenance.publication`) into `--downloads` instead of failing the source as unresolved | | `--out`, `-o` | Path | No | `None` (beside each input) | Directory the `.v12.yaml` outputs are written into (created when missing); defaults to each input's own directory | -Directory mode converts every `*.yaml` file — skipping `*.v12.yaml` outputs from earlier runs, so -a rerun never re-converts its own output — and prints one status line per file: +Directory mode converts every `*.yaml` file (skipping `*.v12.yaml` outputs from earlier runs, so +a rerun never re-converts its own output) and prints one status line per file: `CONVERTED -> ` or `FAILED ()`. One failure never aborts the batch, but the command exits non-zero when ANY file failed (the full coded message for every failure is printed on stderr). `--out` is optional in directory mode too: without it each `.v12.yaml` -lands beside its input. The glob is `*.yaml` ONLY — `.yml` files and every other extension are +lands beside its input. The glob is `*.yaml` ONLY: `.yml` files and every other extension are never picked up. -Exit codes: `0` everything converted; `1` any conversion failed; `2` usage error — a missing input +Exit codes: `0` everything converted; `1` any conversion failed; `2` usage error (a missing input or `--downloads` path, an `--out` that is not a directory, or a directory holding no `*.yaml` -files. Conversion is all-or-nothing per file: a construct v12 cannot express fails the file with +files). Conversion is all-or-nothing per file: a construct v12 cannot express fails the file with `legacy-unsupported-syntax`, and an unresolvable `source.local` fails it with -`legacy-source-unresolved` — the same coded error `--fetch` ends in when the network is down or +`legacy-source-unresolved`, the same coded error `--fetch` ends in when the network is down or the article is not open access (a bounded fetch, never a hang). ```bash @@ -227,7 +227,7 @@ The MOKG corpus (26 legacy table configs) has an executable ingestability accept `TABLASSERT_MOKG_DIR`. It skips with a printed reason when the variable is unset; when set, every corpus file must either convert and validate against the downloads directory (`TABLASSERT_MOKG_DOWNLOADS`, recursive basename match with the `source.url` basenames as -fallback) or fail loudly with exactly `legacy-source-unresolved` — no other error class, no +fallback) or fail loudly with exactly `legacy-source-unresolved`: no other error class, no silent skip, and at least 15 of the 26 must convert (with the full downloads tree below, 24 convert; only QIN9 and WAINBERG3 stay unresolved because their payloads are absent): @@ -240,7 +240,7 @@ uv run pytest tests/test_legacy.py -q -k corpus For the remaining files, run the batch with `--fetch` so every open-access article payload missing from the downloads directory is downloaded from PMC open access first (network required); `--out` keeps the outputs out of the curated corpus directory. QIN9 stays -unresolved either way — its payload is hosted on figshare, not in the PMC article bundle: +unresolved either way: its payload is hosted on figshare, not in the PMC article bundle: ```bash tablassert convert-legacy /home/skyeav/Code/ISB/TableConfigs/TABLE/MOKG \ @@ -252,7 +252,7 @@ tablassert convert-legacy /home/skyeav/Code/ISB/TableConfigs/TABLE/MOKG \ ## validate -Use this to validate a configuration against a schema without running the build — ideal for CI and +Use this to validate a configuration against a schema without running the build, ideal for CI and pre-commit hooks. The required `--schema` flag selects which schema to validate against (the kind is no longer sniffed from the YAML). @@ -263,8 +263,8 @@ tablassert validate -f CONFIGURATION-FILE --schema table | Option | Type | Required | Default | Description | | --- | --- | --- | --- | --- | -| `CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | — | Configuration file to validate | -| `--schema`, `-s` | `graph` \| `table` | Yes | — | Schema to validate against: `graph` validates the `Graph` model **and** every referenced table; `table` validates section syntax only | +| `CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Configuration file to validate | +| `--schema`, `-s` | `graph` \| `table` | Yes | n/a | Schema to validate against: `graph` validates the `Graph` model **and** every referenced table; `table` validates section syntax only | Exits non-zero on any schema error. See [Table Configuration](configuration/table.md) and [Graph Configuration](configuration/graph.md). @@ -280,7 +280,7 @@ tablassert validate graph.yaml --schema graph Use this to check that a completed build is actually Biolink-compliant. Where [`validate`](#validate) checks your *configuration*, `validate-kgx` checks the *output*: every node -and edge is constructed as the Biolink Pydantic class named by its own `category` — the same classes +and edge is constructed as the Biolink Pydantic class named by its own `category`, the same classes [`NCATSTranslator/translator-ingests`](https://github.com/NCATSTranslator/translator-ingests) builds when it ingests your files. @@ -290,8 +290,8 @@ tablassert validate-kgx --nodes MY_KG_1.0.0.nodes.ndjson --edges MY_KG_1.0.0.edg | Option | Type | Required | Default | Description | | --- | --- | --- | --- | --- | -| `--nodes`, `-n` | Path | Yes | — | Built `*.nodes.ndjson` file to validate | -| `--edges`, `-e` | Path | Yes | — | Built `*.edges.ndjson` file to validate | +| `--nodes`, `-n` | Path | Yes | n/a | Built `*.nodes.ndjson` file to validate | +| `--edges`, `-e` | Path | Yes | n/a | Built `*.edges.ndjson` file to validate | | `--limit` | int | No | `20` | Maximum example failures to retain per file | Failures are grouped by field and error type, so a systematic modelling problem shows up as one line @@ -305,7 +305,7 @@ KGX output is Biolink-compliant. ``` Exits non-zero when any record fails, so it can gate a release in CI. A missing or misspelled path is -reported as `file not found` and also exits non-zero — a file that was never read must never count as +reported as `file not found` and also exits non-zero: a file that was never read must never count as a pass. Edges carrying `effect_size` / `effect_type` are reported invalid until a `biolink-model` release @@ -325,14 +325,14 @@ The strict count is what `ok` and the exit code use; the pending count is what ## Typical workflow 1. Author a table config, then a graph config that references it. -2. `tablassert validate graph.yaml --schema graph` — fail fast on schema errors. -3. `tablassert build-kg graph.yaml` — produce KGX NDJSON + RIG (add `--qc` to audit mappings). -4. `tablassert validate-kgx -n MY_KG_1.0.0.nodes.ndjson -e MY_KG_1.0.0.edges.ndjson` — confirm the +2. `tablassert validate graph.yaml --schema graph`: fail fast on schema errors. +3. `tablassert build-kg graph.yaml`: produce KGX NDJSON + RIG (add `--qc` to audit mappings). +4. `tablassert validate-kgx -n MY_KG_1.0.0.nodes.ndjson -e MY_KG_1.0.0.edges.ndjson`: confirm the output validates against the Biolink Model before shipping it downstream. ## Next Steps -- **[Tutorial](tutorial.md)** — complete example walkthrough -- **[Configuration Guide](configuration/graph.md)** — YAML configuration reference -- **[Fullmap](fullmap.md)** — entity-resolution database build and schema -- **[Agent](agent.md)** — autonomous PMC → KG pipeline depth +- **[Tutorial](tutorial.md)**: complete example walkthrough +- **[Configuration Guide](configuration/graph.md)**: YAML configuration reference +- **[Fullmap](fullmap.md)**: entity-resolution database build and schema +- **[Agent](agent.md)**: autonomous PMC → KG pipeline depth diff --git a/docs/configuration/advanced-example.md b/docs/configuration/advanced-example.md index d051c6f..f8c0447 100644 --- a/docs/configuration/advanced-example.md +++ b/docs/configuration/advanced-example.md @@ -1,8 +1,8 @@ # Advanced Example: Real-World Configuration A fully-annotated real-world table configuration (ALAMV6.yaml) showing complex regex, taxonomic -filtering, and statistical annotations working together. **Source:** microbiome–chemical correlation -analysis from PMC11708054 — an Excel file from which we extract correlations between gut microbiota +filtering, and statistical annotations working together. **Source:** microbiome-chemical correlation +analysis from PMC11708054, an Excel file from which we extract correlations between gut microbiota and tamoxifen metabolites. ## Full Configuration @@ -66,22 +66,22 @@ template: encoding: Correlation analysis between microbial composition and 13C-tamoxifen abundance after FDR correction ``` -`miscellaneous_notes` is a freetext escape hatch — use `method: value` for a constant note across the +`miscellaneous_notes` is a freetext escape hatch: use `method: value` for a constant note across the whole table or `method: column` to pull per-row notes from the source (see [allow-list and auto-folding](table.md#allow-list-and-auto-folding)). ## Key Techniques -- **Excel column letters** — `encoding: A`/`B`/`C` reference the first/second/third columns of the +- **Excel column letters**: `encoding: A`/`B`/`C` reference the first/second/third columns of the headerless source (organism names, Spearman rho, p-value). -- **Regex pipeline** — the subject runs three substitutions in order: `.*g__` → `` +- **Regex pipeline**: the subject runs three substitutions in order: `.*g__` → `` (`d__Bacteria;p__Firmicutes;g__Lactobacillus` → `Lactobacillus`), then `;s__` → ` ` (`Lactobacillus;s__rhamnosus` → `Lactobacillus rhamnosus`), then `sp` → `sp. `. -- **Taxonomic filtering** — `prioritize: [OrganismTaxon]` + `avoid: [Gene]` stop "Lactobacillus" +- **Taxonomic filtering**: `prioritize: [OrganismTaxon]` + `avoid: [Gene]` stop "Lactobacillus" resolving to a similarly-named gene. -- **Mixed annotations** — `method: value` for constants (same every row), `method: column` for +- **Mixed annotations**: `method: value` for constants (same every row), `method: column` for per-row values. -- **Subject-predicate-object** — subject varies per row (column), predicate `correlated_with` is +- **Subject-predicate-object**: subject varies per row (column), predicate `correlated_with` is fixed, object `CHEBI:41774` is fixed → `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`. ??? note "Regex dialect constraint" @@ -101,8 +101,8 @@ whole table or `method: column` to pull per-row notes from the source (see **Edges:** Allow-listed annotation columns (`supporting_study_size`, `p_value`, `effect_size`, `effect_type`) stay as -top-level edge fields (numeric annotations as controlled-notation strings). Any non-Biolink-slot name — -here `assertion_method`, `multiple_testing_correction_method`, `miscellaneous_notes` — folds into the +top-level edge fields (numeric annotations as controlled-notation strings). Any non-Biolink-slot name +(here `assertion_method`, `multiple_testing_correction_method`, `miscellaneous_notes`) folds into the edge's `supporting_text` list as `"name: value"` entries (sorted alphabetically), alongside the built-in `extracted_from_row_number`: @@ -208,7 +208,7 @@ template: ``` Each column-mapped node gets its own `prioritize` list to guide disambiguation. `remove` strips each -listed pattern (replace with empty string); `regex` applies an ordered `pattern`→`replacement` list — +listed pattern (replace with empty string); `regex` applies an ordered `pattern`→`replacement` list; both transform cell text in place before resolution, and neither drops rows. --- @@ -217,7 +217,7 @@ both transform cell text in place before resolution, and neither drops rows. Wide tables where each column encodes a different object (e.g., 24 metabolite columns for the same microbe rows). Sections inherit the template's `source`, `provenance`, and `subject`, overriding only -the `object` (and optionally `row_slice`) per section — one section entry per metabolite column keeps a +the `object` (and optionally `row_slice`) per section: one section entry per metabolite column keeps a 24-metabolite config out of 24 separate files. ```yaml diff --git a/docs/configuration/graph.md b/docs/configuration/graph.md index 885c6d3..8f8e89d 100644 --- a/docs/configuration/graph.md +++ b/docs/configuration/graph.md @@ -1,6 +1,6 @@ # Graph Configuration Reference -Graph configurations orchestrate one or more [table configurations](table.md) into a single knowledge-graph build — author one to produce KGX output with `tablassert build-kg` (see the [CLI reference](../cli.md#build-kg)). To check a single table config on its own, use `tablassert validate --schema table`. +Graph configurations orchestrate one or more [table configurations](table.md) into a single knowledge-graph build: author one to produce KGX output with `tablassert build-kg` (see the [CLI reference](../cli.md#build-kg)). To check a single table config on its own, use `tablassert validate --schema table`. ## Purpose @@ -11,7 +11,7 @@ A graph configuration file specifies: - Database location for entity resolution - The **required `rig:` section**: all Resource Ingest Guide (RIG) metadata emitted as `_.RIG.yaml` -QC auditing and verbose logging are controlled at build time via the `build-kg --qc` and `build-kg --log` flags — they are **not** graph-config fields. +QC auditing and verbose logging are controlled at build time via the `build-kg --qc` and `build-kg --log` flags: they are **not** graph-config fields. ## Schema @@ -29,7 +29,7 @@ The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation ### The `rig:` section -The `rig:` section carries every human-authored RIG fact. Its shape mirrors the released [RIG schema](https://github.com/biolink/resource-ingest-guide-schema), so the generated `.RIG.yaml` is always schema-shaped. The generator derives only mechanical facts from the build (generated artifact file entries, observed edge/node type summaries) and **validates the complete document before writing anything** — a build never leaves behind an invalid or incomplete RIG. +The `rig:` section carries every human-authored RIG fact. Its shape mirrors the released [RIG schema](https://github.com/biolink/resource-ingest-guide-schema), so the generated `.RIG.yaml` is always schema-shaped. The generator derives only mechanical facts from the build (generated artifact file entries, observed edge/node type summaries) and **validates the complete document before writing anything**, so a build never leaves behind an invalid or incomplete RIG. | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -66,7 +66,7 @@ The `rig:` section carries every human-authored RIG fact. Its shape mirrors the | `ingest_categories` | List[Enum] | No | Defaults to `[translator_knowledge_creator]`; also `primary_knowledge_provider`, `aggregation_provider`, `aggregation_interpreter`, `supporting_data_provider`, `ontology_provider`, `node_property_only_provider`, `other` | | `utility` | String | Yes | Why the source is ingested and its utility for Translator use cases | | `scope` | String | Yes | High-level narrative of what is included and excluded | -| `relevant_files` | List[Object] | No | **Upstream** source files: `file_name`, `location` (URL), optional `description`. Entries are cross-checked against the table configs' source URLs/local files — an entry matching no configured source fails the build | +| `relevant_files` | List[Object] | No | **Upstream** source files: `file_name`, `location` (URL), optional `description`. Entries are cross-checked against the table configs' source URLs/local files, and an entry matching no configured source fails the build | | `included_content` | List[Object] | No | Upstream `file_name` / `included_records` / optional `fields_used` entries | | `filtered_content` | List[Object] | No | `file_name` / `filtered_records` / `rationale` entries | | `future_considerations` | List[Object] | No | `category` (`edge_content`, `node_property_content`, `edge_property_content`, `other`), `consideration`, optional `relevant_files` | @@ -87,7 +87,7 @@ Everything under `target_info.edge_type_info` and `target_info.node_type_info` i - **Edge types** (one per observed predicate): subject/object categories resolved from the emitted nodes, list-valued `knowledge_level`/`agent_type`, role-separated `primary_knowledge_sources` / `supporting_data_sources` / `aggregator_knowledge_sources` from each edge's `sources` retrieval provenance, observed `edge_properties`, qualifier shapes (enumerated literal values or identifier prefixes for CURIE-valued qualifiers), and `source_files` taken from the upstream `source_record_urls` (never the output filenames). - **Node types**: observed categories and the identifier prefixes actually emitted (`source_identifier_types`); categories with prefix-less identifiers get a factual free-text entry. -- **UI explanation**: `rig.ui_explanation` (when set) followed by the built-in Tablassert explanation — the default text is always present. +- **UI explanation**: `rig.ui_explanation` (when set) followed by the built-in Tablassert explanation; the default text is always present. ### Built-in RIG validation @@ -211,7 +211,7 @@ When the generated RIG will back a PR to [`NCATSTranslator/translator-ingests`]( - Use an **infores that is registered** (or being registered) in the [information resource registry](https://github.com/biolink/information-resource-registry). - Replace any `file://` artifact base with the **public https location** where the KGX files will be served. - Fill `terms_of_use_info` with the source's actual license/terms assessment, and `data_versioning_and_releases` with how the upstream source releases data. -- Describe upstream source files in `rig.ingest_info.relevant_files` / `included_content` / `filtered_content` — the generator cross-checks them against your table configs but the semantics are yours. +- Describe upstream source files in `rig.ingest_info.relevant_files` / `included_content` / `filtered_content`: the generator cross-checks them against your table configs but the semantics are yours. - Give every edge type's provenance real contributors under `rig.provenance_info.contributions`. ## Next Steps diff --git a/docs/configuration/table.md b/docs/configuration/table.md index d346e45..a830a8f 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -1,6 +1,6 @@ # Table Configuration Reference -Table configurations define how Tablassert transforms tabular data (Excel, CSV, TSV) into knowledge-graph assertions — author one per source table to declare its source, triple mappings, entity-resolution rules, provenance, and optional edge annotations. +Table configurations define how Tablassert transforms tabular data (Excel, CSV, TSV) into knowledge-graph assertions: author one per source table to declare its source, triple mappings, entity-resolution rules, provenance, and optional edge annotations. ## Template vs Sections @@ -89,8 +89,8 @@ Defines the data file location and format. | Field | Type | Required | Description | |-------|------|----------|-------------| | `kind` | String | No | Source kind. Model default is `"excel"`, but specify it explicitly in configs. | -| `local` | Path | Yes | Local file path the source is read from. The file must already exist here — Tablassert does not download it. | -| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only — the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | +| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | +| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. | | `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. | | `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | @@ -107,15 +107,15 @@ source: row_slice: [1, auto] # Start at the second physical row, read to end ``` -> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind` — `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. +> **Specify `kind` explicitly.** Tablassert selects the reader purely from the declared `kind`: `excel` reads a workbook (`sheet`), `text` scans delimited text (`delimiter`); the file on disk is never inspected to infer its format. Because `kind` carries a default, a source whose `kind` is omitted or does not match the actual file is still accepted and fed to the wrong reader, surfacing only later as a read error or garbled rows. Stating `kind` explicitly makes a mis-declared source fail fast. #### Text Source (CSV/TSV) | Field | Type | Required | Description | |-------|------|----------|-------------| | `kind` | String | No | Source kind. Model default is `"text"`, but specify it explicitly in configs. | -| `local` | Path | Yes | Local file path the source is read from. The file must already exist here — Tablassert does not download it. | -| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only — the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | +| `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | +| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `delimiter` | String | No | Field delimiter. Defaults to `","`. | | `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. | | `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | @@ -214,7 +214,7 @@ At runtime those letters are converted internally to Polars column names such as #### `split_by` -**`split_by`** — annotations only; splits each cell of a `method: column` encoding into a real JSON array. +**`split_by`**: annotations only; splits each cell of a `method: column` encoding into a real JSON array. ```yaml annotations: @@ -226,18 +226,18 @@ annotations: The separator is a property of the data, not of the slot. Inspect the table's cells and set `split_by` to the separator the cells actually use: `","` for comma-joined ids like `"EFO:0001,EFO:0002"` above, `";"` for `"EFO:0001;EFO:0002"`, `"|"` only if the cells happen to be pipe-joined. -`split_by` is the one multivalued encoding: every row's cell becomes its own JSON array, so an array that differs per row — the shape a literal can never express — is declared directly. Values are trimmed and blanks dropped; a null cell stays null. +`split_by` is the one multivalued encoding: every row's cell becomes its own JSON array, so an array that differs per row, the shape a literal can never express, is declared directly. Values are trimmed and blanks dropped; a null cell stays null. -Single-value cells need no `split_by` at all: `prune_to_class` wraps a scalar bound for a uniformly multivalued slot into a one-element list, so a lone `EFO:0001` cell already emits as `has_evidence: ["EFO:0001"]`. Reach for it when the cells actually join multiple values. Leave such a column without `split_by` and the joined cell stays a scalar: the same wrapping yields a one-element list holding the whole string — `has_evidence: ["EFO:0001;EFO:0002"]` — structurally valid Biolink that hands consumers one unusable blob instead of two ids. +Single-value cells need no `split_by` at all: `prune_to_class` wraps a scalar bound for a uniformly multivalued slot into a one-element list, so a lone `EFO:0001` cell already emits as `has_evidence: ["EFO:0001"]`. Reach for it when the cells actually join multiple values. Leave such a column without `split_by` and the joined cell stays a scalar: the same wrapping yields a one-element list holding the whole string (`has_evidence: ["EFO:0001;EFO:0002"]`), structurally valid Biolink that hands consumers one unusable blob instead of two ids. -`split_by` requires `method: column` and rejects an empty separator, which would split into individual characters. It is unrelated to the `source.delimiter` CSV/TSV field separator. (The earlier annotation `delimiter` field — unrelated to the `source.delimiter` CSV/TSV separator — was replaced by `split_by`.) +`split_by` requires `method: column` and rejects an empty separator, which would split into individual characters. It is unrelated to the `source.delimiter` CSV/TSV field separator. (The earlier annotation `delimiter` field, unrelated to the `source.delimiter` CSV/TSV separator, was replaced by `split_by`.) -**`split_by` and `explode_by` are the same split, with different destinations.** Both read a delimited cell through one shared primitive — items trimmed, blanks dropped (so `"a;b;"` and `"a;;b"` yield two items, not three), a null cell left null — and then differ only in what they do with the items: +**`split_by` and `explode_by` are the same split, with different destinations.** Both read a delimited cell through one shared primitive, with items trimmed, blanks dropped (so `"a;b;"` and `"a;;b"` yield two items, not three), and a null cell left null; they then differ only in what they do with the items: | | Destination | Use for | |---|---|---| -| `explode_by` | one **row** per item | node encodings — each item is its own entity, producing its own edge | -| `split_by` | one **array** on the row | annotations — the items are one multivalued slot on a single edge | +| `explode_by` | one **row** per item | node encodings: each item is its own entity, producing its own edge | +| `split_by` | one **array** on the row | annotations: the items are one multivalued slot on a single edge | So `explode_by` is not an alternative to `split_by` for a multivalued annotation: it multiplies edges rather than filling one edge's array. @@ -295,7 +295,7 @@ subject: Executed in order. -> **Regex dialect:** Patterns are passed directly to Polars `str.replace_all()`, which uses the Rust [`regex`](https://docs.rs/regex/) crate. Only features supported by that engine work — in particular, **backreferences (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(? **Regex dialect:** Patterns are passed directly to Polars `str.replace_all()`, which uses the Rust [`regex`](https://docs.rs/regex/) crate. Only features supported by that engine work: in particular, **backreferences (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?` when unset) as the primary entry of the Biolink `sources` list on each edge — `{resource_id: "infores:multiomics-kg", resource_role: "primary_knowledge_source", upstream_resource_ids: [...], source_record_urls: [...]}` — with one additional `supporting_data_source` entry per upstream. When `override.upstream_source_record_urls` is set, the primary entry emits no `source_record_urls` and each mapped supporting entry carries its own instead. No flat `primary_knowledge_source` scalar is emitted: current translator-ingests practice carries retrieval provenance only in `sources`, and the Biolink `RetrievalSource` class is where `resource_id` / `upstream_resource_ids` / `source_record_urls` are defined. (Each entry also mirrors `resource_id` into `id` because the generated Biolink classes still require it; that mirror disappears once biolink-model [#1706](https://github.com/biolink/biolink-model/issues/1706) lands.) The override cannot set a per-section primary source; manual infores CURIEs belong in `upstream_resource_ids`. Older flat `resource_id` / `primary_knowledge_source` output has been removed so generated KGX matches the Biolink edge contract. +Tablassert emits the graph-level infores (or `infores:` when unset) as the primary entry of the Biolink `sources` list on each edge, `{resource_id: "infores:multiomics-kg", resource_role: "primary_knowledge_source", upstream_resource_ids: [...], source_record_urls: [...]}`, with one additional `supporting_data_source` entry per upstream. When `override.upstream_source_record_urls` is set, the primary entry emits no `source_record_urls` and each mapped supporting entry carries its own instead. No flat `primary_knowledge_source` scalar is emitted: current translator-ingests practice carries retrieval provenance only in `sources`, and the Biolink `RetrievalSource` class is where `resource_id` / `upstream_resource_ids` / `source_record_urls` are defined. (Each entry also mirrors `resource_id` into `id` because the generated Biolink classes still require it; that mirror disappears once biolink-model [#1706](https://github.com/biolink/biolink-model/issues/1706) lands.) The override cannot set a per-section primary source; manual infores CURIEs belong in `upstream_resource_ids`. Older flat `resource_id` / `primary_knowledge_source` output has been removed so generated KGX matches the Biolink edge contract. ### Annotations @@ -463,7 +463,7 @@ Optional edge attributes (statistical metadata, notes, etc.). | `split_by` | String | No | Separator splitting each cell of a `method: column` encoding into a real JSON array. See [`split_by`](#split_by). | | (inherits Encoding) | | | All Encoding fields available (method, encoding, regex, etc.) | -Multivalued Biolink slots such as `has_evidence` or `FDA_regulatory_approvals` — whose consumers iterate the value — must emit a real JSON array rather than a scalar. [`split_by`](#split_by) is the multivalued encoding: point it at a column whose cells join multiple values, set it to the separator the cells actually use, and each cell's delimited text splits into a per-row array. A single-value column needs no `split_by` — the scalar is wrapped into a one-element array. +Multivalued Biolink slots such as `has_evidence` or `FDA_regulatory_approvals`, whose consumers iterate the value, must emit a real JSON array rather than a scalar. [`split_by`](#split_by) is the multivalued encoding: point it at a column whose cells join multiple values, set it to the separator the cells actually use, and each cell's delimited text splits into a per-row array. A single-value column needs no `split_by`; the scalar is wrapped into a one-element array. **Example:** ```yaml @@ -475,7 +475,7 @@ annotations: - {annotation: has_evidence, method: column, encoding: E, split_by: ","} # cells like "EFO:0001,EFO:0002" -> a per-row JSON array - {annotation: approval_ids, method: column, encoding: F} # Curated pass-through -> emitted verbatim as a scalar (e.g. "011111|022222") - # Descriptive name of your choice — folded into `supporting_text` on output. + # Descriptive name of your choice: folded into `supporting_text` on output. - annotation: log2fc_relative_to_vehicle_control method: value encoding: "Values are log2 fold-change relative to vehicle control; n=3 biological replicates per arm" @@ -485,9 +485,9 @@ annotations: Annotation names fall into three groups at build time: -- **Allowed edge fields** — names on the edge allow-list: [Biolink Association](https://biolink.github.io/biolink-model/) slots, qualifier slots, and curated KGX/Tablassert edge fields (e.g. `p_value`, `adjusted_p_value`, `knowledge_level`, `primary_knowledge_source`, `supporting_text`, `publications`, `effect_size`, `effect_type`, `approval_ids`, qualifier slots like `severity_qualifier` / `disease_context_qualifier`) are written to edges verbatim. `approval_ids` (FDA application numbers, following the DAKP translator-ingest precedent) is deliberately a **scalar pass-through**: a pipe-joined cell such as `011111|022222` is emitted verbatim as its own top-level edge field — no `split_by`, not a JSON array. -- **Unsatisfiable slots** — names the Biolink LinkML schema declares but attaches to **no** Pydantic class: `supporting_study_size`, `sample_size`, `relationship_strength`, `statistical_significance_qualifier`, and the other `supporting_study_*` slots. A record carrying one could never validate, so their values are routed onto the edge's **inlined supporting study** (`has_supporting_studies` → `Study` → `StudyResult`, the COHD/ICEES pattern) rather than emitted as edge fields. Declaring one is legal and loses nothing, but Tablassert emits a `BiolinkRelocationWarning` naming where the value went. This set is derived from the *installed* `biolink-model`, so a slot leaves it automatically once a release attaches it. -- **Tablassert pipeline fields** — `upstream_resource_ids`, `source_record_urls`. +- **Allowed edge fields**: names on the edge allow-list: [Biolink Association](https://biolink.github.io/biolink-model/) slots, qualifier slots, and curated KGX/Tablassert edge fields (e.g. `p_value`, `adjusted_p_value`, `knowledge_level`, `primary_knowledge_source`, `supporting_text`, `publications`, `effect_size`, `effect_type`, `approval_ids`, qualifier slots like `severity_qualifier` / `disease_context_qualifier`) are written to edges verbatim. `approval_ids` (FDA application numbers, following the DAKP translator-ingest precedent) is deliberately a **scalar pass-through**: a pipe-joined cell such as `011111|022222` is emitted verbatim as its own top-level edge field (no `split_by`, not a JSON array). +- **Unsatisfiable slots**: names the Biolink LinkML schema declares but attaches to **no** Pydantic class: `supporting_study_size`, `sample_size`, `relationship_strength`, `statistical_significance_qualifier`, and the other `supporting_study_*` slots. A record carrying one could never validate, so their values are routed onto the edge's **inlined supporting study** (`has_supporting_studies` → `Study` → `StudyResult`, the COHD/ICEES pattern) rather than emitted as edge fields. Declaring one is legal and loses nothing, but Tablassert emits a `BiolinkRelocationWarning` naming where the value went. This set is derived from the *installed* `biolink-model`, so a slot leaves it automatically once a release attaches it. +- **Tablassert pipeline fields**: `upstream_resource_ids`, `source_record_urls`. Any other annotation name is treated as **supporting context**. At the end of `compile_graph`, tablassert sweeps the edge columns: for each non-allow-listed name it emits `"name: value"` entries into the edge's `supporting_text` (a `list[str]`), then drops the original column. Behavior worth knowing: @@ -498,9 +498,9 @@ Any other annotation name is treated as **supporting context**. At the end of `c This means nothing in your source data is silently dropped: context that doesn't map to a structured Biolink slot travels along inside `supporting_text` instead. -In addition to user-declared annotations, every edge automatically carries `extracted_from_row_number`, a 1-based index into the original source table (matching Excel-style row numbering). It is not declared as an annotation — tablassert emits it internally so each edge always carries its source-row provenance. Together with the sheet name it identifies the edge's **inlined supporting study** (`has_supporting_studies`), where it is carried alongside any relocated unsatisfiable slots; neither is folded into `supporting_text`. +In addition to user-declared annotations, every edge automatically carries `extracted_from_row_number`, a 1-based index into the original source table (matching Excel-style row numbering). It is not declared as an annotation; tablassert emits it internally so each edge always carries its source-row provenance. Together with the sheet name it identifies the edge's **inlined supporting study** (`has_supporting_studies`), where it is carried alongside any relocated unsatisfiable slots; neither is folded into `supporting_text`. -The supporting study is only emitted when it carries something. Biolink defines `has supporting studies` as "studies that produced information used as evidence", so a section that declares **no `publications`** and has **no** relocated slots or class-pruned values emits no `has_supporting_studies` at all: its `study_id` would fall back to the config filename, making every edge assert a `Study` named `my_table.yaml` whose only result is a row index. A section with a real publication always keeps the struct — `PMID:123#Table_S7 row 12` is genuine provenance — as does any section with values to preserve. The row and sheet columns are consumed either way. +The supporting study is only emitted when it carries something. Biolink defines `has supporting studies` as "studies that produced information used as evidence", so a section that declares **no `publications`** and has **no** relocated slots or class-pruned values emits no `has_supporting_studies` at all: its `study_id` would fall back to the config filename, making every edge assert a `Study` named `my_table.yaml` whose only result is a row index. A section with a real publication always keeps the struct (`PMID:123#Table_S7 row 12` is genuine provenance), as does any section with values to preserve. The row and sheet columns are consumed either way. #### Automatic column coercion @@ -515,11 +515,11 @@ Before the allow-list sweep runs, tablassert renames statistical columns to thei | Effect type | `effect_type` | `effect type`, `effect metric`, `statistic type`, `metric` | - **`effect_type` values are also coerced.** Each cell is matched case/separator-insensitively against an alias table (e.g. `"OR"` → `odds_ratio`, `"Cohen's d"` → `cohens_d`, `"Spearman"` → `spearmans_rho`), then by `rapidfuzz` fallback against the 25 permissible `EffectTypes` values; anything matching nothing is dropped to `null` rather than carried through (the Biolink range is the enum). -- **`statistical_significance_qualifier` is auto-derived** from the p-value column into five bands — `biolink:very_strongly_significant` (p ≤ 0.001), `biolink:strongly_significant` (≤ 0.01), `biolink:significant` (≤ 0.05), `biolink:suggestive` (≤ 0.10), `biolink:not_significant` (> 0.10). The same rigorous selection picks the source column: a raw `p_value` column is preferred, `adjusted_p_value` is the fallback, and the qualifier is omitted entirely when no p-value column is present. -- **`effect_size` and `effect_type` travel as a pair.** A section declaring one without the other does not fail validation: the unpaired annotation is **dropped** from the section with an `UnpairedEffectAnnotationWarning` naming what was dropped and from where, and the section's edges are kept — neither half carries evidence alone (a bare effect size is uninterpretable — 0.85 of *what*, an odds ratio or a Spearman rho? — and Biolink PR #1774 only populates `effect_type` alongside a numeric `effect_size`, so the build nulls an unpaired type anyway). Declare both together to retain the full evidence. Alias spellings count: `odds ratio` and the legacy `relationship_strength` coerce to `effect_size`, so both still need a sibling `effect_type`. Use `method: value` when every row shares one statistic and `method: column` when the table provides it; in a `template` + `sections` config the two lists are concatenated, so a constant `effect_type` declared once on the template pairs with each section's own `effect_size` column. +- **`statistical_significance_qualifier` is auto-derived** from the p-value column into five bands: `biolink:very_strongly_significant` (p ≤ 0.001), `biolink:strongly_significant` (≤ 0.01), `biolink:significant` (≤ 0.05), `biolink:suggestive` (≤ 0.10), `biolink:not_significant` (> 0.10). The same rigorous selection picks the source column: a raw `p_value` column is preferred, `adjusted_p_value` is the fallback, and the qualifier is omitted entirely when no p-value column is present. +- **`effect_size` and `effect_type` travel as a pair.** A section declaring one without the other does not fail validation: the unpaired annotation is **dropped** from the section with an `UnpairedEffectAnnotationWarning` naming what was dropped and from where, and the section's edges are kept; neither half carries evidence alone. A bare effect size is uninterpretable (0.85 of *what*, an odds ratio or a Spearman rho?), and Biolink PR #1774 only populates `effect_type` alongside a numeric `effect_size`, so the build nulls an unpaired type anyway. Declare both together to retain the full evidence. Alias spellings count: `odds ratio` and the legacy `relationship_strength` coerce to `effect_size`, so both still need a sibling `effect_type`. Use `method: value` when every row shares one statistic and `method: column` when the table provides it; in a `template` + `sections` config the two lists are concatenated, so a constant `effect_type` declared once on the template pairs with each section's own `effect_size` column. - **Biolink class rules are enforced.** `effect_type` is nulled on every row where `effect_size` is null (and nulled entirely when no `effect_size` column exists); `statistical_significance_qualifier` is only set when `p_value`/`adjusted_p_value` is populated, and null p-values yield a null qualifier. -This is why declaring an annotation like `{annotation: p value, method: column, encoding: E}` still produces a top-level `p_value` edge field — the header is normalized to the Biolink name before folding is considered. +This is why declaring an annotation like `{annotation: p value, method: column, encoding: E}` still produces a top-level `p_value` edge field: the header is normalized to the Biolink name before folding is considered. ## Next Steps diff --git a/docs/development.md b/docs/development.md index d791f0c..e753915 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,6 +1,6 @@ # Development -Set up a Tablassert dev environment and learn the contributor workflow — build the Rust extension, +Set up a Tablassert dev environment and learn the contributor workflow: build the Rust extension, run the test and docs gates locally, and lint/format. The canonical contributor guide is [`CONTRIBUTING.md`](https://github.com/SkyeAv/Tablassert/blob/main/CONTRIBUTING.md). ## Setup from source @@ -46,7 +46,7 @@ This site is built with MkDocs Material. After editing anything under `docs/` or uv run mkdocs build --strict ``` -`--strict` promotes any warning — a dead cross-link, a nav entry pointing at a missing file, or a missing referenced doc — to a build failure, so documentation drift is caught here rather than only on the deployed site. Style and PR expectations live in the canonical contributor guide, [`CONTRIBUTING.md`](https://github.com/SkyeAv/Tablassert/blob/main/CONTRIBUTING.md). +`--strict` promotes any warning (a dead cross-link, a nav entry pointing at a missing file, or a missing referenced doc) to a build failure, so documentation drift is caught here rather than only on the deployed site. Style and PR expectations live in the canonical contributor guide, [`CONTRIBUTING.md`](https://github.com/SkyeAv/Tablassert/blob/main/CONTRIBUTING.md). ## Fullmap builds diff --git a/docs/examples.md b/docs/examples.md index 5756434..a8832c3 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1,6 +1,6 @@ # Use Case Gallery -**Turn real tabular sources — CSV, TSV, or Excel — into KGX-compliant nodes and edges.** Each pattern +**Turn real tabular sources (CSV, TSV, or Excel) into KGX-compliant nodes and edges.** Each pattern below leads with the data type and the outcome it produces, then gives a complete, schema-valid configuration and the techniques that make it work. @@ -147,7 +147,7 @@ template: **Key techniques:** -- **Regex pipeline** cleans raw taxonomic strings (`d__Bacteria;p__Firmicutes;g__Lactobacillus` → `Lactobacillus`). Patterns must be Polars `str.replace_all()`-compatible (Rust `regex` engine) — no backreferences (`\1`, `\2`, …) or lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?= 100; multiple reindex conditions are ANDed together. -- **Comparison operators** — `lt` (less than), `ge` (greater or equal), `eq`, `ne`, `gt`, `le`. +- **Comparison operators**: `lt` (less than), `ge` (greater or equal), `eq`, `ne`, `gt`, `le`. --- @@ -296,7 +296,7 @@ template: **Key techniques:** - **Forward fill** (`fill: forward`) propagates the last non-null value downward, mapping subcategory rows to their parent category. -- **Other fill strategies** — `backward`, `min`, `max`, `mean`, `zero`, `one`. +- **Other fill strategies**: `backward`, `min`, `max`, `mean`, `zero`, `one`. --- diff --git a/docs/fullmap.md b/docs/fullmap.md index 9af9826..167ee5c 100644 --- a/docs/fullmap.md +++ b/docs/fullmap.md @@ -1,16 +1,16 @@ # Fullmap Build a fullmap once and every `resolve()` / `resolve_many()` call maps free text to the right -biological CURIE — the completeness and freshness of this database directly sets how many of your +biological CURIE; the completeness and freshness of this database directly sets how many of your entities resolve correctly and how trustworthy the resulting graph is. Fullmap is Tablassert's embedded entity-resolution database: a small set of [redb](https://github.com/cberner/redb) -files — a primary file holding the dimensions, CURIEs, and schema metadata, plus hash-sharded RECORDS -files (`fullmap.s0.redb` … `fullmap.s15.redb` by default) holding the term→postings index — containing -biological synonyms, CURIEs, Biolink categories, taxon IDs, and source provenance, built from NCATS -Translator BABEL export files. +files containing biological synonyms, CURIEs, Biolink categories, taxon IDs, and source provenance, +built from NCATS Translator BABEL export files. It comprises a primary file holding the dimensions, +CURIEs, and schema metadata, plus hash-sharded RECORDS files (`fullmap.s0.redb` … `fullmap.s15.redb` +by default) holding the term→postings index. -Fullmap is built entirely in-process by Tablassert's own Rust extension — no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). If you opt into `build-fullmap --aria2c` / `-a`, only the download stage uses the bundled aria2c binary from the optional `[aria2]` extra (`pip install "tablassert[aria2]"`; Linux/Windows wheels only). +Fullmap is built entirely in-process by Tablassert's own Rust extension: no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). If you opt into `build-fullmap --aria2c` / `-a`, only the download stage uses the bundled aria2c binary from the optional `[aria2]` extra (`pip install "tablassert[aria2]"`; Linux/Windows wheels only). ## Build Command @@ -30,23 +30,23 @@ cache directory, BABEL snapshot version, worker threads, the optional `--aria2c` and the `--force` / `-f` rebuild flag), their defaults, and more examples. By default, `build-fullmap` first downloads a **prebuilt** database published for this Tablassert -version — a `fullmap.tar.zst` under `.../fullmap//` (the version directory is the +version, a `fullmap.tar.zst` under `.../fullmap//` (the version directory is the installed package version, never hardcoded), verified against a co-published `sha256sum.txt` and extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd → tar (the multi-GB decompressed tar is never materialized on disk) with the GIL released, extracts into a -temp directory on the output's filesystem, and — before renaming anything into place — validates the +temp directory on the output's filesystem, and, before renaming anything into place, validates the bundle against the same contract a `--force` build must satisfy: the `meta` schema tag is exactly `tablassert.fullmap.v5`, a `build_id` is recorded, the shard files are exactly the set the primary advertises (no gaps, no extras), and every shard's `build_id` equals the primary's. Only a bundle that passes is atomically renamed into place (primary → `--output`, shards beside it); any failure raises and the command falls back to the from-scratch build below. If no prebuilt is published for this version it falls back the same way; `--force` / `-f` skips the prebuilt attempt and always builds. The -optional `--aria2c` / `-a` accelerates **either** download — the multi-GB prebuilt archive is the ideal +optional `--aria2c` / `-a` accelerates **either** download: the multi-GB prebuilt archive is the ideal aria2 use case. Two facts matter most when planning a build: -- The BABEL **version** flag selects a RENCI BABEL snapshot date (default `2026jul22`) — *not* +- The BABEL **version** flag selects a RENCI BABEL snapshot date (default `2026jul22`), *not* Tablassert's package version. Bumping it fetches a different snapshot and requires rebuilding; the value used is recorded in the primary's `meta` table (`source_version`). - With **threads** left unset, the Rust build caps workers at `min(available_CPUs, MemAvailable_GB / 2)` @@ -57,13 +57,13 @@ Two facts matter most when planning a build: The build is a parallel, **memory-bounded** pipeline executed by the Rust extension: -1. **Download** — fetch BABEL class and synonym files from RENCI into the cache (resumable, reused). By default this uses Tablassert's Python downloader; `--aria2c` / `-a` opts into the bundled `aria2c` binary from the `[aria2]` extra, preserving aria2 resume control files across dropped downloads and failing loud if the extra is missing, unsupported on the current platform, or the download fails. -2. **Equivalents index** — parse class files into sorted on-disk runs, then k-way merge them into a +1. **Download**: fetch BABEL class and synonym files from RENCI into the cache (resumable, reused). By default this uses Tablassert's Python downloader; `--aria2c` / `-a` opts into the bundled `aria2c` binary from the `[aria2]` extra, preserving aria2 resume control files across dropped downloads and failing loud if the extra is missing, unsupported on the current platform, or the download fails. +2. **Equivalents index**: parse class files into sorted on-disk runs, then k-way merge them into a memory-mapped index mapping each primary CURIE to its equivalents. -3. **Synonym pass** — a producer/consumer pool streams byte-bounded line-chunks; workers dedup CURIEs, +3. **Synonym pass**: a producer/consumer pool streams byte-bounded line-chunks; workers dedup CURIEs, accumulate normalized-term → (CURIE, source) postings, and spill per-shard runs to disk so peak RAM stays flat regardless of input size. -4. **Write** — one redb transaction writes the primary tables, then `shard_count` independent k-way +4. **Write**: one redb transaction writes the primary tables, then `shard_count` independent k-way merges write the shard `records` files in parallel (one thread per shard). ??? note "Pipeline details" @@ -72,32 +72,32 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens memory budget; the [mimalloc](https://github.com/microsoft/mimalloc) allocator keeps heavy multi-threaded allocation from bloating resident memory. - - **Download** — files come from `https://stars.renci.org/var/babel_outputs` via resumable, + - **Download**: files come from `https://stars.renci.org/var/babel_outputs` via resumable, range-request downloads; cached files are reused. Passing `--aria2c` / `-a` switches only this stage to the bundled `aria2c` binary from the optional `[aria2]` extra, using aria2's segmented HTTP downloads and retry/resume control files while suppressing aria2's own progress UI so Tablassert's progress bar stays clean. The progress detail remains file-level (`aria2c downloading`) rather than byte-level in this mode. - - **Equivalents index** — class files parse in parallel into sorted on-disk runs, k-way merged into a + - **Equivalents index**: class files parse in parallel into sorted on-disk runs, k-way merged into a single memory-mapped CURIE→equivalents index; only a compact `(hash, offset)` index lives in RAM, the string data is mmap'd. - - **Synonym pass** — uses **intra-file parallelism**: a small pool of producer threads + - **Synonym pass**: uses **intra-file parallelism**: a small pool of producer threads decompresses/reads the synonym files and pushes byte-bounded line-chunks through a bounded channel, and every worker draws from one shared queue, so the few very large files (protein/smallmolecule/gene/drugchemicalconflated) are processed by **all** workers, not one thread each. Per row, the build collects dimension sets (CURIE prefixes, Biolink categories, sources), assigns compact integer CURIE IDs via a hash-keyed dedup map (`xxh3_128(curie) → id`), and accumulates normalized-term → (CURIE, source) postings. Each worker drains its per-CURIE rows and - term postings to bounded on-disk spill runs once its buffer fills — the term postings partitioned + term postings to bounded on-disk spill runs once its buffer fills, the term postings partitioned per-shard at spill time (each `run_s{shard}_{id}.bin` holds only terms with `xxh64(term) & (shards-1)` matching that shard), which keeps peak RAM flat regardless of input size and lets the write phase merge each shard independently. Dead terms (purely numeric, or generic labels like `none`/`nan`/`null`) are skipped, since they can never be queried. - - **Write** — a single redb write transaction in the primary file emits the dimension tables + - **Write**: a single redb write transaction in the primary file emits the dimension tables (`prefixes`, `categories`, `sources`), the `curies` table (streamed from its spill runs), and the `meta` schema tag (recording the shard count). The `records` table is then written **in parallel across the shard files**: building on the per-shard spill partitioning, the write phase runs - `shard_count` independent k-way merges — one thread per shard, each merging only its own shard's + `shard_count` independent k-way merges, one thread per shard, each merging only its own shard's runs and inserting the merged term groups inline into that shard's redb file (one database per shard, since redb allows a single writer per file) in hash-sorted batches for near-sequential B-tree appends (each batch is appended through redb's end-of-table cursor API, the faster ascending @@ -127,7 +127,7 @@ override only when targeting an unusual machine. ## Output Artifact A primary redb file (default `./fullmap/data/fullmap.redb`) plus its sibling RECORDS shard files -(`fullmap.s0.redb` … `fullmap.s15.redb` — one per shard, named after the output +(`fullmap.s0.redb` … `fullmap.s15.redb`, one per shard, named after the output file stem in the same directory). The shard count is fixed at 16 (the read path still honors the count recorded in an existing database's `meta` table). Together they hold six tables (see `rust/src/fullmap.rs`): @@ -141,21 +141,21 @@ they hold six tables (see `rust/src/fullmap.rs`): | `curies` | Compact `u32` id → CURIE record (CURIE, preferred name, category, taxon, source) (primary file) | | `meta` | Schema version tag (`tablassert.fullmap.v5`), the shard count (`shards`), and the BABEL `source_version` used to build the file (primary file) | -The shard files must remain alongside the primary file — lookups discover them as siblings of the +The shard files must remain alongside the primary file: lookups discover them as siblings of the resolved primary path. Lookups (`lookup_fullmap_terms`) check the primary's `meta` schema tag before reading `records`, read the `shards` count to open exactly that many shard files, and fan the query terms out across the shards in parallel (releasing the GIL, one reader per non-empty shard, re-merged into input order); a mismatched or missing tag raises rather than silently reading incompatible data. Databases built under the older -`v1`/`v2`/`v3`/`v4` schemas are rejected — there is no automatic schema migration, so a schema bump +`v1`/`v2`/`v3`/`v4` schemas are rejected: there is no automatic schema migration, so a schema bump (including the v3→v4 move to sharded files and the v4→v5 move to the redb 4 engine) requires rebuilding via `tablassert build-fullmap`. Readers open every fullmap file READ-ONLY with a SHARED file lock (redb ≥ 3 `ReadOnlyDatabase`), so any number of processes can run lookups against the same fullmap concurrently; only a `build-fullmap` rebuild (an exclusive-lock writer) briefly blocks readers. Each lookup pins one primary-plus-shards file -generation — cached handles are validated against the file's `(dev, ino)` on every use — so a reader +generation, cached handles are validated against the file's `(dev, ino)` on every use, so a reader follows a rebuild on the next lookup. ## Usage in Graph Config @@ -194,6 +194,6 @@ rig: ## Programmatic Usage -Pass the fullmap path (file or base directory) as the `fullmap` argument to `resolve_many()` — see +Pass the fullmap path (file or base directory) as the `fullmap` argument to `resolve_many()`; see [Batch Resolution](api/lib.md) for the full reference and example, and [Entity Resolution](api/fullmap.md) for the lower-level `resolve()` API. diff --git a/docs/index.md b/docs/index.md index a02894e..260978a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,20 +1,20 @@ # Tablassert Tablassert turns biomedical tables (Excel, CSV, TSV) into NCATS Translator-compliant KGX knowledge -graphs — declaratively, with entity resolution built in and optional quality control. **Installing?** +graphs, declaratively, with entity resolution built in and optional quality control. **Installing?** See [Installation](installation.md). **First build?** Follow the [Tutorial](tutorial.md). **Automating?** Use the [CLI](cli.md) or the autonomous [Agent](agent.md). ## Why Tablassert -- **Declarative YAML configuration** — define data transformations without code -- **Entity resolution** — map free text to biological entities (genes, diseases, chemicals) with +- **Declarative YAML configuration**: define data transformations without code +- **Entity resolution**: map free text to biological entities (genes, diseases, chemicals) with taxonomic filtering and provenance, backed by an embedded redb database -- **Optional quality control** — four-stage audit (exact → fuzzy → abbreviation → SapBERT embeddings) flags +- **Optional quality control**: four-stage audit (exact → fuzzy → abbreviation → SapBERT embeddings) flags low-confidence mappings -- **KGX compliance** — emits NCATS Translator-compatible node/edge NDJSON with Biolink categories and +- **KGX compliance**: emits NCATS Translator-compatible node/edge NDJSON with Biolink categories and predicates -- **Performance & reproducibility** — lazy Polars pipelines and a UV-based, deterministic development +- **Performance & reproducibility**: lazy Polars pipelines and a UV-based, deterministic development environment ## Quick Start @@ -29,18 +29,18 @@ the full install matrix, extras, and development setup. ## Documentation Sections -- **[Installation](installation.md)** — install methods, extras, and development setup -- **[CLI Reference](cli.md)** — complete command-line flag reference -- **[Tutorial](tutorial.md)** — step-by-step example with synthetic data -- **[Use Case Gallery](examples.md)** — real-world configuration patterns -- **[Configuration](configuration/graph.md)** — graph and table configuration reference -- **[API Reference](api/fullmap.md)** — core functions documentation +- **[Installation](installation.md)**: install methods, extras, and development setup +- **[CLI Reference](cli.md)**: complete command-line flag reference +- **[Tutorial](tutorial.md)**: step-by-step example with synthetic data +- **[Use Case Gallery](examples.md)**: real-world configuration patterns +- **[Configuration](configuration/graph.md)**: graph and table configuration reference +- **[API Reference](api/fullmap.md)**: core functions documentation ## Authors -- **[Skye Lane Goetz](mailto:sgoetz@isbscience.org)** — Institute for Systems Biology -- **[Gwênlyn Glusman](mailto:gglusman@isbscience.org)** — Institute for Systems Biology -- **Jared C. Roach** — Institute for Systems Biology +- **[Skye Lane Goetz](mailto:sgoetz@isbscience.org)**: Institute for Systems Biology +- **[Gwênlyn Glusman](mailto:gglusman@isbscience.org)**: Institute for Systems Biology +- **Jared C. Roach**: Institute for Systems Biology ## License diff --git a/docs/installation.md b/docs/installation.md index f37b8eb..01cec78 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -103,7 +103,7 @@ never lets it surface as a bare `ModuleNotFoundError`. Every one of these paths distribution **and** the command that fixes it: ```text -Missing optional dependencies 'scikit-learn', 'sentence-transformers' — required by the QC audit. +Missing optional dependencies 'scikit-learn', 'sentence-transformers', required by the QC audit. Install the [qc] extra: pip install "tablassert[qc]" (uv: uv tool install "tablassert[qc]") ``` @@ -111,7 +111,7 @@ Where the gap is knowable up front, it is reported up front rather than mid-run: | Command | Checked | When | |---|---|---| -| `build-kg --qc` | `[qc]` | Before the build starts — the QC audit runs at the very end of the build, so a late failure would cost the entire entity-resolution pass | +| `build-kg --qc` | `[qc]` | Before the build starts: the QC audit runs at the very end of the build, so a late failure would cost the entire entity-resolution pass | | `tablassert agent` | `[agent]` | After flag validation, before any model is built or any article fetched | | `tablassert agent --optimize` | `[agent]` + `[optimize]` | Same point; both are reported at once | | `build-fullmap --aria2c` | `[aria2]` | Before any download starts | @@ -121,8 +121,8 @@ rather than a retry loop. Library calls that reach an optional import directly ( `fullmap_audit()` or the agent's lazy `dspy` import) raise the same message at that point. The `rt` extra is the exception: it installs `polars[rtcompat]`, which imports as plain `polars`, so -it cannot be detected by inspection. It is suggested when polars itself fails to import — the usual -cause being a CPU that lacks the instructions the default polars wheel requires. +it cannot be detected by inspection. It is suggested when polars itself fails to import; the usual +cause is a CPU that lacks the instructions the default polars wheel requires. ### Method 3: Install from GitHub main diff --git a/docs/tutorial.md b/docs/tutorial.md index 4b05153..1dcbbc8 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -1,7 +1,7 @@ # Tutorial: Your First Knowledge Graph **By the end of this tutorial you will have built a KGX-compliant knowledge graph from a CSV of -gene-disease associations** — nodes and edges with standardized CURIEs, biolink categories, provenance, +gene-disease associations**: nodes and edges with standardized CURIEs, biolink categories, provenance, and statistical annotations, ready for NCATS Translator. You'll learn the complete workflow: creating configurations, running Tablassert, and examining the output. @@ -119,7 +119,7 @@ rig: ``` **Important:** Replace `fullmap` with the path to your fullmap redb (and adjust `tables` if your table -config lives elsewhere). The `rig:` section is required — see the +config lives elsewhere). The `rig:` section is required; see the [Graph configuration reference](configuration/graph.md) for every field. **What this does:** @@ -184,7 +184,7 @@ Example output (p-values are emitted as controlled scientific-notation strings; cat tutorial-output/TUTORIAL_KG_1.0.0.RIG.yaml ``` -The Resource Ingest Guide records the graph's source metadata and terms of use (`rig.source_info`), ingest utility and scope (`rig.ingest_info`), provenance (`rig.provenance_info`), the generated artifact locations, and a summary of the emitted node and edge types — all validated in memory before the file is written, for NCATS Translator registration. +The Resource Ingest Guide records the graph's source metadata and terms of use (`rig.source_info`), ingest utility and scope (`rig.ingest_info`), provenance (`rig.provenance_info`), the generated artifact locations, and a summary of the emitted node and edge types, all validated in memory before the file is written, for NCATS Translator registration. ## Understanding the Transformation diff --git a/examples/agent/README.md b/examples/agent/README.md index bcf8edd..3117e01 100644 --- a/examples/agent/README.md +++ b/examples/agent/README.md @@ -5,7 +5,7 @@ These artifacts come from running the Tablassert `[agent]` GEPA prompt-optimizat ## Files -- **`optimized_instructions.yaml`** — a GEPA-optimized agent prompt (the `instructions` the inner +- **`optimized_instructions.yaml`**: a GEPA-optimized agent prompt (the `instructions` the inner `CodeAgent` runs with), plus the per-predictor `descriptions`. Load it directly to skip the optimization cost in production: @@ -16,17 +16,17 @@ These artifacts come from running the Tablassert `[agent]` GEPA prompt-optimizat Compared to the built-in `INSTRUCTIONS`, this prompt adds explicit, feedback-derived guidance: a `source.url`-is-required rule, a `prioritize` entity-type mapping table (raw column headers like - `Symbol`/`HGNC` are invalid — map them to biolink types), a per-error-code recovery cheat-sheet, + `Symbol`/`HGNC` are invalid; map them to biolink types), a per-error-code recovery cheat-sheet, section de-duplication limits, and a **source-path-fidelity** rule (copy the candidate table's exact absolute path into `source.local` verbatim). -- **`gepa-dataset.yaml`** — an example GEPA dataset (two open-access PMC gene tables). Each entry carries +- **`gepa-dataset.yaml`**: an example GEPA dataset (two open-access PMC gene tables). Each entry carries `table_summary` + `coverage_feedback` (the program inputs) and optionally `fullmap` / `workdir` / `head` so the GEPA metric scores each proposed config with **real** fullmap coverage. -- **`QC_REPORT.md` / `QC_REVIEW.md`** — the assay report and the LLM-as-judge review produced by +- **`QC_REPORT.md` / `QC_REVIEW.md`**: the assay report and the LLM-as-judge review produced by `qc/qc_report.py` and `qc/qc_reviewer.py` from a shared agent state dir. Every per-PMC entry in both carries the config's `sha256` so a future report/review/config drift is detectable. -- **`qc/`** — the two QC scripts. `qc_report.py` is deterministic (no LLM); `qc_reviewer.py` calls the +- **`qc/`**: the two QC scripts. `qc_report.py` is deterministic (no LLM); `qc_reviewer.py` calls the judge LM (needs `QWEN_TOKEN_PLAN_URL` / `QWEN_TOKEN_PLAN_API_KEY`) and accepts ` --rerender` to rebuild the markdown from an existing `qc_review.json` without re-querying the judge. @@ -34,7 +34,7 @@ These artifacts come from running the Tablassert `[agent]` GEPA prompt-optimizat ## Reproducing the optimization The dataset paths (`fullmap`, `workdir`, and the table paths embedded in `table_summary`) are -**machine-specific** — adapt them to your environment first. Then: +**machine-specific**, so adapt them to your environment first. Then: ```bash export TABLASSERT_AGENT_MODEL_ID="qwen3.8-max-preview" # strong reflection LM @@ -60,7 +60,7 @@ few instruction edits, while a **fast task LM** (`--task-model`) runs the many c Configs generated from `gepa-dataset.yaml` point `source.local` at tables under the **GEPA run's state dir** (here `.tablassert/gepa/downloads/…`, since the dataset's `workdir` is `.tablassert/gepa`), but `qc/qc_reviewer.py` only reads a config's `source.local` when it resolves INSIDE its own -`STATE_DIR/downloads` allowlist (an injection defense — see `get_table_summary`). So the QC scripts must +`STATE_DIR/downloads` allowlist (an injection defense, see `get_table_summary`). So the QC scripts must be pointed at the SAME state dir that holds `downloads/`, or the tables must be staged there: ```bash diff --git a/tests/agent_fixtures/PMC11708054/README.md b/tests/agent_fixtures/PMC11708054/README.md index a8ef082..544431c 100644 --- a/tests/agent_fixtures/PMC11708054/README.md +++ b/tests/agent_fixtures/PMC11708054/README.md @@ -1,7 +1,7 @@ # Golden fixture: PMC11708054 (microbiome ~ tamoxifen correlations) Offline replay pair for the US-011 eval harness. **Everything here is SYNTHETIC and -OFFLINE** — no live network fetch and no LLM call is ever made against these files. They +OFFLINE**: no live network fetch and no LLM call is ever made against these files. They give the eval tests a deterministic, schema-valid config + a tiny table to score against. ## Attribution (CC-BY) @@ -9,7 +9,7 @@ give the eval tests a deterministic, schema-valid config + a tiny table to score The *shape* of this fixture (an organism `correlated_with` chemical table with Spearman rho / p-value annotations) mirrors the real ALAMV6 supplementary table from: -- **PMC11708054** — DOI [10.1128/mbio.01679-24](https://doi.org/10.1128/mbio.01679-24) +- **PMC11708054**: DOI [10.1128/mbio.01679-24](https://doi.org/10.1128/mbio.01679-24) - Licensed CC-BY; cite the article DOI when reusing the real data. The actual cell values and organism list in `source_table.csv` are **fabricated** for @@ -17,18 +17,18 @@ testing and do NOT reproduce any real measurement from the article. ## Files -- **`ALAMV6.yaml`** — the reference *table config*: a faithful copy of the `template:` +- **`ALAMV6.yaml`**: the reference *table config*: a faithful copy of the `template:` block documented in `docs/configuration/advanced-example.md` (excel source, full annotation set, lineage-glue regex). Schema-valid (`validate_section(...) is True`). The `local:` excel path is illustrative only; the file is not present and is never read. -- **`source_table.csv`** — a SMALL synthetic snapshot (7 rows, **headerless**) shaped like +- **`source_table.csv`**: a SMALL synthetic snapshot (7 rows, **headerless**) shaped like ALAMV6's "all correlations" sheet: column A = organism name (taxonomic string), column B = Spearman rho (float), column C = p_value (float). Deterministic + tiny. -- **`reference_config.yaml`** — the golden config the agent should approximate, trimmed to +- **`reference_config.yaml`**: the golden config the agent should approximate, trimmed to `source_table.csv`'s columns (text/CSV source; subject = column A `OrganismTaxon`; object = fixed literal `CHEBI:41774`; p_value = C, relationship_strength = B). Schema-valid. -- **`README.md`** — this file. +- **`README.md`**: this file. ## Reference KGX is computed in-test (NOT committed) diff --git a/tests/fixtures/edgecount/README.md b/tests/fixtures/edgecount/README.md index a58b0a9..ea56c6e 100644 --- a/tests/fixtures/edgecount/README.md +++ b/tests/fixtures/edgecount/README.md @@ -4,7 +4,7 @@ Offline acceptance pair for the **edge-count harness**: an agent-produced config reach at least `REFERENCE_EDGE_FRACTION` (0.5) of the reference config's KGX edge count when both build the same payload against the same tiny real redb. -**Everything here is SYNTHETIC and OFFLINE** — no network fetch, no LLM call. +**Everything here is SYNTHETIC and OFFLINE**: no network fetch, no LLM call. ## Attribution (shape only) @@ -15,27 +15,27 @@ testing; nothing reproduces a real measurement from any article. ## Files -- **`payload.xlsx`** — 3 worksheets, 50 data rows total (row 0 = title, row 1 = header, +- **`payload.xlsx`**: 3 worksheets, 50 data rows total (row 0 = title, row 1 = header, data from row 2, so configs skip both with `row_slice: [2, auto]`): - `disease_system` (28 rows): disease | organ_system | beta | p_value. - - `locus_hits` (8 rows): locus | **;-joined** diseases | beta | p_value — the + - `locus_hits` (8 rows): locus | **;-joined** diseases | beta | p_value; the multi-valued sheet `explode_by: ";"` turns into ~20 edges. - `secondary_endpoints` (14 rows): same columns as `disease_system`, disjoint pairs. -- **`agent_config.yaml`** — the improved-agent config (US-006 shape): multi-section, +- **`agent_config.yaml`**: the improved-agent config (US-006 shape): multi-section, correct `sheet` + `row_slice` per section, breadth via `prioritize` and `explode_by`, and the statistical annotation pair (`effect_size` column + `effect_type` `method: value`). Covers 2 of the 3 sheets. -- **`reference_config.yaml`** — the richer reference: all 3 sheets, one section each. -- **`agent_config_poor.yaml`** — the negative control: single section over `locus_hits` +- **`reference_config.yaml`**: the richer reference: all 3 sheets, one section each. +- **`agent_config_poor.yaml`**: the negative control: single section over `locus_hits` WITHOUT `explode_by`; the joined cells never resolve, so it lands far below the fraction gate. -Every (disease, system) / (locus, disease) pair is unique across all sheets — edges are +Every (disease, system) / (locus, disease) pair is unique across all sheets; edges are keyed by (subject, predicate, object), so no pair silently collapses. ## Resolution `tests/test_agent_edgecount.py` builds a tiny REAL redb (`tablassert.rs.build_fullmap_db`) registering the 12 diseases (MONDO), 8 organ systems (UBERON, AnatomicalEntity), and 8 -loci (HGNC) — the e2e recipe from `tests/test_e2e_smoke.py`. The configs' relative +loci (HGNC), the e2e recipe from `tests/test_e2e_smoke.py`. The configs' relative `local` path is rewritten to the absolute fixture payload in-test.