`darnit-reproducibility`: environment-generation sieve (deterministic-first, LLM fallback)

[mlieberman85]

Summary

darnit is positioned to help with the reproducibility edge cases --
the places where existing environment-generation tooling (repo2docker,
Cloud Native Buildpacks, devcontainer templates) doesn't reach. This
issue scopes a "sieve" for environment generation that mirrors
darnit's own control-sieve pattern: try deterministic tools first,
escalate to an LLM only where they can't conclude, always with
operator sign-off before applying.

This is not a plan to compete with repo2docker or buildpacks. It's a
plan to route to them first, then fill the gaps.

Prior art (deterministic tier)

• repo2docker (Jupyter/Binder): detects requirements.txt,
  environment.yml, Pipfile, install.R, apt.txt, postBuild,
  runtime.txt, Dockerfile, default.nix. Produces a Dockerfile.
  Stops at: no env files -> bare python:3.11; pyproject.toml
  without a lockfile; compiled deps beyond apt.txt (CUDA, MKL,
  custom C libs).
• Cloud Native Buildpacks (pack): ordered buildpack list; first
  match wins. Stops at: needs Procfile / entry point; doesn't fit
  scientific workflow shapes (Nextflow, Snakemake, papermill); emits
  OCI images, not inspectable Dockerfiles.
• docker init: interactive prompt; not a fit for automation.
• Devcontainer templates: static templates keyed on
  .devcontainer/devcontainer.json presence + language; requires
  operator to pick a template.
• Nix flake generators: minimal skeleton, not repo-analysis-driven.

Where LLM tier earns its place

Concrete edge cases the deterministic tools do not handle today:

• Compiled deps beyond apt.txt (repo mentions PyTorch + CUDA
  version in README prose; no lockfile encodes the CUDA choice).
• Non-buildpack workflow shapes (snakemake --cores 8,
  nextflow run main.nf, papermill notebook.ipynb -p param 3).
• System prerequisites in README prose that no deterministic tool
  parses (Install OpenBabel first, then run pip install .).
• pyproject.toml with editable install (pip install -e .) but no
  lockfile -- repo2docker punts.
• Multi-stage builds where the researcher's own tool must be compiled
  before the experiment runs (Layer 1 vs Layer 2 per darnit-reproducibility: reproducibility layer + tightness taxonomy #415).

Proposed shape

Same pass shape as sieve controls:

sieve = [
  ("repo2docker",  detect: any env file in the list above),
  ("buildpacks",   detect: `pack builder inspect` returns a match),
  ("heuristic",    detect: single-language repo with one obvious lockfile),
  ("llm_eval",     always-runs-if-preceding-inconclusive; prompt: file
                    listing, top-level READMEs, detected language
                    signals, evidence of compiled-deps),
  ("manual",       operator prompt with best-effort skeleton to edit),
]

Each pass either produces a Dockerfile + justification, or falls
through. LLM output is a candidate, not a conclusion (Constitution
IV -- Never Guess User Values). Operator confirmation gates
application; the artifact is not written into the repo silently.

Scope

• Design the sieve above.
• Prototype the deterministic tier via repo2docker's Python API
  (repo2docker.Repo2Docker) as the reference integration.
• Design the LLM tier's prompt shape and evidence bundle (leans on
  the LLM-reasoning-capture work from Determinism Tier 3: capture LLM reasoning and evidence trail in attestations, logs, and reports #421 -- prompt and reasoning
  MUST land in a durable artifact next to the Dockerfile).
• Decide artifact surface: Dockerfile vs devcontainer.json vs both;
  where it lands in the repo; what darnit records in the attestation.

Non-goals for v0

• Rewriting repo2docker's or buildpacks' detection logic.
• Nix flake generation (narrow audience; revisit once the sieve
  shape stabilizes).
• Runtime execution verification (that's the ambitious Option C from
  the design conversation -- separate issue if pursued).
• Auto-applying LLM-generated Dockerfiles without operator sign-off.

Constitution alignment

• I. Plugin Separation: the sieve lives inside
  darnit-reproducibility; darnit core doesn't grow a Docker
  dependency.
• II. Conservative-by-Default: LLM tier produces candidates, not
  conclusions.
• IV. Never Guess User Values: operator sign-off is a hard
  precondition to writing anything into the repo.
• V. Sieve Pipeline Integrity: the deterministic->LLM->manual
  ordering is the same pattern the control sieve uses; each pass
  either conclusively produces an artifact or hands off.

Related

• darnit-reproducibility: signal extraction module for computational experiments #414 (signal extraction module) -- the sieve's detect signals are
  the same signals this feeds.
• darnit-reproducibility: reproducibility layer + tightness taxonomy #415 (layer + tightness taxonomy) -- the produced environment
  should declare which (layer, tightness) it targets.
• darnit-reproducibility: heuristic checks for execution and build reproducibility signals #416 (heuristic checks) -- an audit that finds "no environment
  declaration" is the trigger; this issue is the remediation half.
• Determinism Tier 3: capture LLM reasoning and evidence trail in attestations, logs, and reports #421 (LLM reasoning capture) -- prerequisite for LLM tier's output
  being auditable.
• Determinism Tier 1: normally deterministic operations must be deterministic across runs #418 / Determinism Tier 2: network / side-effect failures must be distinguishable from real verdicts #419 / Determinism Tier 2: remediation must be atomic and idempotent under partial failure #420 (determinism set) -- companion; the whole point
  of a hermetic env is to make audit + remediation more
  deterministic downstream.