Skip to content

INTERFACES.md

Latest commit

 

History

History
142 lines (124 loc) · 8.29 KB

INTERFACES.md

File metadata and controls

142 lines (124 loc) · 8.29 KB
Raw

Interfaces between the task tooling and the run lifecycle

compression-bench reuses ServerBench's framework (MLE-Bench serverbench/): tasks are rendered from sources by taskctl, Harbor composes each run's container, the harness drives the agent CLI inside it, and the hidden verdict runs once, host-side, after the deadline. This page fixes the seams so each half can be built and tested on its own.

The repository layout is in the root README and the harness's in compression_bench/README.md.

A task directory

Sources: contract.yaml and assets/. Everything else is generated by taskctl render and checked by taskctl check.

contract.yaml                 SOURCE  identity, model pin, capacity, treatment, policy, budget
assets/public/                SOURCE  public perplexity + decode inputs and configs
assets/hidden/                SOURCE  hidden perplexity, decode and correctness-gate inputs (Verifier image only)
assets/debug-9b/              SOURCE  optional 9B debug inputs (public Checker only)
assets/tokenizer/             SOURCE  official tokenizer files (tokenizer.json restored by fetch_tokenizers.py)
assets/model_facts.json       SOURCE  the checkpoint's structure, stated in the instruction
task.toml instruction.md README.md MODE.md                       GENERATED
environment/{Dockerfile,docker-compose.yaml,check_submission.py,model_inventory.py,
             submission_guard.py,inference.py,platform/,public-checker/}  GENERATED
tests/{Dockerfile,docker-compose.yaml,test.sh,<evaluator>.py,evaluator-assets/,platform/}  GENERATED

contract.yaml keys the lifecycle reads

task_id: qwen3_8_27b_12g_standard_1xH200NVL
task_hash: "8a661afb"            # sha256(task_id)[:8]
model: {id: Qwen/Qwen3.8-27B, revision: <sha>, slug: qwen3_8_27b, local_name: Qwen3.8-27B}
capacity_gib: 12
treatment: standard
hardware: 1xH200NVL
budget: {agent_hours: 8, verifier_timeout_s: 3600, public_check_timeout_s: 3600, build_timeout_s: 3600}
debug_model: {id: Qwen/Qwen3.5-9B, local_name: Qwen3.5-9B} | null

Compose variables (both environment/ and tests/ compose files)

variable meaning
COMPRESSION_BENCH_RUN_DIR the run root; holds .submission.lock, work/, submission/
COMPRESSION_BENCH_MODEL_DIR the pinned BF16 checkpoint (mounted read-only at /input/model)
COMPRESSION_BENCH_GPU_DEVICE the one GPU index for main, public-checker and verifier
COMPRESSION_BENCH_HOST_UID, COMPRESSION_BENCH_HOST_GID main runs as the host user
COMPRESSION_BENCH_DEBUG_MODEL_DIR only with the debug overlay compression_bench/scripts/agent/debug-model-compose.yaml

In-container paths: /input/model (ro), /app/work (rw, $HOME=/app/work/.agent-home), /app/submission (rw in main, ro elsewhere), lock /run/compression-bench-submission.lock (ro bind of $COMPRESSION_BENCH_RUN_DIR/.submission.lock), checker socket /run/compression_bench/checker.sock. The agent container's /tmp is the run's work/.tmp.

main's command is flock -s /run/compression-bench-submission.lock sleep infinity: it holds a shared flock on the lock for as long as the container lives, and submission_guard.py ready is its health check. The verifier's test.sh takes an exclusive flock and refuses to grade while any main container is alive: the single copy of the weights is never graded while writable.

A run directory: compression_bench/evolve_space/<ns>/<task>/<hash8>-<stamp>-<hex6>/

identity.json  journal.jsonl  SUMMARY.json  container.json  harbor_environment.json
.submission.lock  work/ (→ /app/work)  submission/ (→ /app/submission)
secrets/ (staged credentials, refreshed in place, mounted ro)
prompts/  agent.<kind>.<n>.log  agent.stream.jsonl  trajectory/  resources.jsonl  TIMING.json
CREDIT_HOLD.json (only while the run is held for provider credit)
verifier/ (hidden verdict: <stage>.json|.log for check, smoke, ppl, accuracy, decode, score; reward.txt)
checkpoint-verdicts/<vN>/ (the hidden verdict of each ./version checkpoint)
public-checks/checks.jsonl (every public check; Checker-only mount)
METRICS.json (after the run: every metric, and what the cleanup deleted)

<ns> is evolve for a production run and evolve-scratch for --scratch and --dry-run.

Stages (lifecycle/run.py)

create → claim GPU → model identity → image freshness → build workspace → start container (Harbor) → boot proof → develop (agent CLI loop: continuation / cold restart until the deadline) → finish (final version capture, stop the environment) → final verdict (hidden Verifier via Harbor on tests/, after the lock is free; writes FINAL.json) → checkpoint verdicts (final_validate/tools/verdict_checkpoints.py) → release (the GPU claim) → record (SUMMARY.json, evaluation_progress/<task>.csv) → harvest (every metric into METRICS.json, read back, then every weight a verdict measured is deleted; lifecycle/harvest.py).

Lifecycle details

  • Harbor 0.21's own egress allowlist (its egress-control sidecar) confines main: the agent reaches only its provider (api.anthropic.com, statsig.anthropic.com, sentry.io for Claude; chatgpt.com, auth.openai.com, api.openai.com for Codex; or the host of a provider route's ANTHROPIC_BASE_URL alone). The sidecar gets a gost config with a 600 s read timeout (compression_bench/scripts/agent/egress-gost.yaml). The public Checker keeps network_mode: none; the Verifier runs with no-network.
  • The agent CLI is the host's native binary, bind-mounted read-only at /opt/compression_bench/bin/<cli> (its SHA-256 is recorded); the base image is not rebuilt per CLI version. Codex needs the native binary from its npm package's vendor/ tree (the codex on PATH is a Node launcher).
  • Run directory extras: compose/ (copies of every compose file Harbor used, plus compose.json to replay docker compose down), harbor/, harbor-verifier/, harbor_env.<command>.log, final_validation.log, and verifier.previous.<n>/ when a verdict is re-run.
  • Lineage files (compression_output/<lineage>/): LINEAGE.json, versions.jsonl, <lineage>_vN/{VERSION.json,MANIFEST.json,files/}, FINAL.json, FINAL.attempts.jsonl, FINAL_VALIDATION/<run id>/, CHECKPOINT_VERDICTS/<vN>/, checkpoint_verdicts.jsonl, CHECKPOINT_NOTES/ (versions the scheduler's janitor cleaned), .live-run.json while a run writes to it, METRICS/<run id>.json and TRAJECTORY/<run id>/ after it. Weight checkpoints (reflink snapshots) live in compression_output/.objects/<lineage>/vN/; when they are measured and deleted is in OPERATIONS.md §3. Each version's VERSION.json keeps its metrics whatever happens to its weights: manifest identity, composition (weight bytes by suffix), public_check (the Checker's result for exactly its file listing) and verdict.
  • Public check history: the Checker appends every request and response to <run>/public-checks/checks.jsonl (server.py --history), a mount only the Checker has, so the agent can neither read nor rewrite it.
  • After the run (harvest.py, also compression_bench/scripts/harvest_runs.py): <run>/METRICS.json and <lineage>/METRICS/<run id>.json (schema compression_bench.run_metrics/1) are written and read back before any weight is deleted; what they hold and what is deleted is in OPERATIONS.md §3. evolve --keep-weights skips the deletion.
  • budget.build_timeout_s bounds Harbor's build + start of both the agent and the Verifier environments.
  • Checkpoints from the HF cache (relative symlinks into blobs/) get the blob directory mounted read-only where the links resolve, for main, the public Checker and the Verifier. A checkpoint directory without weight files (*.safetensors, *.gguf, *.bin) is refused as MODEL_MISMATCH.
  • Per-host bindings live in the gitignored compression_bench/scripts/env.local.sh (OPERATIONS.md §1).
  • Image freshness: before starting, the run checks the task's environment/Dockerfile base against compression_bench/environments/images.json with image_freshness.py check <key>; a stale or missing managed image is HARNESS_SETUP_FAILED. A base not listed there is journalled as unmanaged.