fix(edgebench): default CLI cohorts to shared integration source

benchmark/edgebench/README.md

@@ -3,10 +3,68 @@
 This research adapter calls SForge's native `run_agent`. Task images, prompts,
 submission cooldowns, judge isolation and scoring remain owned by the pinned
 EdgeBench checkout. Best-only adds host capture/admission scheduling; native and
-blind retain native automatic evaluation. Install SForge and Harbor in an isolated
-runner environment. Run the controller on Linux with Docker and native SForge
+blind retain native automatic evaluation. The CLI defaults to the shared
+[EdgeBench integration branch](https://github.com/huangruiteng/EdgeBench/tree/codex/edgebench-integration),
+maintained jointly by `huangruiteng` and `loopx-agent`. The canonical repository
+and branch live in [`source-default.json`](source-default.json).
+Run the controller on Linux with Docker and native SForge
 iptables permissions; a macOS Docker socket alone does not provide that boundary.
 
+## Select and freeze the native provider
+
+The first CLI entry for a log directory fetches the selected branch into
+`.edgebench-source/checkout` and atomically records its exact commit in
+`.edgebench-source/receipt.json`. Concurrent worker/judge startup shares that lock.
+Later entries reuse the frozen checkout without fetching, even if the branch or
+repository default changes. Dirty/moved source, conflicting explicit selection,
+and existing runs without a source receipt fail before model/container entry.
+Use a new log directory to adopt a newer integration revision; never repin an
+active or historical cohort.
+
+Prepare dependencies in an isolated environment before starting a study:
+
+```sh
+python -m benchmark.edgebench.source --log-dir /data/private-runs
+python -m pip install /data/private-runs/.edgebench-source/checkout harbor
+```
+
+The bootstrap selects this checkout before importing SForge, so an unrelated
+installed package or ambient `PYTHONPATH` cannot silently supply the provider.
+Worker `runtime-receipt.json` includes the verified `sforge_source`. This records
+code selection, not enabled task/scoring options or benchmark qualification.
+No dependency installation is performed automatically.
+
+Start the native judge and workers with the **same log directory**:
+
+```sh
+SFORGE_TASKS_DIR=/data/tasks python -m benchmark.edgebench.serve \
+  --log-dir /data/private-runs --port 8080
+python -m benchmark.edgebench.run \
+  --task TASK_ID --tasks-dir /data/tasks --log-dir /data/private-runs \
+  --run-id UNIQUE_ATTEMPT --worker native-goal --model MODEL --effort high \
+  --feedback native --submission-cooldown 1800 --eval-interval 300 \
+  --judge-url http://HOST:8080
+```
+
+For an alternative source, pass `--sforge-repository URL` with `--sforge-ref REF`
+or a full `--sforge-revision SHA` on the first entry (including preparation).
+For example, `--sforge-repository https://github.com/ByteDance-Seed/EdgeBench.git
+--sforge-ref refs/heads/main` selects upstream instead. These source options also
+apply to the online judge CLI; `SFORGE_LOG_DIR` is equivalent to `--log-dir` there.
+Use an explicit SHA to replay a known revision. A server in another log directory
+is a separate cohort: the launcher cannot attest a remote server's loaded code.
+
+These defaults apply to the LoopX EdgeBench CLI. Direct `sforge serve` and imported
+Python APIs remain caller-managed compatibility entrypoints; use them only with
+an explicitly pinned provider environment. Keep offline backfill pinned to the
+original scorer, or record regrading under a different source separately.
+Maintainers update the shared branch through reviewed integration work and keep
+its PR manifest current. Changing the default repository/branch requires updating
+the single source file and this protocol; adding a second contributor-specific
+default would split the source of truth again.
+
+## Worker profiles
+
 The five worker profiles are `official`, `single`, `native-goal`,
 `heartbeat-resume`, and `heartbeat-explore`. `official` uses SForge's Codex Stop
 hook and outer recovery. `single` executes once with neither. `native-goal` uses
@@ -65,7 +123,7 @@ run's slot after verified worker removal. An evaluation still in flight keeps
 that slot until its native result is terminal; stopping a solver alone does not
 free evaluator capacity. Native histories, archives and original registration
 identities remain readable, and released registrations reject new captures.
-Native/blind use ordinary `sforge serve` instead. Existing v1 cohorts remain
+Native/blind use `python -m benchmark.edgebench.serve` instead. Existing v1 cohorts remain
 frozen: this does not upgrade or reclaim slots in an already-running server.
 
 An operator may explicitly pass `--allow-resource-overcommit` for a monitored
@@ -196,7 +254,7 @@ resource policy in the study manifest before launching a new comparison.
 Use explicit `--feedback best-only` to retain strict-improvement publication, or
 `--feedback blind` as an evaluator-feedback-free control. Harbor is unchanged.
 
-For native archive submission, start ordinary `sforge serve` and use the runner
+For native archive submission, start `python -m benchmark.edgebench.serve` and use the runner
 without `--feedback` to select the defaults. Explicit
 `--feedback native --submission-cooldown 120 --eval-interval 300` restores the
 previous native timing for a new attempt. Do not edit running receipts. Game

benchmark/edgebench/online_judge.py

@@ -17,6 +17,11 @@
 from enum import StrEnum
 from pathlib import Path
 
+from benchmark.edgebench.source import bootstrap
+
+if __name__ == "__main__":
+    bootstrap(keep_log_arg=False)
+
 from fastapi import File, Form, HTTPException, Query, UploadFile
 from sforge.harness.config import load_config
 from sforge.harness.judge_server import (

benchmark/edgebench/run.py

@@ -1,4 +1,8 @@
-"""Launch one native EdgeBench trial; run the judge with native `sforge serve`."""
+"""Launch a native trial; start its judge with benchmark.edgebench.serve.
+
+CLI source options: --sforge-repository, --sforge-ref, --sforge-revision.
+Defaults and the experiment-local source lock are owned by edgebench.source.
+"""
 
 from __future__ import annotations
 
@@ -14,6 +18,11 @@
 from pathlib import Path
 from urllib.parse import urlsplit
 
+from benchmark.edgebench.source import active_source_receipt, bootstrap
+
+if __name__ == "__main__":
+    bootstrap()  # Select SForge before any provider or adapter imports.
+
 from sforge.harness.benchmark import load_benchmark
 from sforge.harness.config import SForgeConfig
 from sforge.harness.constants import get_admin_secret
@@ -352,6 +361,7 @@ def main(argv=None):
            if agent.task_entry == "loopx-planned" else {}),
         "model": args.model, "effort": args.effort, "timeout_seconds": args.timeout,
         "loopx_commit": pins[0], "runner_commit": pins[1],
+        "sforge_source": active_source_receipt(),
         **turn_envelope_receipt(explicit=args.turn_envelope),
         **({"native_goal_continue": True} if args.native_goal_continue else {}),
         "task_sha256": hashlib.sha256(task_file.read_bytes()).hexdigest(),

benchmark/edgebench/serve.py

@@ -0,0 +1,18 @@
+"""Native SForge judge using the same frozen source as EdgeBench workers."""
+from __future__ import annotations
+
+import sys
+
+from benchmark.edgebench.source import bootstrap
+
+
+def main():
+    bootstrap(keep_log_arg=False)
+    from sforge.cli import main as native_main
+
+    sys.argv.insert(1, "serve")
+    native_main()
+
+
+if __name__ == "__main__":
+    main()

benchmark/edgebench/source-default.json

@@ -0,0 +1,4 @@
+{
+  "repository": "https://github.com/huangruiteng/EdgeBench.git",
+  "ref": "refs/heads/codex/edgebench-integration"
+}

benchmark/edgebench/source.py

@@ -0,0 +1,145 @@
+"""Experiment-local SForge selection, before importing the native provider.
+
+This research adapter owns a dependency choice, not scoring or LoopX policy.
+One atomic source directory freezes the worker/judge cohort's revision.
+"""
+from __future__ import annotations
+
+import argparse
+import fcntl
+import importlib.util
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import sys
+import tempfile
+
+DEFAULT_FILE = Path(__file__).with_name("source-default.json")
+STATE_DIR = ".edgebench-source"
+RECEIPT_ENV = "LOOPX_EDGEBENCH_SOURCE_RECEIPT"
+
+
+def _git(checkout: Path, *args: str) -> str:
+    try:
+        result = subprocess.run(["git", "-C", str(checkout), *args],
+                                capture_output=True, text=True, timeout=180)
+    except subprocess.TimeoutExpired:
+        raise RuntimeError(f"EdgeBench git {args[0]} timed out") from None
+    if result.returncode:
+        # A transport error can contain a credential-bearing remote URL.
+        raise RuntimeError(f"EdgeBench git {args[0]} failed (exit {result.returncode})")
+    return result.stdout.strip()
+
+
+def _validate(state: Path) -> dict:
+    receipt = json.loads((state / "receipt.json").read_text())
+    if (not isinstance(receipt, dict)
+            or receipt.get("schema_version") != "edgebench_source_v1"
+            or not isinstance(receipt.get("commit"), str)
+            or not re.fullmatch(r"[0-9a-f]{40}", receipt["commit"])
+            or not all(isinstance(receipt.get(k), str) and receipt[k]
+                       for k in ("repository", "ref"))):
+        raise ValueError("Invalid EdgeBench source receipt; preserve it and use a new log directory")
+    checkout = state / "checkout"
+    if (_git(checkout, "rev-parse", "HEAD") != receipt["commit"]
+            or _git(checkout, "remote", "get-url", "origin") != receipt["repository"]
+            or _git(checkout, "status", "--porcelain", "--untracked-files=normal")):
+        raise ValueError("Frozen EdgeBench source changed; restore it or use a new log directory")
+    if not (checkout / "sforge/__init__.py").is_file():
+        raise ValueError("Selected source does not contain the SForge package")
+    return receipt
+
+
+def prepare_source(log_dir: Path, *, repository: str | None = None,
+                   ref: str | None = None, revision: str | None = None) -> tuple[Path, dict]:
+    """Resolve once; an existing cohort can be reused offline, never refreshed."""
+    if revision is not None and not re.fullmatch(r"[0-9a-f]{40}", revision):
+        raise ValueError("--sforge-revision requires a full lowercase commit SHA")
+    if ref is not None and revision is not None:
+        raise ValueError("Select either --sforge-ref or --sforge-revision")
+    if ref is not None and (not ref or ref.startswith("-")):
+        raise ValueError("--sforge-ref must be a nonempty Git reference")
+    if repository is not None and ref is None and revision is None:
+        raise ValueError("An alternate repository requires --sforge-ref or --sforge-revision")
+    log_dir = log_dir.resolve()
+    log_dir.mkdir(parents=True, exist_ok=True)
+    state = log_dir / STATE_DIR
+    with (log_dir / f"{STATE_DIR}.lock").open("a") as lock:
+        fcntl.flock(lock, fcntl.LOCK_EX)
+        if state.exists():
+            receipt = _validate(state)
+            for key, requested in (("repository", repository), ("ref", ref), ("commit", revision)):
+                if requested is not None and receipt[key] != requested:
+                    raise ValueError("EdgeBench selection conflicts with this cohort; use a new log directory")
+            return state / "checkout", receipt
+        if (log_dir / "runs").exists() and any((log_dir / "runs").iterdir()):
+            raise ValueError("Existing runs have no source lock; preserve them and use a new log directory")
+        defaults = json.loads(DEFAULT_FILE.read_text())
+        repository = repository or defaults["repository"]
+        ref = ref or (revision if revision else defaults["ref"])
+        with tempfile.TemporaryDirectory(prefix=".edgebench-source-", dir=log_dir) as temporary:
+            staged = Path(temporary) / "source"
+            checkout = staged / "checkout"
+            checkout.mkdir(parents=True)
+            _git(checkout, "init", "--quiet")
+            _git(checkout, "remote", "add", "origin", repository)
+            _git(checkout, "fetch", "--quiet", "--depth=1", "--", "origin", revision or ref)
+            commit = _git(checkout, "rev-parse", "FETCH_HEAD^{commit}")
+            if revision is not None and commit != revision:
+                raise ValueError("Fetched EdgeBench commit differs from the requested revision")
+            _git(checkout, "checkout", "--quiet", "--detach", commit)
+            receipt = dict(schema_version="edgebench_source_v1", repository=repository,
+                           ref=ref, commit=commit)
+            (staged / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
+            _validate(staged)
+            staged.rename(state)
+        return state / "checkout", receipt
+
+
+def bootstrap(*, keep_log_arg: bool = True) -> None:
+    """CLI-only bootstrap; imported Python APIs retain their explicit environment."""
+    parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False,
+        description="Select and freeze the shared EdgeBench source for a log directory.")
+    parser.add_argument("--log-dir", type=Path, default=os.environ.get("SFORGE_LOG_DIR"))
+    parser.add_argument("--sforge-repository")
+    parser.add_argument("--sforge-ref")
+    parser.add_argument("--sforge-revision")
+    if "--help" in sys.argv[1:] or "-h" in sys.argv[1:]:
+        parser.print_help()
+        return
+    args, remaining = parser.parse_known_args()
+    if args.log_dir is None:
+        parser.error("provide --log-dir or SFORGE_LOG_DIR to freeze the EdgeBench source")
+    try:
+        checkout, _ = prepare_source(args.log_dir, repository=args.sforge_repository,
+                                     ref=args.sforge_ref, revision=args.sforge_revision)
+    except (OSError, ValueError, RuntimeError) as error:
+        parser.error(str(error))
+    sys.path.insert(0, str(checkout))
+    os.environ["PYTHONPATH"] = os.pathsep.join(filter(None, (str(checkout), os.environ.get("PYTHONPATH"))))
+    os.environ["SFORGE_LOG_DIR"] = str(args.log_dir.resolve())
+    os.environ[RECEIPT_ENV] = str(checkout.parent / "receipt.json")
+    sys.argv[1:] = remaining + (["--log-dir", str(args.log_dir)] if keep_log_arg else [])
+    active_source_receipt()
+
+
+def active_source_receipt() -> dict | None:
+    """Prove the module Python will load belongs to the frozen source."""
+    path = os.environ.get(RECEIPT_ENV)
+    if not path:
+        return None  # Legacy imported API: the embedding caller owns its source.
+    state = Path(path).resolve().parent
+    receipt = _validate(state)
+    spec = importlib.util.find_spec("sforge")
+    if spec is None or not spec.origin or Path(spec.origin).resolve() != state / "checkout/sforge/__init__.py":
+        raise ValueError("Imported SForge differs from the experiment's source receipt")
+    return receipt
+
+
+if __name__ == "__main__":
+    # Prepare dependencies without launching a worker, judge or model call.
+    bootstrap(keep_log_arg=False)
+    if "--help" not in sys.argv[1:] and "-h" not in sys.argv[1:]:
+        print(json.dumps(active_source_receipt(), indent=2))