diff --git a/evals/harbor/README.md b/evals/harbor/README.md new file mode 100644 index 000000000..fadb7cb2f --- /dev/null +++ b/evals/harbor/README.md @@ -0,0 +1,142 @@ +# Harbor adapter for Corbits Code + +Thin Harbor `BaseInstalledAgent` that runs headless product Corbits: + +```text +corbits exec --cwd … --config … --provider … --model … + --dangerously-skip-permissions +``` + +No second agent loop. Credentials are translated from Harbor kwargs/env into a +temporary `settings.json` passed with `--config` — product Corbits still does +not read API keys from the environment. + +## Layout + +| Path | Role | +| ----------------------- | ------------------------------------------------------ | +| `argv.py` | Pure settings + argv builders (unit-tested, no Harbor) | +| `session.py` | Parse harvested `~/.corbits` usage (unit-tested) | +| `agent.py` | `Corbits` installed agent (requires Harbor at import) | +| `tasks/trivial/` | Minimal smoke task (`hello.txt`) | +| `tests/test_argv.py` | Argv/settings unit tests | +| `tests/test_session.py` | Session usage harvest unit tests | + +## Prerequisites + +1. **Harbor CLI** installed in the host Python env (`pip install harbor` / uv). +2. **Linux ELF `corbits` binary** for the task container (Darwin host builds + will not run inside Linux Docker). Acquire one of: + - Build on Linux: `bun run build:bin` → `dist/corbits` + - Release / CI tarball that unpacks a `corbits` binary +3. **git** inside the task image (adapter also installs it via Harbor system + packages). Corbits storage requires git — there is no git-less fallback. +4. **Provider API key** for the model under test. +5. **Provider `base_url`** — stock Harbor `-m xai/…` infers `https://api.x.ai/v1`. Override only when the catalog default is wrong. The adapter fail-closes if nothing resolves. + +## Secrets / credentials + +Pass a key through Harbor agent kwargs or env. The adapter writes it into the +temp settings file only: + +| Source | Notes | +| ----------------------- | ---------------------------------------------- | +| `api_key=` agent kwarg | Preferred for one-off runs | +| `CORBITS_API_KEY` | Generic adapter env | +| `{PROVIDER}_API_KEY` | Only the selected provider, e.g. `XAI_API_KEY` | +| Harbor model connection | Falls back to `model_connection.api_key` | + +### Required base URL + +`providers..baseURL` is always written. Resolve it via one of: + +| Source | Notes | +| ----------------------- | ----------------------------------------------------------------------------------------- | +| `base_url=` agent kwarg | Preferred for one-off runs | +| Harbor configured URL | `model_connection.configured_base_url` (`CORBITS_BASE_URL` / `XAI_BASE_URL` / …) | +| Harbor inferred catalog | `model_connection.base_url` — e.g. xAI → `https://api.x.ai/v1` when an API key is present | +| `CORBITS_BASE_URL` | Adapter env (also feeds Harbor configured URL) | + +Stock `harbor run -m xai/grok-4.5` with `XAI_API_KEY` therefore does **not** need an explicit `base_url`. The adapter still fail-closes if nothing resolves. + +Do **not** point Harbor cells at the grok-cli OAuth proxy (`https://cli-chat-proxy.grok.com/v1`). That URL only accepts OAuth tokens from `~/.corbits/xai-auth.json`, which Harbor containers do not have. Profile-qualified product cells (`xai/`, `codex/`) are OAuth; use an API-key provider name (`xai`, `openai`, …) inside Harbor. + +Example values: + +| Cell | Example `base_url` | +| ----------------------------- | ------------------------------------------------------- | +| xAI API key | `https://api.x.ai/v1` (Harbor infers this) | +| Product OAuth / grok-cli path | Not usable in Harbor containers | +| OpenAI-compatible | e.g. `https://api.openai.com/v1` or your cell's gateway | + +Optional: `shell_timeout_ms=` → `shell.timeoutMs` in settings. + +Default provider/model when Harbor does not pass `provider/model`: **xai** / +**grok-4.5**. Codex cells typically use `--model openai/` (or pass +`provider=` / `model=` kwargs). + +## Linux binary acquisition + +The adapter installs the binary onto PATH from **one** of: + +| Kwarg | Behavior | +| --------------------- | ------------------------------------------------- | +| `corbits_binary_path` | Upload a host file into `/usr/local/bin/corbits` | +| `corbits_binary_url` | `curl` a raw binary URL | +| `corbits_tarball_url` | `curl` + extract; expects a `corbits` file inside | + +Archive vs raw binary is detected from the URL suffix (`.tar.gz`, `.tgz`, +`.tar`) — the install script does not call `file(1)`. + +If none are set and `corbits` is not already on PATH in the environment, +`install()` raises with this README pointer. + +## Invoke + +From the repo root (so `evals.harbor.agent` is importable): + +```bash +# Unit tests (no Harbor package required; pytest may be absent) +PYTHONPATH=. python3 -m unittest evals.harbor.tests.test_argv evals.harbor.tests.test_session -v + +# Dry-run trivial task (needs Harbor CLI + Linux binary + API key) +export CORBITS_API_KEY=… # or XAI_API_KEY=… +harbor run \ + -p evals/harbor/tasks/trivial \ + -a evals.harbor.agent:Corbits \ + -m xai/grok-4.5 \ + --ak corbits_binary_path=/absolute/path/to/linux/corbits +``` + +Equivalent kwargs via Harbor job config: + +```yaml +agents: + - name: evals.harbor.agent:Corbits + kwargs: + provider: xai + model: grok-4.5 + api_key: ${CORBITS_API_KEY} + base_url: https://api.x.ai/v1 + corbits_binary_path: /absolute/path/to/linux/corbits + # or: corbits_tarball_url: https://…/corbits-linux.tar.gz +``` + +Harbor's own trial/job timeout is the only wall-clock cap. Do **not** wrap +`harbor run` in a parent tool timeout — killing the CLI mid-trial wastes the +tokens already spent and leaves no verifier result. + +After each trial Harbor copies `/logs/agent`. The adapter tars `~/.corbits` +there as `corbits-home.tar.gz` and fills `AgentContext` token fields from +session `metadata.json`. `cost_usd` is set only when the harvested +`models-pricing.json` cache has that model; unknown cells stay `null` rather +than inventing a rate. Tool traces are under +`agent/corbits-home/.corbits/projects/…/turns-*.jsonl`. + +## Known gaps + +- Trivial Harbor dry-run has been run locally (`hello.txt`, verifier reward 1.0) + with OpenCode Go `mimo-v2.5`. Cost-capped Terminal-Bench smoke is the parent + ticket **CL-6922** (the old CL-6924 slice ticket was canceled). +- This adapter does not convert Corbits trajectories into Harbor ATIF. Usage + and the session tar are enough to audit a trial without that conversion. diff --git a/evals/harbor/__init__.py b/evals/harbor/__init__.py new file mode 100644 index 000000000..492167254 --- /dev/null +++ b/evals/harbor/__init__.py @@ -0,0 +1,8 @@ +"""Harbor installed-agent adapter for Corbits Code (CL-6923). + +Import the agent class as ``evals.harbor.agent:Corbits`` when running Harbor. +Pure helpers live in ``argv`` and ``session`` and are unit-tested without +the Harbor package. +""" + +__all__ = ["argv", "session"] diff --git a/evals/harbor/agent.py b/evals/harbor/agent.py new file mode 100644 index 000000000..26f8bc73d --- /dev/null +++ b/evals/harbor/agent.py @@ -0,0 +1,321 @@ +"""Harbor ``BaseInstalledAgent`` bridge that runs headless ``corbits exec``. + +Thin adapter: install a Linux Corbits binary + git, write a temporary +settings.json from Harbor kwargs/env, then exec with +``--dangerously-skip-permissions``. No second agent loop. + +Requires the Harbor package at import time (normal for Harbor plugins). +Unit tests import ``evals.harbor.argv`` only. +""" + +from __future__ import annotations + +import json +import shlex +from pathlib import Path, PurePosixPath +from typing import Any, override + +from harbor.agents.installed.base import ( + BaseInstalledAgent, + with_prompt_template, +) +from harbor.agents.model_connection import ModelConnectionSpec +from harbor.environments.base import BaseEnvironment +from harbor.models.agent.context import AgentContext +from harbor.models.trial.paths import EnvironmentPaths + +from evals.harbor.argv import ( + api_key_env_names, + build_exec_argv, + build_settings, + resolve_base_url, +) +from evals.harbor.session import usage_from_logs_dir + +_REMOTE_BIN_DIR = PurePosixPath("/usr/local/bin") +_REMOTE_CORBITS = _REMOTE_BIN_DIR / "corbits" +_REMOTE_SETTINGS = PurePosixPath("/tmp/corbits-settings.json") +_OUTPUT_FILENAME = "corbits.txt" +_DEFAULT_PROVIDER = "xai" +_DEFAULT_MODEL = "grok-4.5" +_DEFAULT_TASK_CWD = "/app" + + +class Corbits(BaseInstalledAgent): + """Installed-agent adapter: Harbor → ``corbits exec`` (yolo for that process).""" + + # No default_provider: ``-m xai/grok-4.5`` and ``-m openai/…`` both work. + # CORBITS_* is tried before Harbor's per-provider catalog env names. + MODEL_CONNECTION = ModelConnectionSpec( + api_key_envs=("CORBITS_API_KEY",), + base_url_envs=("CORBITS_BASE_URL",), + ) + + def __init__( + self, + *args: Any, + provider: str | None = None, + model: str | None = None, + api_key: str | None = None, + base_url: str | None = None, + shell_timeout_ms: int | None = None, + corbits_binary_url: str | None = None, + corbits_tarball_url: str | None = None, + corbits_binary_path: str | None = None, + task_cwd: str = _DEFAULT_TASK_CWD, + **kwargs: Any, + ) -> None: + super().__init__(*args, **kwargs) + self._provider_override = provider + self._model_override = model + self._api_key_override = api_key + self._base_url = base_url + self._shell_timeout_ms = shell_timeout_ms + self._corbits_binary_url = corbits_binary_url + self._corbits_tarball_url = corbits_tarball_url + self._corbits_binary_path = corbits_binary_path + self._task_cwd = task_cwd + self._last_exit_code: int | None = None + self._resolved_provider: str | None = None + self._resolved_model: str | None = None + + @staticmethod + @override + def name() -> str: + # Custom import path agent — do not require AgentName enum membership. + return "corbits" + + @override + def get_version_command(self) -> str | None: + # The CLI has no --version flag; record the help banner as a best-effort + # identity string. Harbor ignores failures here. + return "corbits --help | head -n 1" + + def _resolve_provider_model(self) -> tuple[str, str]: + if self._provider_override and self._model_override: + return self._provider_override, self._model_override + + if self.model_name and "/" in self.model_name: + provider, model = self.model_name.split("/", 1) + return ( + self._provider_override or provider, + self._model_override or model, + ) + + if self.model_name: + return ( + self._provider_override or _DEFAULT_PROVIDER, + self._model_override or self.model_name, + ) + + return ( + self._provider_override or _DEFAULT_PROVIDER, + self._model_override or _DEFAULT_MODEL, + ) + + def _resolve_api_key(self, provider: str) -> str: + if self._api_key_override: + return self._api_key_override + + # Adapter-only translation: Harbor env/kwargs → settings.json. + # Product Corbits still sees credentials only via --config. + for name in api_key_env_names(provider): + value = self._get_env(name) + if value: + return value + + access = self.model_connection + if getattr(access, "api_key", None): + return str(access.api_key) + + raise ValueError( + "No API key for Corbits Harbor adapter. Pass api_key=… in agent " + "kwargs, set CORBITS_API_KEY or {PROVIDER}_API_KEY, " + "or configure Harbor model credentials. Keys are written into a " + "temporary settings.json for --config only." + ) + + def _resolve_base_url(self) -> str: + access = self.model_connection + return resolve_base_url( + override=self._base_url, + configured=getattr(access, "configured_base_url", None), + env_url=self._get_env("CORBITS_BASE_URL"), + inferred=getattr(access, "base_url", None), + ) + + @override + async def install(self, environment: BaseEnvironment) -> None: + # Corbits storage requires git in the environment (no git-less fallback). + await self.ensure_system_dependencies( + environment, ("git", "curl", "ca_certificates", "tar") + ) + + check = await environment.exec(command="command -v corbits >/dev/null 2>&1") + if check.return_code == 0: + self.logger.debug("corbits already on PATH") + return + + if self._corbits_binary_path: + await self._install_from_host_path(environment, self._corbits_binary_path) + return + + url = self._corbits_binary_url or self._corbits_tarball_url + if url: + await self._install_from_url(environment, url) + return + + raise RuntimeError( + "Corbits Harbor adapter needs a Linux ELF binary. Pass one of " + "corbits_binary_path (host file uploaded into the env), " + "corbits_binary_url, or corbits_tarball_url in agent kwargs. " + "Build with `bun run build:bin` on Linux, or use a release tarball. " + "See evals/harbor/README.md." + ) + + async def _install_from_host_path( + self, environment: BaseEnvironment, host_path: str + ) -> None: + source = Path(host_path).expanduser() + if not source.is_file(): + raise FileNotFoundError(f"corbits_binary_path not found: {source}") + remote = _REMOTE_CORBITS.as_posix() + await environment.upload_file(source, remote) + await self.exec_as_root( + environment, + command=f"chmod 755 {shlex.quote(remote)} && corbits --help >/dev/null", + ) + + async def _install_from_url(self, environment: BaseEnvironment, url: str) -> None: + quoted_url = shlex.quote(url) + remote = _REMOTE_CORBITS.as_posix() + # Detect archive by URL suffix — do not use file(1) (absent in trivial images). + path_part = url.lower().split("?", 1)[0] + is_archive = path_part.endswith((".tar.gz", ".tgz", ".tar")) + if is_archive: + command = ( + "set -euo pipefail; " + f"tmp=$(mktemp -d); " + f"curl -fsSL {quoted_url} -o \"$tmp/artifact\"; " + "tar -xaf \"$tmp/artifact\" -C \"$tmp\"; " + "bin=$(find \"$tmp\" -type f -name corbits | head -n 1); " + "if [ -z \"$bin\" ]; then " + " echo 'tarball did not contain a corbits binary' >&2; exit 1; " + "fi; " + f"install -m 755 \"$bin\" {shlex.quote(remote)}; " + "corbits --help >/dev/null" + ) + else: + command = ( + "set -euo pipefail; " + f"curl -fsSL {quoted_url} -o {shlex.quote(remote)}; " + f"chmod 755 {shlex.quote(remote)}; " + "corbits --help >/dev/null" + ) + await self.exec_as_root(environment, command=command) + + async def _harvest_corbits_home(self, environment: BaseEnvironment) -> None: + # Harbor copies /logs/agent after run(); put the session there so + # populate_context_post_run can read tokens without a second download. + dest = (EnvironmentPaths.agent_dir / "corbits-home.tar.gz").as_posix() + command = ( + "set +e; " + 'home="${HOME:-/root}"; ' + 'if [ ! -d "$home/.corbits" ] && [ -d /root/.corbits ]; then home=/root; fi; ' + f"dest={shlex.quote(dest)}; " + 'if [ -d "$home/.corbits" ]; then ' + ' tar -czf "$dest" -C "$home" .corbits && chmod a+r "$dest"; ' + "fi; " + "exit 0" + ) + try: + await environment.exec(command=command) + except Exception: + self.logger.warning("failed to tar Corbits ~/.corbits into agent logs") + + @override + def populate_context_post_run(self, context: AgentContext) -> None: + meta = dict(context.metadata or {}) + if self._last_exit_code is not None: + meta["exit_code"] = self._last_exit_code + meta["agent"] = self.name() + + provider = self._resolved_provider or self._provider_override or _DEFAULT_PROVIDER + model = self._resolved_model or self._model_override or _DEFAULT_MODEL + usage = usage_from_logs_dir( + self.logs_dir, + model_ids=(f"{provider}/{model}", model), + ) + if usage.n_input_tokens is not None: + context.n_input_tokens = usage.n_input_tokens + context.n_cache_tokens = usage.n_cache_tokens + context.n_output_tokens = usage.n_output_tokens + if usage.cost_usd is not None: + context.cost_usd = usage.cost_usd + if usage.session_relpath is not None: + meta["session_dir"] = usage.session_relpath + if usage.pricing_model is not None: + meta["pricing_model"] = usage.pricing_model + context.metadata = meta + + @override + @with_prompt_template + async def run( + self, + instruction: str, + environment: BaseEnvironment, + context: AgentContext, + ) -> None: + provider, model = self._resolve_provider_model() + self._resolved_provider = provider + self._resolved_model = model + api_key = self._resolve_api_key(provider) + settings = build_settings( + provider, + model, + api_key, + base_url=self._resolve_base_url(), + shell_timeout_ms=self._shell_timeout_ms, + ) + + remote_config = _REMOTE_SETTINGS.as_posix() + await self._upload_config_text( + environment, + content=json.dumps(settings, indent=2) + "\n", + remote_path=remote_config, + filename="settings.json", + ) + + # Persist a redacted copy under agent logs for debugging. + redacted = json.loads(json.dumps(settings)) + redacted["providers"][provider]["apiKey"] = "***" + (self.logs_dir / "settings.redacted.json").write_text( + json.dumps(redacted, indent=2) + "\n" + ) + + argv = build_exec_argv( + cwd=self._task_cwd, + config_path=remote_config, + provider=provider, + model=model, + prompt=instruction, + binary="corbits", + ) + command = ( + "set -euo pipefail; " + + " ".join(shlex.quote(part) for part in argv) + + f" 2>&1 | tee {shlex.quote((EnvironmentPaths.agent_dir / _OUTPUT_FILENAME).as_posix())}; " + "exit ${PIPESTATUS[0]}" + ) + + (self.logs_dir / "command.txt").write_text(command + "\n") + + # Bypass Harbor's _exec so the real exit code survives a failure; it + # raises NonZeroAgentExitCodeError without exposing the code. + try: + result = await environment.exec(command=command) + self._last_exit_code = result.return_code + if result.return_code != 0: + raise self._classify_exec_error(command, result) + finally: + await self._harvest_corbits_home(environment) diff --git a/evals/harbor/argv.py b/evals/harbor/argv.py new file mode 100644 index 000000000..8478ef5fc --- /dev/null +++ b/evals/harbor/argv.py @@ -0,0 +1,105 @@ +"""Pure Corbits settings + exec argv builders for the Harbor adapter. + +No Harbor imports — unit-testable in isolation. +""" + +from __future__ import annotations + +from typing import Any + + +def build_settings( + provider: str, + model: str, + api_key: str, + *, + base_url: str, + shell_timeout_ms: int | None = None, +) -> dict[str, Any]: + """Build a Corbits ``settings.json`` dict for ``--config``. + + Shape matches product settings: ``providers..{apiKey,models,baseURL}`` + plus optional ``shell.timeoutMs``. Credentials go into the file only — Corbits + does not read API keys from the environment. + + ``base_url`` is required and always written as ``baseURL``. Callers must + resolve it (agent kwargs / ``CORBITS_BASE_URL`` / Harbor model connection); + this helper does not invent a default URL. + """ + provider_entry: dict[str, Any] = { + "apiKey": api_key, + "models": [model], + "baseURL": base_url, + } + + settings: dict[str, Any] = { + "providers": { + provider: provider_entry, + }, + } + if shell_timeout_ms is not None: + settings["shell"] = {"timeoutMs": shell_timeout_ms} + return settings + + +def api_key_env_names(provider: str) -> tuple[str, str]: + """Env vars consulted for the API key, in priority order. + + Deliberately no cross-provider fallback: a key for another provider is + never silently written into ``providers..apiKey``. + """ + return ("CORBITS_API_KEY", f"{provider.upper()}_API_KEY") + + +def resolve_base_url( + *, + override: str | None, + configured: str | None, + env_url: str | None, + inferred: str | None, +) -> str: + """Pick the Corbits ``providers.

.baseURL`` from adapter sources. + + Precedence: explicit kwarg, Harbor ``configured_base_url`` (env such as + ``XAI_BASE_URL`` / ``CORBITS_BASE_URL``), adapter env, then Harbor's + inferred catalog default (e.g. xAI → ``https://api.x.ai/v1`` when an API + key is present). Does not invent the grok-cli OAuth proxy URL. + """ + for value in (override, configured, env_url, inferred): + if value: + return value + raise ValueError( + "No base URL for Corbits Harbor adapter. Pass base_url=… in agent " + "kwargs, set CORBITS_BASE_URL, or configure Harbor model connection " + "base URL. Corbits settings require providers..baseURL." + ) + + +def build_exec_argv( + *, + cwd: str, + config_path: str, + provider: str, + model: str, + prompt: str, + binary: str = "corbits", +) -> list[str]: + """Build the exact ``corbits exec`` argv for a Harbor trial. + + Always includes ``--dangerously-skip-permissions`` so the headless process + cannot block on operator approval. + """ + return [ + binary, + "exec", + "--cwd", + cwd, + "--config", + config_path, + "--provider", + provider, + "--model", + model, + "--dangerously-skip-permissions", + prompt, + ] diff --git a/evals/harbor/session.py b/evals/harbor/session.py new file mode 100644 index 000000000..38159a92e --- /dev/null +++ b/evals/harbor/session.py @@ -0,0 +1,186 @@ +"""Parse harvested Corbits session usage for Harbor AgentContext. + +No Harbor imports — unit-testable in isolation. Does not invent a dollar +figure when models.dev pricing is missing (unlike product faremeter's +fallback rate). +""" + +from __future__ import annotations + +import json +import sys +import tarfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any + + +_HOME_TAR = "corbits-home.tar.gz" +_UNPACKED_DIR = "corbits-home" + + +@dataclass(frozen=True) +class SessionUsage: + n_input_tokens: int | None + n_cache_tokens: int | None + n_output_tokens: int | None + cost_usd: float | None + session_relpath: str | None + pricing_model: str | None + + +def unpack_home_tar(logs_dir: Path) -> Path | None: + """Extract ``corbits-home.tar.gz`` under ``logs_dir`` if present.""" + tar_path = logs_dir / _HOME_TAR + if not tar_path.is_file(): + unpacked = logs_dir / _UNPACKED_DIR + return unpacked if unpacked.is_dir() else None + + dest = logs_dir / _UNPACKED_DIR + dest.mkdir(parents=True, exist_ok=True) + kwargs: dict[str, Any] = {} + if sys.version_info >= (3, 12): + kwargs["filter"] = "data" + with tarfile.open(tar_path, "r:gz") as archive: + archive.extractall(dest, **kwargs) + return dest + + +def usage_from_unpacked_home( + unpacked: Path, + *, + model_ids: tuple[str, ...], +) -> SessionUsage: + """Read the newest session ``metadata.json`` and optional pricing cache.""" + metadata_path = _newest_metadata(unpacked) + if metadata_path is None: + return SessionUsage(None, None, None, None, None, None) + + token_usage = _token_usage_from_metadata(metadata_path) + if token_usage is None: + return SessionUsage( + None, + None, + None, + None, + _relpath(unpacked, metadata_path.parent), + None, + ) + + input_tokens = token_usage["input"] + output_tokens = token_usage["output"] + cache_read = token_usage["cacheRead"] + cache_write = token_usage["cacheWrite"] + thinking = token_usage["thinking"] + + n_input = input_tokens + cache_read + cache_write + n_cache = cache_read + cache_write + n_output = output_tokens + thinking + + pricing_model, prices = _lookup_pricing(unpacked, model_ids) + cost_usd = None + if prices is not None: + cost_usd = ( + input_tokens * prices["inputPricePerToken"] + + output_tokens * prices["outputPricePerToken"] + + cache_read * prices["cacheReadPricePerToken"] + ) + + return SessionUsage( + n_input, + n_cache, + n_output, + cost_usd, + _relpath(unpacked, metadata_path.parent), + pricing_model, + ) + + +def usage_from_logs_dir(logs_dir: Path, *, model_ids: tuple[str, ...]) -> SessionUsage: + unpacked = unpack_home_tar(logs_dir) + if unpacked is None: + return SessionUsage(None, None, None, None, None, None) + return usage_from_unpacked_home(unpacked, model_ids=model_ids) + + +def _newest_metadata(root: Path) -> Path | None: + found = [path for path in root.rglob("metadata.json") if path.is_file()] + if not found: + return None + found.sort(key=lambda path: (path.stat().st_mtime, str(path))) + return found[-1] + + +def _token_usage_from_metadata(path: Path) -> dict[str, int] | None: + try: + payload = json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(payload, dict): + return None + raw = payload.get("tokenUsage") + if not isinstance(raw, dict): + return None + try: + return { + "input": _as_int(raw.get("input")), + "output": _as_int(raw.get("output")), + "cacheRead": _as_int(raw.get("cacheRead")), + "cacheWrite": _as_int(raw.get("cacheWrite")), + "thinking": _as_int(raw.get("thinking")), + } + except TypeError: + return None + + +def _as_int(value: object) -> int: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise TypeError + return int(value) + + +def _lookup_pricing( + unpacked: Path, + model_ids: tuple[str, ...], +) -> tuple[str | None, dict[str, float] | None]: + cache_path = unpacked / ".corbits" / "cache" / "models-pricing.json" + if not cache_path.is_file(): + nested = list(unpacked.rglob("models-pricing.json")) + cache_path = nested[0] if nested else cache_path + if not cache_path.is_file(): + return None, None + try: + payload = json.loads(cache_path.read_text()) + except (OSError, json.JSONDecodeError): + return None, None + if not isinstance(payload, dict): + return None, None + models = payload.get("models") + if not isinstance(models, dict): + return None, None + for model_id in model_ids: + entry = models.get(model_id) + prices = _parse_prices(entry) + if prices is not None: + return model_id, prices + return None, None + + +def _parse_prices(entry: object) -> dict[str, float] | None: + if not isinstance(entry, dict): + return None + try: + return { + "inputPricePerToken": float(entry["inputPricePerToken"]), + "outputPricePerToken": float(entry["outputPricePerToken"]), + "cacheReadPricePerToken": float(entry["cacheReadPricePerToken"]), + } + except (KeyError, TypeError, ValueError): + return None + + +def _relpath(root: Path, path: Path) -> str: + try: + return str(path.relative_to(root)) + except ValueError: + return str(path) diff --git a/evals/harbor/tasks/trivial/environment/Dockerfile b/evals/harbor/tasks/trivial/environment/Dockerfile new file mode 100644 index 000000000..69c2a9924 --- /dev/null +++ b/evals/harbor/tasks/trivial/environment/Dockerfile @@ -0,0 +1,11 @@ +FROM ubuntu:24.04 + +WORKDIR /app + +# git is required by Corbits storage; curl helps the adapter fetch a binary. +RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + git \ + && rm -rf /var/lib/apt/lists/* diff --git a/evals/harbor/tasks/trivial/instruction.md b/evals/harbor/tasks/trivial/instruction.md new file mode 100644 index 000000000..9e2447dee --- /dev/null +++ b/evals/harbor/tasks/trivial/instruction.md @@ -0,0 +1,7 @@ +Create a file named `hello.txt` in the working directory (`/app`) whose entire contents are exactly: + +```text +hello +``` + +Do not add a trailing blank line beyond a single trailing newline. No other files are required. diff --git a/evals/harbor/tasks/trivial/solution/solve.sh b/evals/harbor/tasks/trivial/solution/solve.sh new file mode 100755 index 000000000..5f5ff3590 --- /dev/null +++ b/evals/harbor/tasks/trivial/solution/solve.sh @@ -0,0 +1,3 @@ +#!/bin/bash +set -euo pipefail +printf 'hello\n' > /app/hello.txt diff --git a/evals/harbor/tasks/trivial/task.toml b/evals/harbor/tasks/trivial/task.toml new file mode 100644 index 000000000..2f5f59c6a --- /dev/null +++ b/evals/harbor/tasks/trivial/task.toml @@ -0,0 +1,19 @@ +version = "1.0" + +[metadata] +author_name = "Corbits" +author_email = "dev@corbits.dev" +difficulty_explanation = "Write a single file with a fixed string" +category = "smoke" +tags = ["trivial", "corbits", "harbor"] + +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 300.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 2048 diff --git a/evals/harbor/tasks/trivial/tests/test.sh b/evals/harbor/tasks/trivial/tests/test.sh new file mode 100755 index 000000000..4e870fed3 --- /dev/null +++ b/evals/harbor/tasks/trivial/tests/test.sh @@ -0,0 +1,12 @@ +#!/bin/bash +set -euo pipefail + +mkdir -p /logs/verifier + +if [ -f /app/hello.txt ] && [ "$(cat /app/hello.txt)" = "hello" ]; then + echo 1 > /logs/verifier/reward.txt + exit 0 +fi + +echo 0 > /logs/verifier/reward.txt +exit 1 diff --git a/evals/harbor/tests/__init__.py b/evals/harbor/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/evals/harbor/tests/test_argv.py b/evals/harbor/tests/test_argv.py new file mode 100644 index 000000000..30418a2a4 --- /dev/null +++ b/evals/harbor/tests/test_argv.py @@ -0,0 +1,178 @@ +"""Unit tests for Harbor adapter argv/settings helpers (no Harbor package).""" + +from __future__ import annotations + +import unittest + +from evals.harbor.argv import ( + api_key_env_names, + build_exec_argv, + build_settings, + resolve_base_url, +) + + +class BuildSettingsTests(unittest.TestCase): + def test_requires_base_url_and_always_emits_base_url(self) -> None: + settings = build_settings( + "xai", + "grok-4.5", + "sk-test", + base_url="https://api.x.ai/v1", + ) + self.assertEqual( + settings, + { + "providers": { + "xai": { + "apiKey": "sk-test", + "models": ["grok-4.5"], + "baseURL": "https://api.x.ai/v1", + } + } + }, + ) + + def test_base_url_and_shell_timeout(self) -> None: + settings = build_settings( + "openai", + "gpt-5", + "sk-openai", + base_url="https://example.com/v1", + shell_timeout_ms=120_000, + ) + self.assertEqual( + settings["providers"]["openai"], + { + "apiKey": "sk-openai", + "models": ["gpt-5"], + "baseURL": "https://example.com/v1", + }, + ) + self.assertEqual(settings["shell"], {"timeoutMs": 120_000}) + + def test_omits_shell_when_unset(self) -> None: + settings = build_settings( + "codex", + "gpt-5.3-codex", + "sk", + base_url="https://api.openai.com/v1", + ) + self.assertEqual( + settings["providers"]["codex"]["baseURL"], + "https://api.openai.com/v1", + ) + self.assertNotIn("shell", settings) + + +class BuildExecArgvTests(unittest.TestCase): + def test_exact_order_includes_skip_permissions(self) -> None: + argv = build_exec_argv( + cwd="/app", + config_path="/tmp/corbits-settings.json", + provider="xai", + model="grok-4.5", + prompt="Write hello.txt", + ) + self.assertEqual( + argv, + [ + "corbits", + "exec", + "--cwd", + "/app", + "--config", + "/tmp/corbits-settings.json", + "--provider", + "xai", + "--model", + "grok-4.5", + "--dangerously-skip-permissions", + "Write hello.txt", + ], + ) + + def test_custom_binary(self) -> None: + argv = build_exec_argv( + cwd="/work", + config_path="/cfg.json", + provider="xai", + model="grok-4.5", + prompt="hi", + binary="/usr/local/bin/corbits", + ) + self.assertEqual(argv[0], "/usr/local/bin/corbits") + self.assertEqual(argv[-2:], ["--dangerously-skip-permissions", "hi"]) + self.assertNotIn("--force", argv) + + +class ApiKeyEnvNamesTests(unittest.TestCase): + def test_only_generic_and_selected_provider(self) -> None: + self.assertEqual( + api_key_env_names("openai"), ("CORBITS_API_KEY", "OPENAI_API_KEY") + ) + + def test_no_cross_provider_fallback(self) -> None: + names = api_key_env_names("xai") + self.assertNotIn("OPENAI_API_KEY", names) + self.assertNotIn("ANTHROPIC_API_KEY", names) + + +class ResolveBaseURLTests(unittest.TestCase): + def test_override_wins(self) -> None: + self.assertEqual( + resolve_base_url( + override="https://override.example/v1", + configured="https://configured.example/v1", + env_url="https://env.example/v1", + inferred="https://api.x.ai/v1", + ), + "https://override.example/v1", + ) + + def test_inferred_catalog_default_when_unconfigured(self) -> None: + self.assertEqual( + resolve_base_url( + override=None, + configured=None, + env_url=None, + inferred="https://api.x.ai/v1", + ), + "https://api.x.ai/v1", + ) + + def test_configured_beats_inferred(self) -> None: + self.assertEqual( + resolve_base_url( + override=None, + configured="https://cli-chat-proxy.grok.com/v1", + env_url=None, + inferred="https://api.x.ai/v1", + ), + "https://cli-chat-proxy.grok.com/v1", + ) + + def test_empty_sources_are_skipped(self) -> None: + self.assertEqual( + resolve_base_url( + override="", + configured=None, + env_url="https://from-env.example/v1", + inferred="https://api.x.ai/v1", + ), + "https://from-env.example/v1", + ) + + def test_raises_when_nothing_resolves(self) -> None: + with self.assertRaises(ValueError) as ctx: + resolve_base_url( + override=None, + configured=None, + env_url=None, + inferred=None, + ) + self.assertIn("base URL", str(ctx.exception)) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/harbor/tests/test_session.py b/evals/harbor/tests/test_session.py new file mode 100644 index 000000000..656c0ec7a --- /dev/null +++ b/evals/harbor/tests/test_session.py @@ -0,0 +1,161 @@ +"""Unit tests for Harbor adapter session harvest (no Harbor package).""" + +from __future__ import annotations + +import json +import os +import tarfile +import tempfile +import unittest +from pathlib import Path + +from evals.harbor.session import usage_from_logs_dir, usage_from_unpacked_home + + +def _write_session( + root: Path, + *, + token_usage: dict[str, int], + pricing: dict[str, object] | None = None, + session_id: str = "sess-1", +) -> Path: + session_dir = root / ".corbits" / "projects" / "demo" / session_id + session_dir.mkdir(parents=True) + (session_dir / "metadata.json").write_text( + json.dumps({"tokenUsage": token_usage, "pendingOperations": []}) + "\n" + ) + (session_dir / "turns-0001.jsonl").write_text( + json.dumps({"role": "assistant", "content": [], "timestamp": 1}) + "\n" + ) + if pricing is not None: + cache_dir = root / ".corbits" / "cache" + cache_dir.mkdir(parents=True, exist_ok=True) + (cache_dir / "models-pricing.json").write_text(json.dumps(pricing) + "\n") + return session_dir + + +class UsageFromUnpackedHomeTests(unittest.TestCase): + def test_maps_cumulative_usage_and_known_pricing(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + _write_session( + root, + token_usage={ + "input": 100, + "output": 20, + "cacheRead": 50, + "cacheWrite": 10, + "thinking": 5, + }, + pricing={ + "timestamp": 1, + "models": { + "xai/grok-4.5": { + "inputPricePerToken": 0.000003, + "outputPricePerToken": 0.000015, + "cacheReadPricePerToken": 0.00000075, + } + }, + }, + ) + usage = usage_from_unpacked_home( + root, model_ids=("xai/grok-4.5", "grok-4.5") + ) + self.assertEqual(usage.n_input_tokens, 160) + self.assertEqual(usage.n_cache_tokens, 60) + self.assertEqual(usage.n_output_tokens, 25) + self.assertEqual(usage.pricing_model, "xai/grok-4.5") + self.assertAlmostEqual(usage.cost_usd or 0.0, 0.0006375) + self.assertIsNotNone(usage.session_relpath) + self.assertTrue( + (root / usage.session_relpath / "turns-0001.jsonl").is_file() + ) + + def test_cost_stays_none_when_model_missing_from_cache(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + _write_session( + root, + token_usage={ + "input": 10, + "output": 2, + "cacheRead": 0, + "cacheWrite": 0, + "thinking": 0, + }, + pricing={"timestamp": 1, "models": {}}, + ) + usage = usage_from_unpacked_home(root, model_ids=("opencode-go/mimo-v2.5",)) + self.assertEqual(usage.n_input_tokens, 10) + self.assertEqual(usage.n_output_tokens, 2) + self.assertIsNone(usage.cost_usd) + self.assertIsNone(usage.pricing_model) + + def test_missing_session_returns_empty_usage(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + usage = usage_from_unpacked_home(Path(tmp), model_ids=("xai/grok-4.5",)) + self.assertIsNone(usage.n_input_tokens) + self.assertIsNone(usage.cost_usd) + self.assertIsNone(usage.session_relpath) + + def test_picks_newest_metadata_when_multiple_sessions(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + _write_session( + root, + token_usage={ + "input": 1, + "output": 1, + "cacheRead": 0, + "cacheWrite": 0, + "thinking": 0, + }, + session_id="old", + ) + newer = _write_session( + root, + token_usage={ + "input": 9, + "output": 3, + "cacheRead": 0, + "cacheWrite": 0, + "thinking": 0, + }, + session_id="new", + ) + newer_meta = newer / "metadata.json" + newer_stat = newer_meta.stat() + os.utime( + newer_meta, + (newer_stat.st_atime + 10, newer_stat.st_mtime + 10), + ) + usage = usage_from_unpacked_home(root, model_ids=()) + self.assertEqual(usage.n_input_tokens, 9) + self.assertIn("new", usage.session_relpath or "") + + +class UsageFromLogsDirTests(unittest.TestCase): + def test_unpacks_tar_then_reads_usage(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + staging = Path(tmp) / "staging" + staging.mkdir() + _write_session( + staging, + token_usage={ + "input": 4, + "output": 1, + "cacheRead": 0, + "cacheWrite": 0, + "thinking": 0, + }, + ) + logs_dir = Path(tmp) / "logs" + logs_dir.mkdir() + tar_path = logs_dir / "corbits-home.tar.gz" + with tarfile.open(tar_path, "w:gz") as archive: + archive.add(staging / ".corbits", arcname=".corbits") + usage = usage_from_logs_dir(logs_dir, model_ids=()) + self.assertEqual(usage.n_input_tokens, 4) + self.assertEqual(usage.n_output_tokens, 1) + unpacked = logs_dir / "corbits-home" / ".corbits" + self.assertTrue(unpacked.is_dir()) diff --git a/evals/public/README.md b/evals/public/README.md index 028615c3e..87b370758 100644 --- a/evals/public/README.md +++ b/evals/public/README.md @@ -56,8 +56,9 @@ real public issue?** | -------------------------------- | --------------------------------------------------------- | | Corbits@Grok patch on instance X | Yes (this smoke) | | % resolved on SWE-bench Lite | Only after official Docker eval on a frozen instance list | -| vs Claude Code on TB2 | Harbor adapter (not this script) | +| vs Claude Code on TB2 | Harbor adapter: `evals/harbor/` | ## Related - Product gates: `evals/capability/` +- Harbor adapter: `evals/harbor/`