Skip to content

Add explicit AIRO agentic forecasting harness - #30

Closed
jordan-c-olm wants to merge 1 commit into
forecastingresearch:mainfrom
jordan-c-olm:codex/airo-agentic-harness
Closed

jordan-c-olm wants to merge 1 commit into
forecastingresearch:mainfrom
jordan-c-olm:codex/airo-agentic-harness

Conversation

@jordan-c-olm

Copy link
Copy Markdown

Adds utils.llm.agentic.run(...) so callers can explicitly use AIRO's existing pi forecasting harness from fri-utils. The caller supplies prompts and forecast schemas; results include cells, evidence, usage, and trace artifacts. Existing ModelRun.get_response(...) calls and registry identities remain unchanged.

This PR copies the core from AIRO commit 3ece40dfa87961c4fb03f826d007ed4e3d42b1f9. AIRO keeps its separate version. ForecastBench runs, DWH integration, and consumer migrations are deferred.

Key decisions

  • Preserve pi 0.99.2 and its tool loop. Node is required only for explicit agentic calls.
  • Package the extension and locked npm manifests; install pi outside the Python package. No automatic downloads or env-file discovery.
  • Keep the existing result contract and loop options. Fix three concrete failure cases: normalize credential-directory paths, stop tool descendants on POSIX timeouts, and reject debriefs whose validator crashes.
  • Preserve Python 3.10 support and source attribution.

Validation

  • make lint passes.
  • All 30 agentic tests pass with real pinned pi and a scripted localhost model, including 2 validator-error subtests. No paid model calls.
  • The installed wheel passes from outside the checkout with PYTHONPATH unset, exercising packaged TypeScript, the Python research tool, and validator retries.
  • Full suite: 201 passed, 19 paid integration tests skipped, 2 subtests passed with -W 'ignore:Python 3.14 will, by default:DeprecationWarning'. Plain make test has three existing archive-test failures on Python 3.13; all three reproduce on unchanged main.
  • CI covers Python 3.10/3.13 and Node 22.19.0.

Reported usage still covers only the returned attempt and excludes Tavily charges. Turn and timeout limits are not dollar caps. Other platforms retain process-only timeout termination.

Setup and runbook

See the setup guide for copy-paste commands and one bounded example.

  1. Install fri-utils and Node ≥22.19. Copy the packaged npm manifests to an external runtime directory and run npm ci.
  2. Set FORECASTER_PI to that runtime's cli.js. Set FORECASTER_MODEL to the selected pi provider/model-id.
  3. Set the selected provider key (for example, OPENAI_API_KEY or ANTHROPIC_API_KEY) and TAVILY_API_KEY. Load local .env explicitly with load_dotenv(".env", override=False), or inject deployment secrets under those names. GCP API_KEY_* secrets must be mapped to these environment names; configure_api_keys(from_gcp=True) does not pass its in-memory keys to pi.
  4. Run FRI_REQUIRE_PI=1 pytest tests/unit/agentic for offline validation. For a live check, use the guide's explicit 1-question, 180-second, zero-retry example and inspect errors, missing cells, evidence, and actual cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant