I try to build tests that AI can't game. So far the AI keeps winning, and the reasons are the interesting part.
From September to October 2026 I led JUMP: one person and three AI systems (ChatGPT Pro, Codex and Claude) trying to build a benchmark of autonomous physics research that models couldn't saturate without genuinely getting better.
rounds of design and review ...... about 48
model episodes ................... 162
designs that survived ............ 0
Why every design failed is in the paper: Who Writes the Answer Key? (on GitHub)
- NOETHER: turns physics intuitions into testable predictions, so evidence decides what survives.
- KeplerLab: a physics-informed neural network for orbital dynamics, built to show where it fails and why.
- Artemis II simulator: an Earth–Moon N-body simulation of the Artemis II mission profile (run it in your browser).
34 pull requests across 14 open-source repositories. Codex wrote most of the code; I pick the issue, scope the change and see it through review. A few:
- OpenInference #3662: tracing context now survives coroutines and generators.
- OpenTelemetry Python #5631: named metric views without an instrument name now raise an error, with regression tests.
- Hugging Face tokenizers #2547: Split deserialization honors the configured regex backend.
- OpenTelemetry GenAI #670: retrieval spans record document relevance scores, valid zeros included.
First-year Computing and Information Technologies student at RIT Kosovo. Elsewhere: X · ORCID