Independent Researcher · @lloitesa013
Contact: lloitesa@gmail.com · OpenReview: ~Dana_Kim1
I build the layer that decides when an AI system's output can be trusted — and what happens when it can't: a runtime watcher that abstains and escalates, failure taxonomies that localize why models fail, and a verifier that gates what a self-improving agent may adopt.
AI를 더 똑똑하게 만드는 대신, AI의 출력을 언제 믿어도 되는지 — 믿으면 안 될 때 무엇을 해야 하는지 — 를 결정하는 레이어를 만듭니다: abstain/escalate하는 런타임 감시자(watcher), 실패 원인을 국소화하는 택소노미, 자기개선 에이전트의 수정을 게이트하는 검증기.
I do not build more impressive AI outputs; I build the evidence-bound judgment above them.
| Looking for… | Go to | The one number |
|---|---|---|
| Agent reliability / hallucination control (production) | agent-reliability-bench — Layer 1 watcher: trust score, ABSTAIN, ESCALATE, audit log. Offline tour: python demo.py |
zero-shot reading judge 0.946±0.014 vs 0.789 tuned groundedness rule, 0 unsafe passes (sealed 300-trace bench); abstention lifts 0.84 → 0.91 |
| Driving evaluation & failure analysis (research) | bench2drive-failure-taxonomy + xmod-interpretable-driving | LEAD/TransFuser-V6 DS 93.36 reproduced over 220 routes + per-route failure taxonomy, expert-calibrated; X-MoD = an honest, claims-bounded closed-loop study |
| Verified self-improvement (frontier) | agent-reliability-bench — Layer 2 VSI-0: hidden-test + regression gate over an agent's own {prompt, rule, memory, tool} changes | naive self-modification ships a broken tool 59% of the time, the executing verifier 0% (24 decisions; replicated on MBPP + HumanEval; embodied ACCEPT on CARLA, p=6e-8) |
| 2026 | Shipped |
|---|---|
| May 27–31 | Financial Agent Evidence OS — claim contracts + release gates (early iteration of the thesis) |
| May 28 – Jun 1 | Cognitive OS API — rule-based decision gate ALLOW/DEGRADE/DENY/HANDOFF (early iteration) |
| Jun 12–19 | Bench2Drive failure taxonomy — 220-route reproduction (DS 93.36) + first per-route failure map with expert calibration |
| Jun 14 | Mantle audit devkit — the same evidence-packet pattern applied to smart contracts (hackathon transfer test) |
| Jun 22 | Merged upstream: autonomousvision/carla_garage PR #117 — Bench2Drive integration fix for numpy ≥ 1.24 |
| Jun 25 | X-MoD — interpretable driving policy; bounded negative result published as such, with paper |
| Jul 1–9 | agent-reliability-bench — the watcher (sealed benchmark, R1–R7) + VSI-0 verified self-improvement (R8–R31) |
Works. Reproducible. Measured. Understandable. Bounded.
The public portfolio should not rely on "I am impressive" as a claim. It should show running code, reproducible evidence, measurable gates, clear explanations, and visible limits.
Read: Verification Principles Role map: Role Positioning
| Work | It works | Reproducible | Measured | Boundary |
|---|---|---|---|---|
| Agent-Reliability Watcher × VSI-0 | python demo.py — full offline watcher tour, no key; vsi_gate.py = the verifier as a pure-stdlib library |
Sealed spec (BENCH_SPEC/METRICS/CLAIMS, win conditions fixed before results) + 300-trace data + every result names its script | Judge 0.946±0.014 vs rule 0.789, 0 unsafe passes; abstention 0.84→0.91; naive ships broken self-mods 59% vs verifier 0% (MBPP/HumanEval replicated) | Synthetic bench, 7B models, single GPU; not production, not SOTA; honest negatives are first-class results |
| Bench2Drive Failure Taxonomy | Reproducible SOTA baseline plus per-route failure map | Two harnesses, per-route evidence (~400 raw route JSONs), calibration tooling | LEAD/TransFuser-V6 DS 93.36 vs PDM-Lite 95.71; failures split fixable / structural / infra via privileged expert |
Single-GPU/seed; not a new SOTA; per-route labels refined to failure rates by follow-up (R19–R21) |
| X-MoD on Bench2Drive | Closed-loop B2D agent, paper, eval harness | Code plus 3-seed eval protocol, claims/non-claims | Unseen route 24330: 100% completion, 0 collisions across 3 seeds; gate/safety corr 0.80 |
Does not generalize uniformly; completion uses a classical control stack |
| Cognitive OS API | FastAPI evidence viewer and /evidence/report |
Seed benchmark, baselines, conformance, adversarial runner | Seed-suite gate accuracy 100% (authored against the rule set); external-style challenge pack 37.5% with failure analysis — the honest number | Rule-based early prototype; not AGI, not global LLM safety |
| Financial Agent Evidence OS | Streamlit evidence viewer and release gates | Full unit suite plus release/audit/verifier commands | 118 tests OK, 2 skipped; unfavorable market results preserved in the record |
Not financial advice, not live trading, not future-return prediction |
| Angelos Reproducibility Kit | Public packet validator | Frozen anchors and manifest checks | ped_surprise/v10, merge_cutin/v11, 40 seeds each |
CARLA protocol packet, not deployment readiness |
One system, two layers, one question: when should an AI agent's output — or its own self-modification — be trusted?
- Layer 1, the watcher (runtime): groundedness × relevance trust scoring, ABSTAIN when unsure, ESCALATE to a human with a reason, audit log for every decision. Evidence: sealed 300-trace benchmark, judge 0.946±0.014 vs tuned rule 0.789, 0 unsafe passes.
- Layer 2, VSI-0 (research): every self-modification is UNTRUSTED until it clears a hidden test + regression check — because reward-hacked "improvements" are the norm, not the exception (59% naive deployment rate of broken tools; verifier 0%).
- The two layers are one thesis at two levels of autonomy: watch the outputs, then gate the self-modifications.
Repo: agent-reliability-bench
- Cognitive OS API — 2026-05 rule-based decision
gate (
ALLOW/DEGRADE/DENY/HANDOFF): the HANDOFF-to-human primitive that became the watcher's ESCALATE. - Financial Agent Evidence OS — 2026-05 claim contracts + sealed evidence packets for a financial agent, negative results preserved.
- Mantle audit devkit — the same proof-carrying-evidence pattern transferred to smart-contract audits (hackathon).
My systems grew out of a single question:
How can an AI system decide, restrain, explain, and verify its actions under explicit human values?
- Flow-Based AI Ethics Model: ethical cognition as a decision flow.
- X-MoD: explainable routing over explicit driving values.
- Angelos OS: safety-gated judgment above autonomous driving policies.
- Cognitive OS: user-owned policy and decision gates above LLMs.
- Evidence OS: benchmark and release gates for claims made by agents.
- Agent-Reliability Watcher × VSI-0: the judgment layer at two levels of autonomy — watch the agent's outputs, then gate the agent's self-modifications.
Read more: AI Judgment Lineage
X-MoD and Angelos OS are the research roots of this portfolio. They test the same judgment-layer idea in a harder setting: autonomous driving, where action, restraint, explanation, and failure analysis must be tied to scenario evidence.
- X-MoD: explainable routing over safety, legality, comfort, and efficiency.
- Angelos OS / SynOptic Core: a model-agnostic judgment layer above driving policies, with safety gates and protocol-frozen evaluation.
- Public repos: X-MoD on Bench2Drive (closed-loop interpretable driving, paper, and eval harness) and Bench2Drive Failure Taxonomy (reproducible SOTA baseline plus per-route failure map); plus the offline X-MoD Visual Paper Kit and Angelos Reproducibility Kit.
- Upstream: carla_garage PR #117 (Bench2Drive numpy ≥ 1.24 fix, merged 2026-06-22).
- Boundary: these kits are sanitized research packets, not broad safety or deployment claims. The follow-up reliability work showed single-run per-route failure labels flip on re-run — so claims are stated at failure-rate level (agent-reliability-bench R19–R21).
- AI Verification Portfolio
- AI Judgment Lineage
- Verification Principles
- Role Positioning
- Product Map
- Public / Private Split
- Research Kits
Python - PyTorch/transformers - CARLA/Bench2Drive - FastAPI - Streamlit - CI gates - sealed benchmark specs - evidence manifests - release gates - decision envelopes - trace privacy
This portfolio does not claim external adoption, global SOTA status, AGI, live trading readiness, investment performance, complete AI safety, or any unbounded guarantee. Claims are scoped to the included reference architectures, benchmark suites, and published evidence artifacts.