Sr. Staff AI/ML Engineer, 10+ years. I build multi-agent systems that ship, and I measure them before I believe them.

Built work, with receipts. Every project below runs, and every claim traces to an eval, a probe, or a logged run. The receipts that proved me wrong are here too.

A wrong number in a dashboard fails loudly. A generative system fails plausibly: a mouth trailing the audio by four frames, a cited statistic with nothing behind it. Those are invisible to a type check, obvious to a person, and different again tomorrow. So I capture human judgment as labels, compile it into thresholds, and wire it into gates that can refuse to spend.

autonomous-ads-pipeline-multimodal-evals

7 steps, 4 guards, 10 probes · 10 of 10 named gating thresholds derived from labelled exemplars · 48 exemplars graded by hand, 42 calibration scenes · 28 ad versions across 4 engines · 10 spec ads for real products

Three problems the field has not settled, and one loop that does all three. A multi-agent system I run in production shot thirty-eight ads with nobody watching, against metered video engines, and it could only spend when its own checks said yes.

1. It shoots with no director. Boards, prompts, engine calls, re-rolls and delivery ran unattended, and every request landed in an append-only ledger.
2. It grades pixels, audio and timing instead of text, against 48 exemplars I labelled by hand. The grading splits three ways.
   • Process evals check that every step ran and its gate fired before money moved.
   • Outcome evals check that every claim on screen matches the product's own live page.
   • Quality evals score taste against the golden set, re-derived for each audience.
3. It routes each brief to the engine that wins that audience. Four engines shot the same five briefs and no single engine won all five, so the winners table picks per brief, the way Perplexity picks a model per question, here for video.

The evals overruled me, too. Ten scoring models I built were thrown out in one day for disagreeing with the human labels, and a lip-sync check that shipped backwards was caught doubling the error it was built to remove.

Ranked by priority, one per repository, nothing listed twice.

1. Passing tests say nothing about what you never thought to test. Every check passed while the presenter's dark top vanished into a black background and a floating head shipped, because nobody had written that question down. Now every suite I build carries a written list of what it does not measure.
2. A success message is not a result. Every agent reported success while the files it handed over were empty, because each one looked for the others at an address that only works on one machine. Now every handoff verifies the address and every file reports its size, so an empty one cannot pass as done.

1. Chat output is not evidence. The database write, the rendered file, and the logged artifact size are the evidence.
2. Degrade loudly, never silently. Every fallback is logged; a green checkmark over an empty artifact is the worst bug in agentic systems.
3. Spend where the artifact is judged. Flagship models on deliverables, fast models on hot paths; cost per accepted deliverable beats cost per token.

• Languages: Python, SQL, TypeScript, JavaScript, Go, Bash
• Full-stack & web: React, Node.js, TypeScript, Vite, HTML/CSS, responsive web apps, streaming chat UIs
• APIs, RPC & streaming: REST, FastAPI, gRPC, WebSockets, Server-Sent Events (SSE), streaming APIs, webhooks, OAuth 2.1
• LLMs & agents: Agentic AI, multi-agent systems in production, LangGraph, LangChain, Model Context Protocol (MCP), tool and function calling, prompt and context engineering, OpenAI API, Anthropic Claude API
• Search, retrieval & RAG: RAG, vector search, semantic and hybrid search, embeddings, reranking, approximate nearest neighbor (ANN), vector databases (FAISS, Turbopuffer), relevance engineering
• NLP: Information extraction, text classification, semantic matching, thematic and qualitative analysis
• Evaluation & guardrails: Evaluation engineering, LLM-as-a-judge, golden datasets, groundedness and hallucination evals, multimodal evals, process, outcome and quality evals, eval-gated CI, guardrails, output validation, prompt-injection testing, PII handling, A/B testing, ablation studies, statistical analysis
• Observability & tracing: Langfuse, LangSmith, Datadog, Prometheus, Grafana, distributed tracing, structured logging, metrics and alerting, LLM observability
• Voice & speech: Conversational AI, voice AI, ASR, TTS, turn detection, voice cloning (ElevenLabs), Twilio, PSTN telephony
• Video & creative AI: Text-to-video and video generation, avatar video (HeyGen), ffmpeg, ad tech, creative testing
• Machine learning: PyTorch, deep learning, fine-tuning (Hugging Face, DistilBERT), computer vision (OpenCV, MediaPipe, CNNs), reinforcement learning (multi-agent RL, PPO), ONNX int8 quantization, generative AI
• Infra & MLOps: MLOps, LLMOps, Docker, Kubernetes, Helm, CI/CD (GitHub Actions, Harness), AWS (EKS), Azure, self-hosted LLM serving (Ollama), PostgreSQL, Snowflake
• Models: Claude (Anthropic), GPT (OpenAI), Llama, Voyage AI

Sr. Staff / Principal AI engineering: agentic systems, LLM evals, and the guardrails that make them safe to run unattended. San Francisco, or remote.

Role titles I map to: Staff AI Engineer, Principal AI Engineer, Applied AI Engineer, Machine Learning Engineer, AI/ML Engineer, Forward Deployed Engineer (FDE).

The fastest way to evaluate me is to not take my word for it. No accounts, no API keys, no GPU. Needs python3 and ffmpeg:

git clone https://github.com/jameswniu/autonomous-ads-pipeline-multimodal-evals
cd autonomous-ads-pipeline-multimodal-evals && pip install -r requirements.txt
python3 evals/derive.py

It prints every gating threshold in that harness beside the labelled pass and the labelled reject that bracket it, recomputes the shipped exemplars rather than reciting them, and exits nonzero if any constant has drifted outside its own evidence.

Hiring, or want the walkthrough: LinkedIn | jameswnarch@gmail.com