Rebuild the deployment from nothing, and fix what that turned up

[davidmckayv]

What this changes

The EKS deployment this was proven on was destroyed to nothing — cluster, database, registry — and
rebuilt from an empty account following the docs as written, to find out what a person actually hits.
Then the release was driven through it: Bots, routines, and one Bot handing work to another, happy
path and crappy path. Everything below is something that run turned up.

A conversation that used this release's features died for good

The one that matters. After a handoff, a routine firing and an ask_person in the same channel,
every later turn in it failed — including "what is 2 plus 2" — and the only thing the person was
shown was Tool result is missing for tool call call_gIyW298bIC9e32UiB6d3MIXt, a raw SDK error
class. The conversation could not be recovered.

Reading the stored thread back from the platform explains it. The tool result is stored before
its own tool call, for exactly the tools this release added:

tool	call at	result at
computer_navigate	5	6	correct
message_bot	12	9	result three messages ahead of its call
create_routine	20	17	inverted
ask_person	31	27	inverted
report_refusal	23	24	correct

plus a result at 37 whose call was never stored at all. repairUnansweredToolCalls exists to keep a
thread provider-valid and decided "answered" by collecting every result regardless of position, so
an inverted result marked its call answered, nothing was repaired, and the provider — which matches a
result to the call above it — threw while converting the prompt. That fails the conversation, not the
turn.

A call is now answered only by a result that follows it. An early result is moved to sit after
its call rather than replaced, because it is the real one; only a call with no result anywhere gets
the apology text. A result whose call never appears is dropped. Checked against the real 45-message
thread: seven ordering faults before, none after, all three real results kept. Two tests added, both
failing against the previous implementation.

This makes the client resilient. It does not stop the platform storing them in that order, which is
worth someone's attention separately.

The chart's own examples pointed at a host that does not exist. All five files under
charts/openbot/ci/ set gatewayWsUrl: wss://gateway.cloud.copilotkit.ai, which does not resolve,
and apiUrl: https://api.cloud.copilotkit.ai, which resolves to something answering 404. A target
file is meant to be the shortest honest starting point for a cluster; copying one produced a
deployment whose realtime plane pointed at a host that has never existed. .env.example has carried
the working pair all along.

The release was named for a feature the changelog did not mention. Unreleased covered routines,
screenshots, unread dots and thirty smaller things, and said nothing about one Bot handing work to
another. Written now — and rewritten once, because the first draft said the answer comes back into
the asking conversation. It does not, and it is not supposed to: the addressed Bot answers for
itself. Driving it is what caught a changelog entry that would have shipped describing the opposite
of the behaviour.

Bot-to-bot grants had no screen, and the docs said they did. bot is a grant kind the store and
the API both accept, it had no catalogue entry, nothing in app/src rendered it, and
store.listForAgent filtered to mcp and skill so the read path dropped it. The only way to
switch on the capability this release is named for was a curl.

It is now on the Bot's own screen, under Bots it may ask: a switch per other Bot, with the
directional pair said out loud, because that is the one thing here that is easy to get backwards.
GET /api/agents/:id/handoff reports the grants, whether the person looking may change them, and
whether the deployment's caps leave the capability on at all. That last one is reported rather than
inferred: with the caps at zero a grant is a row nothing will read, and a switch wired to nothing is
worse than no switch. Writing reuses the grant endpoint every other grant uses, so the audit row and
the refusals are the ones already in place.

The Bot screen called every Bot "Browser Bot". The heading was that string, with the right Bot
already resolved two lines away. It is the defect the route default beneath it was fixed for and
says so in its own comment: a Bot name written into a route is wrong on every fork but the one it
came from.

The roster overlapped itself whenever a Bot was open. A card is a fixed 144px in a
four-column grid, so the cards sat on top of each other as soon as the column holding them was
narrower than the card, which opening a Bot does at any window size. The tracks now follow the
container.

The Audit screen named no Bot on either handoff row. Its Bot column reads payload.bot and shows
a dash without it; both handoff events carried the acting Bot only as payload.from, while
agent.escalated twenty lines away sets it. So on a screen headed "Every action a Bot took", the two
handoff rows were the only ones naming no Bot. Fixed, with a test on each event that fails without
the fix. Found by reading the trail on the live deployment, not from the code.

The docs never mentioned Kubernetes. docs/README.md indexed eight pages and none was the chart;
charts/openbot/README.md was linked from nowhere; the deployment page's AWS note offered ECS and
stopped. The chart README also began at helm upgrade --install and assumed three things it does not
create, one written down nowhere: the published image carries linux/amd64 only, so an arm64
node group cannot pull it. The cluster this was rebuilt on is now written down as the shortest thing
that produces a working one.

Two template fallbacks had no drift test. check-new-values-keys.ts asserted the
--reuse-values fallback against values.yaml for the two handoff caps and nothing else, because it
could only read a rendered env var. The routines schedule and the culler's deadline are CronJob
fields, and the routines schedule had already been wrong once, by a factor of five. Both are checked
now, read off the rendered workload by component label. Verified by drifting both deliberately.

Stale measurements. The image is 1.4 GB, not the 5.3 GB written when the container work landed;
595 MB of it is still Firefox and WebKit, so the reason given was right and the number was not. The
README counted four chart targets and ci/ holds five. Helm 4 renamed --atomic, so the rollback
advice now names the flag on both majors.

What the drive proved

Install from zero on the first attempt, --atomic, twelve resources. Migrations ran as a pre-install
hook and deleted themselves; the container correctly did not also migrate on start against an
external database, and started no embedded PostgreSQL. Default gp3 StorageClass, both computer
volumes Bound. A Bot answered, was given a sandbox computer of its own, opened a real page and put
the screenshot in the transcript.

Routines end to end: refused while ungranted, created by asking once granted, and every single minute refused by the 15-minute floor without creating anything. One Bot handed work to another and
the addressed Bot answered a question only it could have seen. ask_person reached the person.
Self-grant and unknown-Bot grants both refused, 403, named. All thirteen chart install-time refusals
fire with the right message.

What this run could not prove

config.singleUser was on, so every request is one administrator. The non-admin paths — the
enumeration oracle, the admin-versus-user roster divergence, the role a hop is resolved as — are
structurally unreachable in that configuration, and are covered by tests rather than by this run
(bot-access.test.ts, agent-handoff.test.ts, plugin-routes.test.ts).

networkPolicy.enabled leaves the routines CronJob unfenced on main: networkpolicy.yaml selects
server and computer and nothing selects component: routines. Not fixed here because #277
already owns that file and two PRs editing it would collide — but #277 predates routines, and its own
new check fails on rebase. Reproduced: merge origin/main into #277, render eks-sandbox with
networkPolicy.enabled, and its gate reports none selects ci-openbot-routines.