fix(chart): wait for MySQL before jobs-manager starts (backend#2913)

[LukasWodka]

Closes tracebloc/backend#2913. Found by the e2e journey, on a real run.

The defect

jobs-manager opens its database connection during initialisation and exits when it fails, so a MySQL a few seconds behind it crashloops the container:

ERROR Jobs manager failed to initialize against https://stg-api.tracebloc.io/
      (CLIENT_ENV=stg, edge=841): 2003 (HY000): Can't connect to MySQL server
      on 'mysql-client:3306' (111)
Warning  BackOff  Back-off restarting failed container api

Errno 111 is connection refused — nothing is listening yet. It is none of the four causes that error message helpfully lists, which is why it cost a real investigation to separate from backend#2492.

It recovers on its own, and that is what makes it expensive. By the time anyone looks the pod is healthy and the reason survives only in the previous container's log. On the journey the leg had already failed.

The fix, and the thing it uncovered

An init container that waits for mysql-client:3306. An init container rather than a retry in the app: the ordering is a property of the pod spec, it needs no image change, and it reaches every edge on the next chart upgrade rather than the next jobs-manager release.

And initContainers: was gated on hostPath.enabled. An edge without hostPath rendered no init containers at all, so a wait added naively inside that block would have been absent exactly where it was needed. The gate now covers only init-writable-data, which is what it was ever for; initContainers: is unconditional, as MySQL is.

Rendered both ways:

hostPath=false ->  ['wait-for-mysql', 'api', 'pods-monitor-container']
hostPath=true  ->  ['wait-for-mysql', 'init-writable-data', 'api', 'pods-monitor-container']

Bounded, and it fails loudly. An unbounded wait converts a crashloop into a pod that hangs in Init forever — quieter, and strictly worse, because a crashloop at least announces itself. On expiry it exits non-zero naming what it waited for, so the pod's events say wait-for-mysql rather than a driver error three layers down.

Verification

The guard renders rather than grepping the template, because the bug is not "the text is missing" but "the container does not run" — a text search would have found the block while an edge without hostPath ran nothing.

the wait is gated on hostPath   ->  FAIL hostPath.enabled=false: first initContainer is 'api'
                                    OK   hostPath.enabled=true
the expiry message is removed   ->  FAIL: the wait has no expiry message
restored                        ->  OK (both modes)

make check green; automount-token-explicit green (14 pod specs, 13 templates); chart-version-guard green. Chart bumped 1.9.90 -> 1.9.91, version and appVersion equal.

One trap worth passing on

The guard's first version used printf '%s' "$out" | grep -q PATTERN and reported failure on a successful match. grep -q exits the moment it matches, printf is killed by SIGPIPE (141), and set -o pipefail returns that. The check failed exactly when the thing it looked for was present. It is a case now, and the reason is written down beside it — this pattern is common enough in this repo's scripts to be worth a wider look.

---

Note

Medium Risk
Changes jobs-manager pod startup ordering on every install/upgrade; misconfiguration could delay or block rollout for up to 300s, though behavior is bounded and heavily guarded by rendered-manifest tests.

Overview
Fixes a startup race where jobs-manager could crashloop on first boot when MySQL was not listening yet (connection refused on mysql-client:3306), then look healthy by the time someone investigated.

The chart adds an unconditional wait-for-mysql init container that runs before the main containers. It probes mysql-client:3306 with a 300s bounded Python socket.create_connection loop and fails loudly on timeout. initContainers: is no longer tied to hostPath.enabled—only init-writable-data stays behind that gate—so CSI / non–hostPath installs get the wait too (they previously rendered no inits at all).

The wait uses the tracebloc/jobs-manager image (non-root USER in the image) instead of busybox, with no runAsUser pins anywhere in the pod, preserving OpenShift arbitrary-UID admission alongside runAsNonRoot.

Verification: helm unit tests now assert inits by name (not index [0]); a new drift guard jobs-manager-waits-for-mysql.sh renders both hostPath modes and checks ordering, expiry messaging, and no UID pins when hostPath is off. Chart version 1.9.91 → 1.9.92.

^{Reviewed by Cursor Bugbot for commit 7861fb2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

Your OpenShift analysis on 8a9dfcc is right, and I built the same fix independently — but I think the unconditional drop trades an OpenShift-only failure for one on every other platform, and I would rather put the evidence in front of you than push over the guard you added to encode the decision.

The measurement. runAsNonRoot: true at pod level does not supply an identity; it only forbids uid 0. With no runAsUser anywhere, the kubelet reads the effective user from the image:

$ docker image inspect busybox:1.35 --format "{{.Config.User}}"
[]                       # empty -> runs as root

Verified on your head, all three preconditions together:

pod-level   runAsNonRoot=True   runAsUser=<none>
init wait-for-mysql  runAsUser=<none>  image=busybox:1.35

So on plain Kubernetes — including the k3d installer path, the primary install — the kubelet should refuse the container with container has runAsNonRoot and image will run as root, and jobs-manager never starts at all. Your comment says "nothing is loosened by omitting it here", which is true of security and is the part I want to be clear I agree with; the consequence I think it misses is admission, which needs an identity from somewhere and now has none.

What I could not do: run it. The local k3d cluster is down (127.0.0.1:6550: connection refused), so this is a measured precondition plus documented kubelet behaviour, not an executed reproduction. Worth one kubectl apply before trusting me.

Two remedies, and I now prefer the second:

1. What I had built — jobsManager.waitForMysql.pinRunAsUser (default true, 1000; OpenShift operators set false). Both platforms work, but OpenShift needs an operator action, and it fails the guard you just added — deliberately, since the guard encodes the opposite invariant.
2. Better: keep your invariant and change the image. The reason api needs no UID is that its image carries a non-root USER. If wait-for-mysql used the jobs-manager image (a python -c TCP check instead of nc -z), it would need no pin on either platform and your guard would stay exactly as written. I could not verify that image's USER — it is not pulled here — so I have not written it.

Not pushing anything: your guard is a design decision and this needs your call, not my override. Happy to implement (2) if you confirm the jobs-manager image runs non-root.

For the record, the rest of my pass on this PR did land earlier (d0d5250): init containers asserted by name rather than ordinal slot, and jobs-manager-waits-for-mysql.sh armed in DRIFT_GUARDS. Your head is green on all of it — helm lint clean, helm unittest 645/645, drift 39/39.

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4a31686. Configure here.}

[aptracebloc]

Reviewed the mechanism against the rendered chart, not just the description — it holds up well.

The wait is genuinely a readiness gate, not a bare port check: wait-for-mysql connects to the mysql-client Service, which is a plain ClusterIP (no headless, no publishNotReadyAddresses), so it only routes to Ready endpoints — and MySQL's readiness is a real mysqladmin ping. The TCP connect therefore succeeds only once MySQL actually responds, which is exactly the condition jobs-manager needs. Bounded at 300s, deadline checked after each attempt, exits non-zero with a clear diagnostic on expiry (loud failure, not a silent Init hang). Image is byte-identical to the api container, so no extra pull, and the no-runAsUser-anywhere posture keeps OpenShift arbitrary-UID SCC admission intact while plain Kubernetes gets UID 1001 from the image. Making initContainers: unconditional rather than gated on hostPath.enabled is the right depth for the fix — an edge without hostPath previously rendered no init containers at all. All four Bugbot findings were addressed in code.

One scope note, not a blocker: this guards cold start only — an init container doesn't re-run if MySQL restarts mid-run — but that matches the ticket, and runtime reconnection is the app's concern.

Not approving yet: the PR is CONFLICTING with develop. develop reworked DRIFT_GUARDS in the Makefile from a single line into a |\-continuation multiline block, and this branch appended its guard to the old single-line form; there's also a Chart.yaml version-line collision (develop is now 1.9.91). Please merge develop in and re-resolve — re-add jobs-manager-waits-for-mysql.sh in develop's new multiline DRIFT_GUARDS format, and re-bump the chart version above 1.9.91 — then it's good to land.

— drafted with Claude Code

[LukasWodka]

Merged develop (7861fb2). Three conflicts, and two would have silently reverted work — worth naming since either could have passed review:

• Makefile DRIFT_GUARDS. Develop moved to one-guard-per-line (chore(make): one drift guard per line, and refuse an empty entry (backend#2626) #933) while this branch added jobs-manager-waits-for-mysql.sh to the old single-line form. Union merged — develop's format plus our guard as its own line. Taking either side alone drops a required guard: ours would have lost hostpath-reads-guarded.sh, theirs would have lost ours. Verified both present: 40 green.
• jobs-manager-deployment.yaml. Ours is this PR (unconditional initContainers); theirs is the nil-guarded hostPath read from fix(chart): guard every hostPath subkey read so --reuse-values renders (backend#2910) #939/backend#2910. Adopted their guard inside our block — taking ours verbatim would have reverted a fix that keeps --reuse-values rendering when the hostPath key is absent entirely. Verified: 3 live reads, 0 unguarded, and the chart renders with hostPath omitted.
• Chart.yaml → 1.9.92, one patch above develop's 1.9.91.

helm unittest 644/644, drift 40/40.

One thing I worked around rather than fixed, and filed instead — backend#2946: hostpath-reads-guarded.sh reads raw template text, so my comment explaining what the old gate looked like failed the gate by quoting the bare read:

client/templates/jobs-manager-deployment.yaml:52:
# It used to open inside `if .Values.hostPath.enabled`, so an edge without

No live read on that line. I reworded the comment to get past it, which is the wrong direction — the guard makes the anti-pattern unmentionable, so the next person documenting this migration either rewords around it or deletes the explanation. The fix is to strip comments before matching, as pod-qos-class.bats and .github's lint-targets-run-in-ci.py already do. Left as its own change since it is develop's guard, not this PR's.

[aptracebloc]

Re-reviewed at head 7861fb2 after the develop merge. The fix survived intact and the two dangerous conflicts were resolved the right way.

Verified against git diff origin/develop...HEAD:

• wait-for-mysql init container — still present and now unconditional (moved out from under the hostPath.enabled gate), still connecting to the mysql-client ClusterIP Service (a real readiness gate, Ready-only endpoints), still bounded at 300s with the deadline checked after each attempt, still failing loudly with the connection-refused diagnosis. UID posture is right: it reuses the jobs-manager app image with no runAsUser, inheriting pod-level runAsNonRoot exactly like api and pods-monitor-container — so admission passes on both OpenShift (project range) and plain Kubernetes (USER 1001 from the image).
• Makefile DRIFT_GUARDS — union-merged correctly: develop's one-guard-per-line |\ format (#933) is preserved and jobs-manager-waits-for-mysql.sh is added as its own line, with hostpath-reads-guarded.sh kept. Neither side's guard was dropped. Drift gate green.
• jobs-manager-deployment.yaml — develop's hostPath nil-guard (#939/backend#2910) was adopted inside the unconditional initContainers block rather than reverted.
• Chart.yaml — 1.9.92, one patch above develop's 1.9.91.

No conflict markers, no dropped hunks. All four prior Bugbot threads resolved; bugbot run on this head reports no new issues; CI is fully green. Approving.

— drafted with Claude Code

[LukasWodka]

/fr-pass

Functional review on staging — passed, with direct evidence.

Journey (tier A), staging · amd64, run 33494337123 — green end to end, every leg:

install via the real installer → client components healthy → CLI installed from its signed release and signed in → dataset ingested for every task type → use case published → model trained and the leaderboard read.

The train leg, which is the one that matters:

experiment echi0zyk (pk 4041) started
experiment echi0zyk: COMPLETED (terminal, 29 poll(s))
inference submit: submitted for inference on cycle no 2
leaderboard: found after 6 poll(s), running_score=0.9, cycle=2
submissions: our run is on the board (accuracy=0.9 loss=0.3515 captured=True); 1 row(s) total

This repo's change is on the path that run exercised, so this is functional evidence rather than an inference from code review.

Two things stated rather than glossed:

• The run is against the deployed code. Backend f991c788 was helm upgraded into staging at 09:50:19Z and this journey started at 09:50:46Z. A separate journey (33493727497) sits red from 09:43:42 — seven minutes before that deploy — so it exercised the previous image and is not evidence about this promotion.
• e2e-test-agent did not ship this hop (blocked by a Bugbot High, e2e-test-agent#368). So this is yesterday's agent, and the run does not exercise e2e's own unshipped changes. It exercises the platform, which is what this card needs.