Docs and operating-model requests 504: content hydration crawls 2,179 GitHub blobs serially inside a 30s Lambda

[alexeygrigorev]

Symptom

The portal Today page shows "Process documents are unavailable — HTTP 504", an empty action queue, and My Plan / operating-model data missing. Work API calls were also failing (fixed separately, see below).

Diagnosis (verified against production 2026-09-26)

• GET /docs/process-quality invoked directly on the backend Lambda returns Sandbox.Timedout after 30.00s; /health on the same container returns 200 in ~2s. CloudFront surfaces the timeout as 504.
• Every docs/operating-model request calls ContentsApiGithubStore.sync() (backend/src/docs/githubStore.ts), which on a cold container hydrates the whole content/ tree one GitHub API call per blob, serially, with no fetch timeout.
• DataTalksClub/dataops-knowledge content/ currently holds 2,179 blobs (443 md, 1,516 jpg, 220 png). At ~0.3–0.5s per blobs call that is ~10–15 minutes of work inside a 30s Lambda, so hydration never completes in one invocation. Each new container restarts the crawl (progress persists only in per-container /tmp), so no container ever becomes warm.
• Blast radius: /docs/*, /search, /api/operating-model, /api/my-plan, and the Today page's process-quality call (operatingModel/loader.ts calls the same store.sync()).
• Side damage: each doomed crawl burns up to ~2,200 GitHub requests; the knowledge token's rate limit already showed 1,301/5,000 used within one hour.

Work API 403s (fixed in data on 2026-09-26, needs a repo-level guard)

• The Show signed-in identity and authorized teammates’ Task/Card work #164 role gate denies users without a supported role. The three live user items predate the role attribute (created 2026-06-28, no role), and seedRuntimeUsers skips existing users as "unchanged", so the role was never backfilled. Every team read/work mutation returned 403 ("Team access requires an active admin or operator role").
• Applied the declared seed state (role: 'admin' for grace/valeriia/alexey) directly to dataops-v1-users via UpdateItem; /api/cards and /api/tasks verified 200 after.
• Follow-up needed: make seedRuntimeUsers update missing attributes on existing users (or migrate in place), so this cannot recur. Data migration was done by hand this time only because the gate shipped without it.

Fix directions (for grooming)

1. Stop eager per-blob hydration. Candidates: hydrate only .md eagerly and fetch images lazily via ensureFile; or download a single tarball and extract content/; or sync the knowledge repo to S3 from CI and hydrate from S3 in one/few calls.
2. Add an explicit timeout (AbortSignal.timeout) to request()/fetchImpl in githubStore.ts so a GitHub hang fails fast with 502 instead of eating the whole Lambda budget.
3. Consider a startup/deploy-time hydration step (offline snapshot in the artifact) instead of on-demand crawling.

Related observability gap

Backend Lambda log streams stop on 2026-08-11 even though the function is actively invoked (CloudWatch metrics show hundreds of invocations/day; direct invokes today produced no streams). The execution role still has AWSLambdaBasicExecutionRole attached. Root cause unknown — needs its own investigation; until fixed, production debugging is blind (this diagnosis had to be reproduced by direct invokes).