Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
229 changes: 207 additions & 22 deletions .dev-loop/INGEST_REPORT.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ follow the cross-pointers in their index or take the next matching seeded domain
| Domain | Status | Route here when |
|--------|--------|-----------------|
| [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming external/LLM APIs and defaults that name repo-external resources) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
| [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing (writing automated test code → testing) |
| [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
| [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |
| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, background services/cron, toolchain version pinning |
| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning |
| [mobile](wiki/mobile/index.md) | **seeded** | App-side iOS/Android/cross-platform: process death/state survival, offline-first sync, mobile-network calls, store rollout/hotfix strategy, startup time |

All ten domains are seeded. New categories grow via `skills/wiki-ingest/SKILL.md`.
4 changes: 4 additions & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,7 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
## [2026-07-13] ingest | databases +1 (query-optimization): streaming-large-result-sets — memory-bounded export of a huge single-query result. Client-side cursor pulls the whole set to libpq on execute (fetchmany caps only the Python-list explosion); only a server-side/named cursor truly streams but needs a transaction, so it fails under autocommit or a proxy that blocks BEGIN → fall back to client-side fetchmany + disk spool + openpyxl write_only (measured 300k rows 838MB→38MB). Derived from RNR-3440 (potential-listing weekly extract memory peak); QueryPie BEGIN-block generalized to "read-only access proxy", field-tested. Sources: psycopg2 usage/cursor docs (named-cursor WITHOUT HOLD + autocommit exception), openpyxl optimized-modes (write-only near-constant memory, lxml=speed-not-memory).
## [2026-07-23] ingest | databases +2: schema-design/online-schema-changes (ACCESS EXCLUSIVE lock avoidance — non-volatile default fast path, ADD CONSTRAINT NOT VALID + VALIDATE at SHARE UPDATE EXCLUSIVE, CHECK-NOT-NULL trick, CREATE INDEX CONCURRENTLY, expand-and-contract to decouple DB migration from app deploy, lock_timeout for lock-queue pile-up) + operations/autovacuum-and-wraparound (NEW category operations: per-table scale_factor/cost_limit tuning for hot tables, age(datfrozenxid)/relfrozenxid + n_dead_tup monitoring, wraparound read-only cliff and superuser VACUUM recovery, VACUUM FULL vs pg_repack). Derived from the Hatchet "Postgres survival guide"; both cross-checked against PostgreSQL official docs (sql-altertable, routine-vacuuming).
## [2026-07-31] ingest | platforms +1 (processes): non-interactive-cli-invocation — a CLI's own `-p/--print/--yes` flag does not stop it reading fd 0, so a scripted/harness call blocks on a terminal stdin and no request ever reaches the remote service; detach stdin (`</dev/null`, GNU nohup's unreadable-fd redirect), close documented prompt channels (`ssh -n`, `GIT_TERMINAL_PROMPT=0`), bound with `timeout`, then split client-vs-server by whether the request appears in the server's access log. Derived from a pi 0.79.1 foreground hang (2×, 0 bytes) with zero gateway-log arrivals that resolved instantly under nohup. Sources: GNU coreutils nohup, OpenBSD ssh(1) -n, git GIT_TERMINAL_PROMPT.
## [2026-07-31] ingest | backend +2 (NEW category common/integrations): llm-response-completeness (HTTP 200 is transport-only; fail on `finish_reason == "length"` and blank `content`; blank content + populated `reasoning_content`/`reasoning` means a reasoning model spent the output budget — raise the cap or route to a non-reasoning alias, never publish the reasoning text; log the returned `model` beside the requested alias) + externally-owned-defaults (a default naming a resource outside the repo is re-queried against the owner's catalog at review/merge time with a dated note, and resolved at startup so a retired name fails the deploy instead of the user). Derived from a LiteLLM gateway measurement (finish=length, content 0 chars / reasoning_content 8,173 chars at 9,317 input tokens) and a PR whose default alias had disappeared from the catalog after its own end-to-end check. Sources: OpenAI reasoning guide + chat-object finish_reason + deprecations + models/list, vLLM reasoning outputs, LiteLLM reasoning_content + proxy model discovery.
## [2026-07-31] revise | infrastructure/config/environment-config +1 edge case: a required value naming a repo-external resource must be resolved against the owner's catalog at startup, not just presence-checked; cross-linked to backend-common-integrations-externally-owned-defaults. `last_verified` left at 2026-07-10 — the page's own 12factor sources were not re-verified for this one-row addition (adversarial cross-check caught the stale-clock reset).
## [2026-07-31] revise | Adversarial cross-check (independent headless claude, VERDICT FAIL → 8 defects applied) on the three pages above: `ssh -n` reclassified from prompt-suppression to stdin-detach with `BatchMode=yes` (+`StrictHostKeyChecking=accept-new`) added as the prompt-channel directive and ssh_config cited; GNU nohup's stdin redirect qualified as terminal-only so the explicit `</dev/null` carries the guarantee; "no request in the server log" widened from "never left the client" to include pre-log rejection (DNS/TLS/proxy) with a `curl -v` discriminator; deprecated `function_call` added to the blank-content carve-out and the five-value `finish_reason` enumeration corrected; the GIT_TERMINAL_PROMPT "default is to prompt" parenthetical dropped as unsourced inference; backend/index.md concern-first subtree row extended (an agent routing by that line, read before the category tables, would otherwise land on `reliability`); externally-owned-defaults' pure-pointer row and startup directive given app-side substance. Declined: splitting externally-owned-defaults per trigger, and relocating it to infrastructure/config — its primary trigger is reviewing application-code defaults and the corpus already mixes numbered lists with case tables.
72 changes: 72 additions & 0 deletions wiki/backend/common/integrations/externally-owned-defaults.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
id: backend-common-integrations-externally-owned-defaults
domain: backend
category: integrations
applies_to: [general]
confidence: verified
sources:
- https://developers.openai.com/api/docs/deprecations
- https://developers.openai.com/api/docs/api-reference/models/list
- https://docs.litellm.ai/docs/proxy/model_discovery
last_verified: 2026-07-31
related: [backend-common-integrations-llm-response-completeness, infrastructure-config-environment-config, qa-process-release-gates]
---

# Defaults That Name a Resource Owned Outside the Repository

## When this applies

A code or config default names something whose existence is controlled outside your
repository — a gateway model alias, a serving endpoint name, a bucket, a queue, a
search index. Also when reviewing or merging a PR that claims end-to-end
verification of such a default, and when a feature that "was tested" fails on its
default path in an environment nobody changed.

## Do this

1. **Re-query the owner's catalog at review/merge time** and confirm the default
still resolves: `GET /v1/models` for an OpenAI-compatible provider or gateway,
the equivalent list/describe call otherwise. A measurement inside the PR body
dates from when it was written; nothing in code review re-checks a name the repo
does not own.
2. **Paste the dated check into the PR** (endpoint queried, result, date), so the
next reviewer reads the age of the evidence instead of trusting its existence.
3. **Resolve the configured name at startup, in the client wrapper that owns the
call, and crash when it is absent** — the process that will use the name is the
one that must prove it exists, so a retired alias fails the deploy instead of the
user's request. Put the resolution in the same module that builds the request, so
a new call site cannot bypass it. Config-shape and startup-validation mechanics:
[infrastructure-config-environment-config].
4. **Make the runtime failure name the resource**: log the requested name and the
provider's error verbatim, so "400 from the gateway" reads as "alias X no longer
exists" without a debugging session.

| Case | Do |
|------|----|
| The named resource is required for the feature | Validate at startup, crash on failure, and gate the deploy on that check |
| Several names are tried in order (primary + fallbacks) | Resolve all of them at startup, log which ones exist, and alert when the primary is unresolvable while a fallback is serving |
| The provider has announced a shutdown date | Record the date next to the default (comment + tracking issue) — preview-tier names can retire on as little as two weeks' notice |
| The name exists in one environment's catalog but not another's | Make it a required per-environment value with no default, and run the resolution check in every environment at boot — one shared default silently points staging at a name only production owns ([infrastructure-config-environment-config] owns the config shape) |

## Edge cases

| Case | Then |
|------|------|
| CI has no credentials for the catalog | Run the check by hand at review and paste the output with its date; add the startup resolution check so runtime still catches drift |
| Catalog lists the name but calls still fail | The name exists and your route/key lacks access — repeat the check with the exact credentials the service uses |
| Gateway resolves wildcards or aliases to upstream models | Query the gateway's own list endpoint (LiteLLM `/v1/models`, `check_provider_endpoint` for wildcards), not the upstream provider's — the gateway's mapping is the one your code hits |
| Default is only reached on a rarely used path | The startup check still runs — that is the point; a path exercised once a month otherwise reveals the dead name to a user first |

## Instead of

| If you are about to | Do this instead | Why |
|---------------------|-----------------|-----|
| Trust a PR's "verified end-to-end" for an externally owned name | Re-query the catalog at merge time and record the dated result | The measurement was true when written; the catalog changes outside the repo while code, tests, and build all keep passing |
| Hardcode a convenient default so local dev "just works" | Require the value and validate it at startup ([infrastructure-config-environment-config]) | A default that suits the author's machine becomes the silent production value once the real name is retired |
| Let the first user request discover that the name is gone | Resolve the name at startup / in the health check | The same failure costs a failed deploy instead of a broken feature plus an incident |

## Sources

- https://developers.openai.com/api/docs/deprecations — deprecation announces a shutdown date, after which the model "will no longer be accessible"; notice periods 6 months (GA), 3 months (specialized), as little as 2 weeks (preview)
- https://developers.openai.com/api/docs/api-reference/models/list — `GET /v1/models` "Lists the currently available models, and provides basic information about each one such as the owner and availability"
- https://docs.litellm.ai/docs/proxy/model_discovery — the proxy's `/v1/models` returns the models actually available behind that gateway; `check_provider_endpoint: true` resolves wildcard entries
66 changes: 66 additions & 0 deletions wiki/backend/common/integrations/llm-response-completeness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
id: backend-common-integrations-llm-response-completeness
domain: backend
category: integrations
applies_to: [openai-compatible]
confidence: verified
sources:
- https://developers.openai.com/api/docs/guides/reasoning
- https://developers.openai.com/api/docs/api-reference/chat/object
- https://docs.vllm.ai/en/latest/features/reasoning_outputs.html
- https://docs.litellm.ai/docs/reasoning_content
last_verified: 2026-07-31
related: [backend-common-reliability-timeouts-and-retries, backend-common-integrations-externally-owned-defaults, backend-common-errors-exception-handling]
---

# Accepting an LLM Chat-Completion Response as a Finished Artifact

## When this applies

Server code posts a prompt to an OpenAI-compatible `/chat/completions` (a provider
directly, or a gateway such as LiteLLM / vLLM) and uses the returned text as an
artifact — persisted, posted to a channel, attached to a ticket, fed downstream.
Which model answers is configurable, so a reasoning model can be routed in without
a code change. Also when empty or mid-sentence output is reaching users while the
call logs as a success.

## Do this

Treat HTTP 200 as "transport succeeded" only, and gate the artifact on the body:

| Case | Do |
|------|----|
| `finish_reason == "stop"` and `content` non-blank | Accept the text |
| `finish_reason == "length"` | Fail the call — the text stopped at your token cap mid-generation. Retry with a higher cap or a smaller input, and keep the partial text out of the artifact |
| `content` blank (empty or whitespace) | Fail the call with a diagnostic carrying `finish_reason`, `usage` (prompt/completion/reasoning tokens), the requested alias, and the returned `model` |
| `content` blank AND a reasoning field non-blank | Report "the reasoning model spent the output budget": raise the token cap (OpenAI advises reserving ≥25,000 tokens for reasoning + output while calibrating) or route to a non-reasoning alias. Source the answer from `content` alone — the reasoning field is scratch work, not the deliverable |
| `finish_reason == "tool_calls"`, or the deprecated `function_call` | Blank `content` is correct here — consume the tool call and skip the blank-content row above |

Then log the `model` value the response carries next to the alias you requested, so
a gateway reroute (alias → different model) is visible in the record afterwards.

## Edge cases

| Case | Then |
|------|------|
| Which reasoning field to read | LiteLLM returns `reasoning_content`; vLLM returns `reasoning` (renamed from `reasoning_content`). Check both keys before concluding no reasoning was produced |
| Streaming responses | Accumulate, then apply the same gate using the final chunk's `finish_reason` — validate before the text leaves your process |
| Caller uses the Responses API, not chat completions | The equivalent signal is `status == "incomplete"` with reason `max_output_tokens`; treat it exactly as `finish_reason == "length"` |
| Retrying a `length` failure | This is a fresh request, not a retry of the same bytes — raise the cap or shrink the input first, otherwise it truncates identically ([backend-common-reliability-timeouts-and-retries] owns retry policy) |
| Reasoning tokens are billed but invisible | Cost is incurred even when `content` is blank — send the blank-content failure to an alert with the token counts attached, and cap automatic retries at one |

## Instead of

| If you are about to | Do this instead | Why |
|---------------------|-----------------|-----|
| Read `choices[0].message.content` and write it straight to the artifact | Gate on `finish_reason` and non-blank content first | A 200 with `finish_reason=length` is a truncated artifact wearing a success status; downstream systems cannot tell |
| Treat an empty string as "the model had nothing to add" | Fail with `finish_reason` + token usage in the message | Blank content is budget exhaustion or a reasoning-only response — both are actionable configuration facts, not model opinions |
| Raise the token cap whenever output is blank | Check the reasoning field first, then choose raise-cap vs non-reasoning route | If reasoning filled the budget, a bigger cap buys more scratch work; the alias choice is what actually decides |
| Fall back to the reasoning text when `content` is blank | Fail the call and report the reasoning-token count | Reasoning text is unformatted intermediate state — publishing it leaks scratch work as a deliverable |

## Sources

- https://developers.openai.com/api/docs/guides/reasoning — reasoning tokens occupy the output budget and are billed as output tokens; a response can be cut off "before any visible output tokens are produced"; detect via `status == incomplete` / `max_output_tokens`; reserve ≥25,000 tokens while calibrating
- https://developers.openai.com/api/docs/api-reference/chat/object — five documented `finish_reason` values: `stop`, `length` ("the maximum number of tokens specified in the request was reached"), `tool_calls`, `content_filter`, and the deprecated `function_call` — the two tool-calling values are why blank `content` is carved out
- https://docs.vllm.ai/en/latest/features/reasoning_outputs.html — OpenAI-compatible servers put reasoning in `reasoning` (formerly `reasoning_content`) and the final answer in `content`
- https://docs.litellm.ai/docs/reasoning_content — LiteLLM normalizes provider thinking into `message.reasoning_content` (plus `thinking_blocks` for Anthropic)
Loading
Loading