Skip to content

config validate: check that a prompt_file exists - #307

Open
polyglotAI-bot wants to merge 1 commit into
ClickHouse:mainfrom
polyglotAI-bot:cron/validate-prompt-file-exists
Open

config validate: check that a prompt_file exists#307
polyglotAI-bot wants to merge 1 commit into
ClickHouse:mainfrom
polyglotAI-bot:cron/validate-prompt-file-exists

Conversation

@polyglotAI-bot

Copy link
Copy Markdown
Contributor

Summary

prompt_file is the one cron setting nothing verified. resolve_prompt re-reads
the file on every run — that is what makes a prompt editable without a
restart — so a path that never resolves is not caught at load. The job loads,
validates, schedules, and then fails only when it fires, which for a nightly job
is a day later. A validator that checks the schedule and every gate field but
not whether the prompt is there is checking the easy half.

[ERR] cron jobs job 'integrations-pr-monitor': prompt_file
      .../prompts/integrations-pr-monitr.md does not exist and the job has no
      inline prompt to fall back to, so every run of it fails

Error vs warning

A missing file is an error only when there is no inline prompt to fall back
to. With one, resolve_prompt degrades to it quietly — survivable, but rarely
what was meant: a fallback is typically a short summary of a long prompt, so the
job silently runs a lesser version of itself. That is a warning.

What stays unflagged

Each of these follows a rule the file already applies elsewhere:

  • a workflow job never calls resolve_prompt, so its prompt settings are inert;
  • a path still holding a literal ${VAR} is unset-by-definition here, and gets
    the same leniency as every other unresolved reference in the bundle;
  • under portable_only, a path outside the tracked config root. A shared
    bundle cannot be expected to carry a file its author deliberately kept on the
    machine, and failing a config repo over one is the same substitution
    _tracked_cron_only refuses one directory over: judging a config repo by what
    the reviewing filesystem happens to have. Without portable_only the
    filesystem being validated is the one that will run the jobs, so every path
    is fair game, absolute included.

The in-repo prompts/<job-id>.md convention — the case the check exists for — is
inside that root, so CI still verifies it.

How this came up

A config repo of mine just moved all 9 cron prompts out of jobs.yaml into
prompts/<job-id>.md and dropped the inline fallbacks, which makes an unreadable
prompt_file fatal by design (loud failure over silently running an 8-line stub
of a 37 KB prompt). Validation passing a typo'd path is a much sharper edge in
that shape, and the CI for that repo is exactly nerve config validate --workspace . --portable-only --strict-keys.

Test plan

  • 13 new tests in TestPromptFileChecking
  • The 5 that assert detection fail without the change (verified by
    stashing the config_validate.py diff and re-running); the other 8 are
    false-positive guards that must pass either way
  • Full suite green: 3203 passed (3190 before)
  • Dogfooded on a real config repo: with the change, a 9-job bundle whose
    prompts all resolve reports no new error; injecting a one-character typo
    into one prompt_file produces the error above

`prompt_file` is the one cron setting nothing verified. resolve_prompt
re-reads the file on every run - that is what makes a prompt editable
without a restart - so a path that never resolves is not caught at load.
The job loads, validates, schedules, and then fails only when it fires,
which for a nightly job is a day later. A validator that checks the
schedule and every gate field but not whether the prompt is there is
checking the easy half.

A missing file is an error only when there is no inline `prompt` to fall
back to. With one, resolve_prompt degrades to it quietly, which is
survivable but rarely meant: a fallback is typically a short summary of a
long prompt, so the job runs a lesser version of itself. That is a
warning.

Three cases stay unflagged, each for a reason the file already respects
elsewhere: a `workflow` job never calls resolve_prompt; a path still
holding a literal ${VAR} is unset by definition here; and under
portable_only, a path outside the tracked config root cannot be judged
from the bundle - failing a config repo because the reviewing filesystem
lacks a deliberately machine-local prompt is the same substitution
_tracked_cron_only refuses one directory over. Without portable_only the
filesystem being validated is the one that will run the jobs, so every
path is fair game, absolute included.

13 tests, and the 5 that assert detection fail without the change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant