Repository navigation
PROD-2542: validate content against the target model before sending it; harden batch result alignment - #234
Merged
Conversation
DerekAgility
approved these changes
Oct 13, 2026
…t; harden batch result alignment
PROD-2542: locale-scoped failed-content registry and run-scoped effective target models
- failedContentRegistry is keyed by locale + contentID, so an es-us failure no
longer labels the en-us module that references the same source contentID.
getFailedContent now takes the locale; process-page passes it.
- New effectiveTargetModels registry in state (cleared per push run and by
resetState). model-pusher registers the saved model after a successful
create/update (the payload when the response has no fields), and the source
model for planned creates/updates under --preflight. The target
models/<id>.json cache isn't rewritten after saveModel, so it is stale for
the rest of the run.
- Preflight page entries note "N component(s) would be dropped" when modules
point at content registered as failed in that locale.
PROD-2542: validate content against the target model before sending it
New pure validator (util/validate-required-fields.ts): a Required field whose
key is absent (matched case-insensitively) or null fails the item for scalar
types (Text, LongText, HTML, Integer, Decimal, Date, DropdownList, Boolean) -
the DB rejects these ("Column 'promoTitle' does not allow nulls."). Attachments,
galleries, linked content, Link and custom fields only warn. Empty values,
Text over its Length and DropdownList values outside Choices ("Label|value"
lines) only warn.
filterContentItemsForProcessing validates every item it would create/update
against the effective target model (run registry -> target cache -> source
model -> none) and routes failures to a new itemsToFail list, so they never
reach a batch. Warnings are logged and the item proceeds.
- Real runs: content-pusher folds itemsToFail into totalFailed and the failure
details (with model + missing fields), so they reach the console/log, the
ERROR SUMMARY and --jsonSummary, for both the linked and normal passes; each
is registered as failed content immediately.
- Preflight: recorded under a new "invalid" action (counted separately, shown
with its reason, never a create/update, no exit-code change); JSON summary
preflight totals/phases gain an additive "invalid" count.
PROD-2542: guarantee batch result alignment; name the skipped child in cascades
- prepareContentPayloads returns one paired {source, payload} list; the
payloads sent and the sources results are matched against both derive from
it. An empty prepared list skips the API call. If the save/poll throws, only
the prepared items are failed (prep failures were already counted, and
results already recorded are not re-failed).
- extractBatchResults: batch.items must have one entry per item sent, or every
item fails with "unverifiable batch result alignment" instead of guessing.
For updates of a target item known to exist, a returned itemID that differs
from the contentID sent fails that item.
- logBatchErrors resolves a failed item via batchItemId -> position in
batch.items -> payload (the old fallback used the index within the failed
list and could name the wrong item); unresolvable -> "unknown".
- "Unresolved content reference(s)" now appends "(child '<ref>' skipped:
<reason>)" when the referenced child failed earlier in the same locale.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5PK
force-pushed
the
feature/PROD-2542-required-field-validation
branch
from
October 14, 2026 18:15
6e66c28 to
0e771f0
Compare
5PK
added a commit
that referenced
this pull request
Oct 14, 2026
…s source PROD-2552: verify comparison library (mapping index, neutral form, checks, report) - mapping-index: Map-backed bidirectional indices over every mapping type, read-only - canonicalize: neutral comparison form (target IDs, linked refs, asset URL translation and host aliasing, null/empty/CRLF/case/numeric normalization); shares no code with the push transforms - checks: mapping integrity, models/containers/templates, content (fields, counts, list order), page fields and zones (section-ID or exact-name pairing, never positional), sitemap (parent, sibling order, missing) - verify-report: findings, tallies, exit codes, console table, JSON report - writeJsonSummary accepts any report object PROD-2552: add `agility verify` command Status guard (bounded 10-minute wait in preview and fetch mode, re-pull once if an update lands during the pull), full pull of both instances, per-locale checks, console + JSON report. Exit 0 pass, 1 differences, 2 could not verify. Accepts only existing flags via a dedicated verifyArgs set; never writes to an instance or to the mapping files. Also satisfies PROD-2549. PROD-2552: document the verify command in the README The required-field check (validate-required-fields.ts) is PROD-2542's (#234); this branch builds on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5PK
added a commit
that referenced
this pull request
Oct 14, 2026
…s source (#235) PROD-2552: verify comparison library (mapping index, neutral form, checks, report) - mapping-index: Map-backed bidirectional indices over every mapping type, read-only - canonicalize: neutral comparison form (target IDs, linked refs, asset URL translation and host aliasing, null/empty/CRLF/case/numeric normalization); shares no code with the push transforms - checks: mapping integrity, models/containers/templates, content (fields, counts, list order), page fields and zones (section-ID or exact-name pairing, never positional), sitemap (parent, sibling order, missing) - verify-report: findings, tallies, exit codes, console table, JSON report - writeJsonSummary accepts any report object PROD-2552: add `agility verify` command Status guard (bounded 10-minute wait in preview and fetch mode, re-pull once if an update lands during the pull), full pull of both instances, per-locale checks, console + JSON report. Exit 0 pass, 1 differences, 2 could not verify. Accepts only existing flags via a dedicated verifyArgs set; never writes to an instance or to the mapping files. Also satisfies PROD-2549. PROD-2552: document the verify command in the README The required-field check (validate-required-fields.ts) is PROD-2542's (#234); this branch builds on it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 14, 2026
Merged
5PK
added a commit
that referenced
this pull request
Oct 14, 2026
… their base branches) (#238) * PROD-2552: add `agility verify` to compare a synced target against its source PROD-2552: verify comparison library (mapping index, neutral form, checks, report) - mapping-index: Map-backed bidirectional indices over every mapping type, read-only - canonicalize: neutral comparison form (target IDs, linked refs, asset URL translation and host aliasing, null/empty/CRLF/case/numeric normalization); shares no code with the push transforms - checks: mapping integrity, models/containers/templates, content (fields, counts, list order), page fields and zones (section-ID or exact-name pairing, never positional), sitemap (parent, sibling order, missing) - verify-report: findings, tallies, exit codes, console table, JSON report - writeJsonSummary accepts any report object PROD-2552: add `agility verify` command Status guard (bounded 10-minute wait in preview and fetch mode, re-pull once if an update lands during the pull), full pull of both instances, per-locale checks, console + JSON report. Exit 0 pass, 1 differences, 2 could not verify. Accepts only existing flags via a dedicated verifyArgs set; never writes to an instance or to the mapping files. Also satisfies PROD-2549. PROD-2552: document the verify command in the README The required-field check (validate-required-fields.ts) is PROD-2542's (#234); this branch builds on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * PROD-2773: warn when content references a CDN container owned by neither the source nor the target PROD-2773: warn about asset URLs into a CDN container owned by neither instance Content synced before the PROD-915 fix kept the upstream instance's CDN URLs, so a source can reference a third instance's container (e.g. cdn-usa2.aglty.io/ upstream-qa/...). The asset mapping cannot map those, so they passed through silently: image/file fields arrived empty on the target and rich-text links kept pointing at the other instance. The content pusher now scans each locale's syncable items (also in --preflight and for unchanged items) and reports every foreign CDN URL once per run, with the items/fields/locales using it. Reporting only: payloads, counts, success and exit codes are unchanged. - foreign-cdn-detector: pure URL extraction (aglty.io, azurewebsites origin, cdn*.agilitycms.com, learned custom edge hosts; JSON-escaped, &, query and percent-encoded variants) and container-ownership check. Source/target containers come from asset mappings and pulled assets; nothing is reported while either side's container is unknown. - foreign-cdn-report: per-run dedupe and rendering; attachment-field URLs are called out separately from text occurrences. - Surfaces: logger warning on first sighting, end-of-run "Foreign CDN references: N URLs in M items" block, additive --jsonSummary warnings (code "foreignCdnReference", url, container, references), and a preflight warnings section that does not affect totals or the exit code. Preflight totals now include PROD-2542's "invalid" count; the push test expects it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * PROD-2782: republish a page's other locale when a save in one locale puts it in staging A page's zones are shared by all of its locales. When a save in one locale adds, removes or reorders components, the server cuts a new page version and gives every other language a staging row on it (ClearPageContent / ReorderPageContent), so the page drops to Staging in its other locales. --autoPublish only republished pages the run saved in that locale, so the other locale stayed in Staging, and the PROD-2628 refresh then recorded its new version as in sync, so later runs skipped it as up to date. refreshOtherLocaleVersions already re-reads the page in each other locale of the run after a save. It now records the page for republishing in that locale when it was Published on the target when pulled, is Staging now, and is Published on the source (a page left in staging on the source stays in staging). executeAutoPublish adds those pages to that locale's publish list (pages/both), de-duplicated, and updates their mappings as usual. A page whose own save in that locale fails is never republished, whichever order the saves ran in. Without auto-publish of pages, the run lists them as non-blocking "staging" warnings in the console and --jsonSummary. Locales not in the run are not checked: there is no pulled state for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Content items missing a value for a Required field are sent to the Management API anyway, and the database rejects them (
Column '<field>' does not allow nulls.on two recent customer runs). Neither--preflightnor the run's own reporting flags these items ahead of time. Batch results are also matched to their source items purely by array position, with no length or ID check.logBatchErrorscould already name the wrong item, because it took a position from the failed list instead of from the batch.Change
Validation before the API call (always on, no new flags)
validate-required-fields.tschecks each item against the target model's field definitions."", text longer than the field's Length, dropdown values not in Choices, and on Link, Image, File, Gallery, CustomField and linked-content fields. We haven't confirmed the API rejects nulls for those.Requiredsetting are matched case-insensitively.effectiveTargetModelsregistry, which the model pusher fills after a save. The target cache isn't rewritten after a save in the same run, so the registry is checked first, then the target cache, then the source model.filterContentItemsForProcessing, before any batch is built.Reporting
--jsonSummaryfailures, with the model, fields, contentID and locale. This covers both the linked-content pass and the normal pass. The item is registered as failed straight away.--preflight: a newinvalidaction, shown as a warning. It isn't counted as a create or update, and the exit code doesn't change.Batch alignment hardening
prepareContentPayloadsreturns a single list of{source, payload}pairs, so payloads and source items can't drift apart.extractBatchResultsImpl: if the number of results differs from the number sent, every item fails as "unverifiable batch result alignment". For updates, a returneditemIDthat differs from thecontentIDsent fails that item.logBatchErrorsfinds the payload throughbatchItemId. If it can't, it prints "unknown" instead of guessing.Testing
npx jest: 119 suites / 2,350 tests pass.tsc --noEmitis clean.17-required-field-missing-in-batch: item 2 of 4 is invalid, and the other three map to the correct target IDs.logBatchErrorsnaming the right itemKnown limits
--overwriterecreate of a deleted target, where the server can assign a new ID.agility verifyto compare a synced target against its source #235 (PROD-2552, the verify command) is stacked on this PR and uses this branch'svalidate-required-fields.ts.Closes PROD-2542.
🤖 Generated with Claude Code