Skip to content

PROD-2552: add agility verify to compare a synced target against its source - #235

Merged
5PK merged 1 commit into
feature/PROD-2542-required-field-validationfrom
feature/PROD-2552-verify-command
Oct 14, 2026
Merged

5PK merged 1 commit into
feature/PROD-2542-required-field-validationfrom
feature/PROD-2552-verify-command

Conversation

@5PK

@5PK 5PK commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #234 (PROD-2542). Merge #234 first; this PR's diff is against that branch.

Problem

A customer has had migrations where the sync reported success but components on pages ended up in the wrong order, and they couldn't be sure the page and sitemap order was right. --preflight only says what a sync intends to do. Nothing checks what it actually did, so the only way to confirm a migration is a manual regression pass over every page. A blue/green cutover needs a step that proves the new instance matches before traffic switches.

Change

New command agility verify --sourceGuid X --targetGuid Y. It adds no new flags: it reuses --locales, --channel, --models, --pages, --containers and --jsonSummary. It never writes to either instance or to the mapping files.

Pull guard

  • Checks the Fetch API status for both instances, in both preview and fetch mode. If an update is in progress, it waits up to a fixed 10 minutes using its own bounded polling. The SDK's wait has no timeout.
  • Then it does a full pull of both instances and checks again. If an update ran during the pull, it pulls once more. If that happens again, it exits 2.
  • Exits 2 if a status check fails or times out. Standalone pull ignores a failed status check, but verify doesn't (see PROD-2754).

Comparison

  • It doesn't reuse the push transforms or change detection, which decides on version IDs. Both sides are converted into the same neutral form, with source IDs translated to target IDs through the mapping files, and compared there. Otherwise a bug in the transforms would produce the same wrong result on both sides and still pass.
  • Checks:
    • Mapping integrity: source items with no mapping, mappings that point at a missing target, and two sources mapped to one target.
    • Models, containers and templates.
    • Content: counts per model and per container, list order, and field values. Linked content, assets (including asset URLs in rich text) and companion fields are resolved through the mappings.
    • Components on pages: the order within each zone and which zone each component is in. Zones are matched by section ID, then by name, and never by position (the PROD-2350 bug class).
    • Sitemap: each page's parent and its order among its siblings, per locale and channel.
  • Severities:
    • Failures: mismatches, missing items, and order, zone or sitemap errors.
    • Warnings: target-only content and publish-state differences.
    • Source defects, listed but not failing the run: an item missing a required field (using validateContentItemAgainstModel) or an item in a deleted container.
  • Output: a console table per locale and the first 20 findings per category, plus a full JSON report. Exit codes: 0 match, 1 differences, 2 couldn't verify.
  • README: new "Verify" section.

Testing

  • npx jest (stacked on PROD-2542: validate content against the target model before sending it; harden batch result alignment #234): 128 suites / 2,479 tests pass. tsc --noEmit is clean.
  • Unit tests cover:
    • the mapping index
    • each normalization rule
    • page zones: swapped components, a component in the wrong zone, a renamed zone matched by section ID, zones defined in a different order on the two templates, and an unmapped zone, which never falls back to position
    • sitemap reordering, reparenting and multi-channel
    • count mismatches, list order, and an item unpublished on one side
    • source-defect classification
    • the status guard: waiting, timeout and an update during the pull
    • exit codes and JSON output
  • End-to-end hermetic test: sync, then verify (exit 0). Reverse one zone in the target, and verify gives exit 1 with only a component-order failure. The mapping files are byte-identical afterwards.
  • Live, read-only, against a two-locale customer clone pair. The full pull took 54s the first time, mostly downloading assets, and about 4s on the repeat run; the comparison took 0.2s. Exit 1: 15 failures, 15 warnings, 67 source defects. The last sync for this pair reported 77 ok / 0 failed, but the failures are real target defects:
    • One page has empty zones on the target in one locale, missing 5 components. Three pages in the other locale are missing 4 components between them.
    • One page has a rich-text component twice, in both locales: the source page's 11 components plus a trailing duplicate. I checked this by hand.
    • Three banner items lost an image field, and one item links a list that only exists on the target.
  • Fully matching: 260 models, 2,170 containers, 6 templates, 1,306 assets, 7 galleries, 121 pages and 117 sitemap entries.
  • False positives found on the live data and fixed: Agility system containers, items in deleted containers, a duplicate component also being reported as a wrong order, empty or dangling list links, and missing page publish-state warnings.

Known limits / follow-ups

  • Status checks and pulls use the Management API region chosen at login, the same limitation sync has. Cross-region pairs aren't tested.
  • With --models, --pages or --containers, the checks for extra target items and per-container counts are skipped. The scoped modes haven't been run against live data.
  • Comparing green against blue, and verifying a reverse-sync, are deferred to v2.
  • validate-required-fields.ts comes from PROD-2542: validate content against the target model before sending it; harden batch result alignment #234, which this PR is stacked on.
  • The live findings mean the sync can leave component gaps and duplicates while reporting success. These need their own ticket.

Closes PROD-2552. Covers PROD-2549.

🤖 Generated with Claude Code

@5PK
5PK requested a review from DerekAgility October 13, 2026 15:00
@5PK
5PK marked this pull request as ready for review October 13, 2026 15:00
…s source

PROD-2552: verify comparison library (mapping index, neutral form, checks, report)

- mapping-index: Map-backed bidirectional indices over every mapping type, read-only
- canonicalize: neutral comparison form (target IDs, linked refs, asset URL
  translation and host aliasing, null/empty/CRLF/case/numeric normalization);
  shares no code with the push transforms
- checks: mapping integrity, models/containers/templates, content (fields,
  counts, list order), page fields and zones (section-ID or exact-name pairing,
  never positional), sitemap (parent, sibling order, missing)
- verify-report: findings, tallies, exit codes, console table, JSON report
- writeJsonSummary accepts any report object

PROD-2552: add `agility verify` command

Status guard (bounded 10-minute wait in preview and fetch mode, re-pull once
if an update lands during the pull), full pull of both instances, per-locale
checks, console + JSON report. Exit 0 pass, 1 differences, 2 could not verify.
Accepts only existing flags via a dedicated verifyArgs set; never writes to an
instance or to the mapping files. Also satisfies PROD-2549.

PROD-2552: document the verify command in the README

The required-field check (validate-required-fields.ts) is PROD-2542's (#234); this branch builds on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK force-pushed the feature/PROD-2552-verify-command branch from 0f3317d to a1e41c8 Compare October 14, 2026 18:15
@5PK
5PK changed the base branch from main to feature/PROD-2542-required-field-validation October 14, 2026 18:16
@5PK
5PK merged commit 9359987 into feature/PROD-2542-required-field-validation Oct 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants