Skip to content

Reject unknown integration configuration IDs and fields - #1455

Merged
prk-Jr merged 11 commits into
mainfrom
fix/unknown-integration-config
Nov 12, 2026
Merged

prk-Jr merged 11 commits into
mainfrom
fix/unknown-integration-config

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Reject unknown integration IDs during config validate/push and name unknown fields, preventing typos from silently changing integration behavior.
  • Apply strict known-field schemas to enabled and disabled integration blocks, tester-cookie settings and Tinybird, while preserving minimal disabled blocks, open payload maps and explicit compatibility fields.
  • Warn and continue at runtime for unknown integration IDs; document the cleanup-before-deploy migration for unknown fields.

Known tables containing unknown fields will fail startup with the new binary. Run the new CLI's ts config validate, correct the reported settings, push the cleaned config while the old binary runs. On Fastly, wait for the config-store propagation window to elapse before activating the new binary; a successful push does not mean every point of presence (POP) has the cleaned config.

Changes

File Change
crates/trusted-server-core/src/config.rs Reject unknown IDs at deploy validation, warn at runtime, cover every known integration schema, and route the gpt_diagnostics/js_asset_proxy pre-pass parsers through the shared redacted parse error.
crates/trusted-server-core/src/settings.rs Include unknown/missing-field details in outer parse-error contexts, retain other serde details in the report cause chain, share that helper with the pre-pass parsers, and make tester-cookie and Tinybird schemas strict.
crates/trusted-server-core/src/integrations/adserver_mock.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/didomi.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/google_tag_manager.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/gpt.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/js_asset_proxy.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/lockr.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/nextjs/mod.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/permutive.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/sourcepoint.rs Reject unknown configuration fields.
crates/trusted-server-core/src/integrations/testlight.rs Reject unknown configuration fields.
crates/trusted-server-core/src/test_support/async_html_finalization.rs Keep the optional fixture schema consistent with strict integration validation.
crates/trusted-server-integration-tests/tests/html_eof_runtime.rs Build the harness CLI with the fixture feature so strict validation accepts the test integration used by the runtime.
crates/trusted-server-cli/tests/config_env_overlay.rs Exercise actual validate/push refusal and verify no invalid local blob is written.
docs/guide/configuration.md Explain field validation, runtime ID warnings, safe upgrade order and the nested missing-field limit for disabled blocks.
docs/guide/integrations/lockr.md Note under the field table that unknown fields, including the formerly documented organization_id and project_id, are rejected; set app_id.
docs/guide/error-reference.md Use an existing integration and supported route in the troubleshooting example.
CHANGELOG.md Document the compatibility break and config-first cleanup order.
docs/superpowers/specs/2026-10-27-strict-integration-validation-design.md Record behavior, alternatives, preservation requirements and the nested missing-field limit for disabled blocks.
docs/superpowers/plans/2026-10-27-strict-integration-validation.md Record implementation and verification steps.

Closes

Closes #1201

Spec

Not a feature (bug fix). The technical design is in docs/superpowers/specs/2026-10-27-strict-integration-validation-design.md.

Test plan

Run with Rust 1.98.1, Node 24.20.0 and Viceroy 0.21.0. Initial sandbox listener/keychain-access failures were resolved by full reruns with the required local access.

  • cargo test-fastly && cargo test-axum
  • cargo clippy-fastly && cargo clippy-axum
  • cargo fmt --all -- --check
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run — 1,232 tests passed with one worker, no type errors
  • JS format: cd crates/trusted-server-js/lib && npm run format
  • Docs format: cd docs && npm run format
  • Remaining adapter/host Clippy gates, including Cloudflare WASM, Spin native/WASM, CLI and codegen
  • cargo test-fastly-reuse, cargo test-cloudflare, cargo test-spin
  • Full host-target CLI suite, native core suite and feature-on core configuration tests
  • Integration parity and native template-build-digest tests/lint
  • JS build, Markdown formatting, WASM clock lint and documentation-proposal script checks
  • EOF harness native unit tests and Clippy
  • ./scripts/async-html-finalization-local-test.sh axum --auction
  • ./scripts/async-html-finalization-local-test.sh fastly --release --auction — includes the test-utils release WASM build
  • Manual fastly compute serve — not performed

Regressions distinguish the original acceptance/diagnostic failures and cover all 15 production integration IDs, enabled/disabled fields, runtime warnings, open payload maps and deprecated compatibility fields.

The EOF harness builds its CLI with trusted-server-core/test-utils so validation recognizes the fixture integration selected by its runtimes. Ordinary production CLI builds retain the production ID list.

Checklist

  • Not a feature; feature-specific spec-review requirement is inapplicable
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code
  • Uses log macros (not println!)
  • New code has tests
  • No secrets or credentials committed

@prk-Jr prk-Jr self-assigned this Oct 27, 2026
Build the harness CLI with the fixture feature used by its runtimes, and correct the Lockr troubleshooting field.
@aram356 aram356 added this to the 202611 milestone Oct 27, 2026
@prk-Jr
prk-Jr marked this pull request as ready for review October 29, 2026 12:11

@dhruv8sh dhruv8sh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Strict field schemas for every known integration (enabled and disabled), the rejection of unknown IDs when validating or pushing, and the warn-and-continue behaviour at runtime are the right split. Configs already stored keep loading, and new typos are caught before deploy.

I checked:

  • deny_unknown_fields is not combined with #[serde(flatten)] on any config struct. The flattened Maps in testlight.rs are on request and response bodies, not TestlightConfig, and the datadome flatten case was already handled.
  • Nested config structs are already strict (JsAssetProxyAsset, and the datadome, prebid and aps sub-structs).
  • validate_disabled_schema still skips only missing field errors. Serde reports unknown fields while visiting the map, before it checks for missing ones, so minimal disabled blocks with a typo are rejected. deploy_validation_rejects_unknown_fields_in_minimal_disabled_integrations covers this.
  • The test-utils fixture ID is gated, so production CLI builds keep the production ID list, and the EOF harness builds with the feature on.
  • The Lockr docs table matches LockrConfig: app_id required with length(min = 1), enabled defaulting to true, the TTL range of 60–86400, and rewrite_sdk_host as a compatibility field.

The CHANGELOG and configuration guide spell out the push-config-first, deploy-binary-second migration clearly.

Two minor notes inline. Neither blocks merging.

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The strict known-field schemas and deploy/runtime integration-ID handling are consistent with the intended contract. The rollout instructions need a Fastly propagation step to avoid activating strict validation against an older config blob.

The inline finding is prose-only because its documentation correction spans multiple files; no one-click suggestions are included.

Blocking

🔧 wrench

  • Wait for Fastly config propagation before activating the strict binary — see inline at docs/guide/configuration.md:192–196

Verification

All 21 changed files were read completely. git diff --check passed. Passing remote CI was reused rather than rerun locally. The propagation risk was source-traced through the pinned EdgeZero dependency; live Fastly propagation was not reproduced.

CI Status

Comment thread docs/guide/configuration.md Resolved
@prk-Jr
prk-Jr marked this pull request as draft October 29, 2026 16:46
Require Fastly config propagation before strict binary activation.
Keep only unknown/missing-field details in outer integration parse
messages, retaining other serde details in the report cause chain.
Document per-request unknown-ID warnings on ordinary Fastly sandboxes.
@prk-Jr
prk-Jr marked this pull request as ready for review October 29, 2026 17:10

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Rejects unknown integration IDs during deploy validation and unknown fields in known configuration schemas, while retaining runtime warnings for unknown IDs and minimal disabled blocks. Reviewed all 21 changed files at 9690c166cc1053f4bca976a06ed7fe7aeeb9fa59; no actionable findings.

No inline comments or suggestions. Existing CI results were reused; no local test suites were rerun. The isolated review worktree is clean.

CI Status

@jwrosewell

Copy link
Copy Markdown
Collaborator

@prk-Jr @aram356, directionally this change is not compatible with V2 (the 55 pull requests tracked in #777, merged in order).

It makes the V1 settings format stricter, with known-field schemas for each [integrations.<id>] block. V2 replaces that format. A module is selected by its type and named by its folder path, and the V2 design checks every settings section against the module that consumes it, in one check that reports every gap at startup and at push.

The V2 spec is settings-and-consumers in #1342, tracked as #1403. A V1 settings document is rewritten into the V2 shape by ts config migrate, designed in config-migrate and raised as #1409.

Could you evaluate the V2 spec and #1409 before progressing this pull request?

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This adds strict known-field schemas to every integration table, plus [tester_cookie] and [tinybird], rejects unknown integration IDs at validate and push time, and warns about them at runtime, matching the spec's acceptance table. The PR now conflicts with main, which blocks merging. Two pre-pass parsers also bypass the new error redaction.

1 of the inline comments below carries a one-click GitHub suggestion. Use Commit suggestion to apply it. The other comments describe the fix in prose because it touches multiple files or lines outside the diff.

Blocking

🔧 wrench

  • Resolve the merge conflicts with main: see Cross-cutting below

Non-blocking

♻️ refactor

  • The two pre-pass parsers still echo invalid values: see inline at crates/trusted-server-core/src/config.rs:318

🤔 thinking

  • A nested missing field hides unknown keys in a disabled block: see inline at crates/trusted-server-core/src/settings.rs:386

⛏ nitpick

  • The enable row also matches the expected-fields list: see inline at crates/trusted-server-cli/tests/config_env_overlay.rs:477

Cross-cutting / body-level findings

  • 🔧 Resolve the merge conflicts with main: CHANGELOG.md and docs/guide/integrations/lockr.md conflict with #1405 and #1406, so GitHub reports the PR as not mergeable. #1406 rewrote lockr.md, and its field table already matches LockrConfig (app_id required, the documented defaults, rewrite_sdk_host deprecated and ignored). Suggested resolution:

    • CHANGELOG.md: keep both bullets.

    • lockr.md: take main's version, then add one note under its field table. The guide at the merge base told operators to set organization_id and project_id, and configs that still carry them now fail validation and startup:

      Unknown fields are rejected, including the `organization_id` and `project_id`
      fields that earlier versions of this guide listed. Remove them and set `app_id`.

    A trial merge with that resolution passes the native core suite (3,349 tests) and config_env_overlay (11 tests). The lockr.md row in the PR description will need a refresh afterwards.

  • 📌 Typo'd env-overlay variables are still ignored silently: TRUSTED_SERVER__INTEGRATIONS__GPT__ENABLE=false and TRUSTED_SERVER__INTEGRATIONS__PREBDI__ENABLED=true both leave ts config validate passing, because the EdgeZero overlay only applies to leaves that already exist in the TOML. It's the same class of silent typo this PR closes for TOML, one layer up, and belongs in EdgeZero as a follow-up rather than in this PR.

  • 📌 Prebid docs still show keys that strict validation rejects: this predates the PR (PrebidBundleBuildConfig was already strict on main), but it's the same copy-the-docs-then-fail path. docs/guide/integrations/prebid.md:784-786 shows [integrations.prebid.bundle] with adapters and user_id_modules; the schema is [integrations.prebid.bundle.modules] with bidder and user_id, as prebid.md:259-270 itself describes. prebid.md:765 and docs/guide/configuration.md:2026 name bundle.user_id_modules in prose. .env.example:46-57 lists commented Prebid variables (SERVER_URL, BIDDERS, BID_PARAM_*, SUPPRESS_NURL_BIDDERS, AUTO_CONFIGURE, TEST_MODE) for keys PrebidIntegrationConfig doesn't have.

  • 📝 Release-candidate configs carry keys main doesn't have: tag v0.1.1-rc.202610.1 isn't on main and accepts [tinybird] auction_enabled and Lockr capture_identity. Configs from that RC will fail on a main-based binary carrying this PR. The documented step to run ts config validate with the new CLI before upgrading catches them, so no change is needed here; it's worth keeping in mind when RC operators move to a main-based release.

Verification

  • All 21 changed files read in full at 9690c166cc1053f4bca976a06ed7fe7aeeb9fa59.
  • deny_unknown_fields coverage: every struct reachable from Settings and every nested integration config struct is strict, and none uses flatten. Every integration ID the runtime reads is in DEPLOY_VALIDATED_INTEGRATION_IDS.
  • Mutation checks: removing Sourcepoint's deny_unknown_fields, the parse_error detail, the runtime warning, Tinybird's deny_unknown_fields, or the deploy ID check each fails at least one of the new tests.
  • Real-binary probes with ts config validate against unknown-ID, unknown-field, and invalid-value configs, plus env-overlay typos.
  • A docs and committed-config scan (162 files) found no key or ID that the newly strict schemas reject, apart from the pre-existing Prebid drift above.

CI Status

…ions

Route the gpt_diagnostics and js_asset_proxy pre-pass parsers through the
shared IntegrationSettings::parse_error helper so invalid values stay out of
the outer message, matching the other integrations. Pin the CLI unknown-ID and
unknown-field rows to full phrases, and document that a nested missing field
in a disabled block hides unknown keys visited after it.
@prk-Jr

prk-Jr commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

@aram356 Body-level items from your review:

  • Merge conflicts: resolved in b9cd00a4b, following your suggested resolution. CHANGELOG.md keeps both sets of bullets. lockr.md takes main's version plus the note under its field table that organization_id and project_id are now rejected. The PR is mergeable again. The native core, CLI, all adapter suites, parity and every clippy gate pass locally on f98238b1b.
  • Env-overlay typos: agreed that this belongs in EdgeZero, so it isn't in this PR.
  • Prebid docs drift: it predates this PR and isn't changed here.
  • RC-only keys: noted. No change, per your note.

@prk-Jr
prk-Jr requested a review from aram356 November 11, 2026 14:00
Comment thread docs/guide/configuration.md Resolved
@prk-Jr

prk-Jr commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

@jwrosewell I evaluated the settings-and-consumers spec and #1409 at 177adad47. The consumer spec explicitly lists #1201 outside its scope. Migration reads the V1 input as raw toml_edit::DocumentMut, rewrites it, then validates the V2 output; it does not deserialize the input through these stricter V1 schemas. Its 15 integration IDs match this PR. I found no demonstrated migration-input incompatibility from this fix; this was source inspection, not a V2 migration runtime test.

This remains a bounded fix for the current main configuration format. When the V2 stack lands, its new selection/validation model must replace or adapt this V1 ID inventory. Can you confirm whether landing this V1 fix separately is acceptable? I have not changed or rebased any V2 PR.

@prk-Jr
prk-Jr merged commit 06e51be into main Nov 12, 2026
30 checks passed
@prk-Jr
prk-Jr deleted the fix/unknown-integration-config branch November 12, 2026 09:04
prk-Jr added a commit that referenced this pull request Nov 12, 2026
Keep both Unreleased breaking-change entries in CHANGELOG.md: main's
integration configuration validation (#1455) and Phase 2's tsjs build
requirements.
prk-Jr added a commit that referenced this pull request Nov 12, 2026
Bring in main's integration configuration validation (#1455) and mediator
fallback (#1493) through Phase 2.
aram356 added a commit that referenced this pull request Nov 12, 2026
…igration

Conflicts keep both sides; #1493's mediator HTTP-error fallback moves onto run_mediator and its tests onto the scripted client, and #1455's checks sit beside the cut's config rejections.
@aram356 aram356 mentioned this pull request Nov 12, 2026
11 of 12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unknown integration IDs and fields in [integrations.*] pass validation and are ignored

5 participants