Skip to content

Configurator MVP: operational acceptance and feature-branch merge readiness #95

Open

Description

@lweberru

Latest operational checkpoint (2026-10-22)

The real Platform Apply and HTTP-to-S3 migration acceptance in #90 is now complemented by successful Corporate Application provisioning and same-order recovery from a reported partial Apply failure; see the current acceptance record in #93. The fresh Plan retained 12 existing resources and the user-executed Apply completed with 3 added, 0 changed and 0 destroyed. The user also confirmed Internet/DNS and external SSH after manual firewall NAT/port-forward, VM security-group and guest-user/key configuration. These are positive acceptance results for the tested Corporate instance, not automated firewall-policy or HA/isolation qualification (#101).

Current Dev: #136 at 3d2f0af26e5bac23f23ff7c462cff1cec0f165b7, release 39767149101, Runner pool size 1, dev-skip-assistant-qualification, build_only=false. Deployment/migrations and isolated cold/warm Runner probes passed; the Assistant remains enabled. Live Assistant qualification was intentionally skipped for this unrelated fix and is not claimed by this release.

Two accounts have been exercised in one organization; this does not satisfy the two-organization/revocation criterion below. Public Application cloud acceptance, hard Runner loss, backup/restore, capacity and rollback criteria remain open. #135 is still open; the current acceptance does not qualify its Automatic IPv4 and identity-completion changes. No production or agent-executed customer Plan/Apply was performed. Earlier queued-merge statements below are historical.

Historical source integration

PR #102 contains the reviewed MVP source integration. Final source: c0e95ac. All 13 final-revision native/Configurator CI checks passed. The configured real models passed a complete 114/114 repeated suite and 13/13 continuous journey; prior failed evidence remains recorded in the PR.

The PR is ready and protected auto-merge is enabled. Required human/code-owner review is still outstanding; repository protections were not weakened or bypassed. Source integration is not an application deployment, customer Apply or production acceptance.

Acceptance checklist

  • Maintain current capability/boundary documentation, linked implementation issues and a distinction between source/mock/native and real customer evidence.
  • Freeze and verify the final revision through build/unit/browser/PostgreSQL isolation, native OpenTofu contracts and catalogue checks.
  • Accept two users/two organizations, revocation, interrupted Runner and recovery paths; retain untested live paths explicitly.
  • Verify PostgreSQL/Secrets Manager/artifact backup/restore, credential rotation and safe operator recovery diagnostics against the acceptance environment.
  • Complete capacity, job cleanup, migration rollback/forward recovery and release rollback compatibility acceptance.
  • Review CODEOWNERS/main protection and CI/CD transition; Configurator provisioning and deployment are now exclusively manual.
  • Prepare a reviewable PR and secret scan with a clean worktree. The sole history scanner result was independently verified as a synthetic mocked-provider password, not a credential.
  • Obtain the user decision authorizing source integration; no customer Apply or production creation is implied.
  • Complete source integration of feat: integrate Landing Zone Configurator MVP #102: GitHub records it merged on 2026-10-16 as 0835c7a31618a5ffd007000bd3d9bcf4b3e693ee. This records merge completion, not independent verification of every historical review event.

Maintained references

The Configurator README in #102 links the user guide, component diagrams/architecture, MVP scope and development map. CONTRIBUTING describes the cross-component workflow; AGENTS.md references the preserved Terraform style guide. Dated workstation handoffs and historical readiness/runtime ledgers have been removed from the maintained tree and are not current acceptance evidence.

Native CI: https://github.com/stackitcloud/stackit-landing-zone/actions/runs/38898519075

Configurator CI: https://github.com/stackitcloud/stackit-landing-zone/actions/runs/38898519039

Remaining scope

This issue remains open for the operational acceptance above. No production environment was created; production identity, network, backup/recovery, capacity and operator qualification are a separate follow-up. Local/model/CI success does not certify every live customer topology.

Product acceptance continues to depend on #89, #90, #91, #92, #93 and #94. #94 is linked for source-merge closure; the other issues retain their unaccepted live criteria.

Activity

  1. added
    priority:p1High impact correctness, security or required product capability; schedule first
    effort:mSeveral components or contract changes; estimate excludes external waiting
    area:configuratorLanding Zone Configurator application, runtime and its infrastructure
    mvp:requiredRequired acceptance gate for the agreed Configurator MVP; not a merge approval
    on Oct 1, 2026
  2. lweberru commented on Oct 4, 2026

    @lweberru
    CollaboratorAuthor

    MVP audit 2026-10-04: Updated the description and acceptance checklist against the current implementation. Implemented criteria are checked; remaining requirements are retained. Local tests are not a release or live customer acceptance. This issue stays open until its remaining criteria are met. The first remaining safety fix (#89, backend-free initial plans on existing/partial/legacy state) is now implemented locally: 23/23 isolated PostgreSQL tests and the full check gate (310 unit tests) pass. No customer Apply, API restart, native artifact rebind, commit or merge was performed.

  3. lweberru commented on Oct 16, 2026

    @lweberru
    CollaboratorAuthor

    MVP source integration is prepared in #102: all changes are committed/pushed; automatic native/Configurator checks are separated from manual cloud operations; public user/architecture docs and contributor/operator references are separated; CONTRIBUTING and AGENTS.md are added/updated; obsolete workstation/evidence artifacts are removed from the maintained tree. Existing main rules and required reviewer approval are unchanged. The user has authorized integration, but that does not bypass repository review rules.

    The PR records exact CI/model evidence and retained failed attempts. The final complete Assistant run is 114/114 plus a 13/13 journey. The final workflow-hardening commit is being verified separately; no deployment, customer Apply or production setup has occurred.

    Keeping this operational acceptance issue open. Two-organization/revocation acceptance, real interrupted-runner/migration recovery, backup/restore/rotation and production capacity/rollback/operator qualification remain separate uncompleted requirements. Source integration and green test jobs are not a declaration that those live paths were accepted. Maintained references are the user guide, architecture, MVP scope and development map linked from the Configurator README in #102, rather than the removed dated readiness/runtime ledgers.

  4. lweberru commented on Oct 16, 2026

    @lweberru
    CollaboratorAuthor

    Final integration update: all 13 checks on c0e95ac passed. PR #102 is ready, and exact-head protected auto-merge is enabled with the required review still pending. The issue checklist now reflects the completed source/CI/documentation work without marking untested live acceptance complete. #59, #82, #84 and #94 are linked for closure when the PR merges; no deployment or production resource was created.

  5. lweberru commented on Oct 17, 2026

    @lweberru
    CollaboratorAuthor

    Development delivery follow-up: #109–#114 are merged. The normal manual release 38986171056 succeeded on exact source 3ab7bfaa92236fc4a4242028a27a901df0c1496d; only the existing protected Development environment was approved. API and Runner current staged droplets were independently verified at that exact source.

    Exact-run evidence: configured Qwen model 114/114 repeated cases and 13/13 continuous dialogue; 817 unit tests passed (66 environment-dependent skips), 60 isolated PostgreSQL cases, 308 browser cases and pinned native contracts. Final actual-host browser smoke passed 5/5, including direct Plan/Apply URLs and reload, both viewport approval/blocker flows, locale, protected boundaries and same-tab STACKIT login.

    The first otherwise successful release exposed real hosted phase-URL 404s hidden by the Vite preview; #114 fixed the explicit API SPA allowlist and added actual-static-serving regression coverage. Earlier model/release/browser failures remain preserved, not replaced by unchanged-source qualification retries.

    Apply UI cases use intercepted synthetic APIs, not customer state; they never click Apply and forbid execution writes. The particular customer denial has not been diagnosed, and Firewall/HA Plan-only restrictions remain intact. No customer Apply or production deployment occurred. Source integration/development delivery is now complete; this issue remains open for the separately agreed operational acceptance, recovery and production follow-up.

  6. lweberru commented on Oct 20, 2026

    @lweberru
    CollaboratorAuthor

    Dev release blocked by Model Serving transport failures

    PRs #119 and #120 are merged. The authorized manual Dev release with runner_pool_size=1, run 39406372348, used exact source fc8ec29141c209769fa905504f83c921e7c22efa and failed qualification before any build, publication, migration or deployment.

    • Real-model cases: 112/114 passed. dns-zone-role-advice, iteration 1: model-phase connection_failed, 5591 ms. dns-owner, iteration 2: model-phase connection_failed, 3192 ms.
    • The corrected platform-public-corporate-networks case passed both independent repetitions. Continuous dialogue: 13/13 passed.
    • Runner image: Ubuntu 24.04. These are pre-response transport errors, not the configured 60000 ms inference deadline. The retained error reason groups allowlisted DNS, refusal, reset and transport-timeout causes; it does not identify which underlying code occurred. Provider-side failure versus the runner network path is therefore not established.
    • Five subsequent read-only /v1/models checks from the workstation returned HTTP 200 and listed the configured model. These checks are not inference qualification, do not reproduce the GitHub network path and do not resolve or replace the two failed repetitions.

    Failed receipts remain attached to the original run. No failed repetition was retried or reclassified, and no oracle, timeout, model or release protection was weakened. Investigating the original transport cause requires runner-side bounded cause diagnostics or corresponding service-side evidence; a speculative client change is not justified by the existing receipts.

    The existing Dev public health check still succeeds. The new release has not been deployed; hosted warm handoff/replacement proof remains unverified. No customer Apply or production operation was performed.

  7. lweberru commented on Oct 20, 2026

    @lweberru
    CollaboratorAuthor

    Bounded production inference retry implemented in #121

    Following the user discussion, #121 adds one additional request for selected transient pre-response network errors, with 250-500 ms jitter inside the original deadline. It preserves cancellation, never replays HTTP/body/semantic failures, and never retries Plan/Apply or persistence. Safe per-attempt codes are retained in production logs and synthetic receipts; no CI-only recovery path or factual oracle change.

    Exact source: 8f65ea8928f6e23bf7c8d62a9a7b262de0d59667. Pinned canonical validation: 971 passed, 69 unchanged default-gated tests. Changed-source real-model DNS cases: 16/16, two independent repetitions; continuous dialogue 13/13, both exit 0. No actual transport failures occurred locally; controlled tests verify retry and exhaustion without claiming the original provider/network cause was reproduced. Knowledge SHA: 6179c3ad16fe011412c165ed1a0d2d71bff0678b9ba3dc2642464fbc3a41db69.

    Normal protected squash auto-merge is queued; CODEOWNER review remains required. Original failed runs and receipts remain failed. Dev is not updated yet: after approved integration, a new exact-source manual release with runner_pool_size=1 must still pass full qualification and hosted warm evidence.

  8. lweberru commented on Oct 20, 2026

    @lweberru
    CollaboratorAuthor

    Next exact-source release: factual setup-routing defect identified

    The user merged #121 as 0c5188bf60c332adcbecf5372e913021b298e126. Its tree matched the validated feature tree. The optional extra local full run was cancelled after four passing cases, with no failures, to avoid concurrent inference with the protected release; it is not qualification evidence.

    New manual Dev release 39412434669, runner_pool_size=1, attempt 1: 113/114 cases, 13/13 dialogue, no transport-error events. firewall-metro-setup-operations repetition 1 omitted the unanswered operations decision. Build, publication, migration and deployment were skipped. The original failed receipt remains unchanged.

    Draft #122 fixes a reproduced structural cause: general ha field hints could bypass the dedicated scope/evidence classifier on a broad setup continuation. Four red regressions prove that bypass; the fix preserves explicit narrow native updates and uses null-only evidence when no bounded quote fragments exist. No factual oracle or topic facts changed. Canonical validation is green: 977 passed, 69 unchanged default-gated tests. Real-model Firewall qualification and CI are in progress; review readiness waits for their results. Dev remains on the previous release, without hosted proof of the new warm pool.

  9. lweberru commented on Oct 20, 2026

    @lweberru
    CollaboratorAuthor

    Dev release: qualification blocked; Model Serving maintenance confirmed

    PR #122 is merged at d0fc3d3ef1cb4c405fdfdb95e2a45cfb707b33c8.
    The fresh manual release run 39417609958
    requested runner_pool_size=1 but did not build or deploy:

    • Model qualification: 113/114 passed.
    • Continuous journey: 13/13 passed.
    • Only failure: synthetic dns-placeholder, repetition 2, upstream_status_403, at 2026-10-20T12:04:25Z.
    • No transport retry events; HTTP 403 remains non-retryable.
    • Build, image publication, migrations and Dev deployment were skipped. Hosted warm-pool acceptance remains unverified.

    The official STACKIT status API
    reported AI Model Serving under_maintenance at 2026-10-20T12:16:36Z:
    GPU Maintenance Notice Part 2, in_progress,
    scheduled until 18:00 CEST / 16:00 UTC. The original notice text says
    17:00 CEST instead; actual service recovery matters, not the advertised clock time.
    Maintenance confirms possible service interruptions but does not establish
    the specific reason for the observed HTTP 403.

    Recovery boundary: wait for the component to become operational and relevant
    maintenance to complete, then run a fresh complete protected manual release
    against unchanged main, with runner_pool_size=1 and unchanged qualification
    criteria. Do not rerun individual failures, erase failed repetitions, retry 403,
    switch models or weaken the gate. The failed run remains part of the evidence.

    UI reproduction after recovery: Actions > Configurator Release > Run workflow,
    branch main, build_only=false, runner_pool_size=1.
    Normal protected Dev approval remains required after qualification/build.
    No customer infrastructure Apply or production deployment is included.

  10. lweberru commented on Oct 21, 2026

    @lweberru
    CollaboratorAuthor

    Live platform acceptance update (2026-10-21)

    The operator supplied a successful real saved-plan Apply log and confirmed the completed workflow. The retained partial-bootstrap deployment was resumed through checkpoint review, explicit same-state reconciliation, a new credential profile, a new configuration revision/preparation and a newly reviewed saved Plan. Existing firewall VMs were preserved.

    Observed results: Apply complete! Resources: 13 added, 0 changed, 1 destroyed. The replacement was only the tainted terraform_data HA bootstrap step, not a firewall VM. Both backup and primary HA configuration commands completed successfully. All three project-qualified management buckets, Object Storage credentials, the audit archive destination, the Observability Starter instance and associated protected secret entries were created. The subsequent tofu init -migrate-state reported successful migration from the HTTP bootstrap backend to S3.

    Delivered by #128, #129 and #130 through Dev release 39608332090, with runner pool size 1. This was the explicit Dev qualification-exception mode, with the Assistant enabled; it is not production acceptance or a new successful full model qualification. Evidence here is a sanitized summary of operator-provided output, not independent access to customer infrastructure. Customer identifiers, credentials, raw plans and State are deliberately omitted.

    A clean follow-up Plan has not been independently established by the supplied log. Actual HA failover, packet paths and audit-event delivery were deferred. Hard Runner loss, interrupted backend migration, backup/restore and multi-tenant acceptance are not proven by this successful run.

    The source-merge checklist has also been corrected against GitHub: #102 merged on 2026-10-16. Operational acceptance remains open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:configuratorLanding Zone Configurator application, runtime and its infrastructureeffort:mSeveral components or contract changes; estimate excludes external waitingmvp:requiredRequired acceptance gate for the agreed Configurator MVP; not a merge approvalpriority:p1High impact correctness, security or required product capability; schedule first

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions