Release 1.19.0 → PROD (Option B content migration + bundled features) - #2038
Merged
Conversation
The freshness scan judged code blocks in isolation and applied generic best-practice dogma, producing false positives (intentional errors, demo base64 credentials, dev-container setup, npm version-pinning advice that contradicts CAP guidance). - extractCodeBlocks now captures contextBefore/contextAfter (the prose paragraph adjacent to each fence). - extractTutorialContext pulls frontmatter + Prerequisites, fed once at the top of the prompt so the model knows the reader's environment. - SYSTEM_PROMPT now instructs the model to judge blocks in context, respect intentional teaching artifacts (deliberate errors, illustrative demo credentials), and follow SAP/CAP conventions (never suggest pinning @sap/* package versions). - Export SYSTEM_PROMPT/buildUserMessage; add extractor + prompt tests.
- PRECISION: prefer no finding over a speculative one; report each issue once. - OUTPUT vs CODE: never flag staleness/secrets inside illustrative output blocks. - SEVERITY defined by reader impact (fails today / deprecated path / cosmetic). - SCOPE: skip prose, screenshots, product-name currency, deliberate simplifications. - GROUNDING: quote the offending token; flag training-data inferences as Low. Reframes the reviewer as helping the tutorial AUTHOR. Adds prompt-guard tests.
…t-fix freshness: feed tutorial context + SAP-aware rules to the scan
…ckfiles setup-node's 'cache: npm' already warms ~/.npm, but 'npm ci' still spends ~48s extracting/linking 1680 packages + native builds into node_modules on every rebuild run (verified in run 32783495862: cache restored, yet 'added 1680 packages in 48s'). Cache the materialized node_modules (root + hugo-apps) on an exact lockfile key and skip the install on hit. Exact-key only (no restore-keys): skipping 'npm ci' means a mismatched tree would go unreconciled, so any lockfile change must miss and reinstall clean. Safe because neither package.json has install/prepare/postinstall lifecycle scripts -- nothing outside node_modules is produced at install time.
…o-end) Design + specs + tasks for making a single-tutorial rebuild scale with changed slugs instead of the full corpus. Four independently-shippable workstreams, grounded in code research (file:line): - content-delta-publish: mutable ContentCurrent + append-only ContentHistory replaces full-snapshot-per-version + carryForwardUnchanged (~95s server cost) - generated-content-cache: cache hugo/content/tutorials (~25MB) with a correctness-first key (hashes /build feeds + parser source + per-slug source) - single-slug-render: scoped Hugo render (R2 verified coherent) - tutorial-discovery: unmask GraphQL errors + fix App-token org-node auth + loud REST fallback Companion CI win (node_modules cache) shipped separately in #2016.
perf(rebuild): cache node_modules to skip ~48s npm ci on unchanged lockfiles
…a-rebuild Design: slug-targeted delta rebuild (O(changed) end-to-end)
graphqlRequest warn-logged a GraphQL errors/null-data response and returned the null data, so callers dereferenced data.organization/data.repository and threw an opaque TypeError — which the outer catch mistook for an outage and SILENTLY degraded discovery to the slow per-slug REST path (~1400 extra calls, secondary-rate-limit risk) every rebuild. - graphqlRequest now throws a typed GraphqlError carrying the GraphQL error type+message; throws on errors[] or null data. - GraphqlError.isAuthError classifies FORBIDDEN/INSUFFICIENT_SCOPES/permission failures (the App-installation-token org-node gap). - discovery + metadata-batch fallbacks log at ERROR level (once) when the cause is an auth/permission error, so a token-scope regression is visible instead of hidden as latency. Workstream A of the slug-targeted-delta-rebuild design (#2017). The auth fix itself (repo-oriented discovery vs PAT vs App Org:Read) is Workstream B and gated on the real error this now surfaces on DEV. Test: test/unit/github-graphql-error-unmask.test.ts (6 cases).
fix(fetch): unmask GraphQL discovery errors + loud REST fallback
Pure, tested helpers for the slug-targeted fast path: feed fingerprint (catalog/co-completions/tag-labels), sidecar read/write, fast-path eligibility decision, and navEntries-by-slug reconstruction. Fail-open throughout. Nothing calls these yet — the fetch-tutorials.ts reuse wiring (flag-gated, requires a byte-identical DEV diff-verify) + the actions/cache workflow step land next. Test: test/unit/content-cache.test.ts (15 cases).
feat(rebuild): generated-content cache — Workstream C (foundation, draft)
…ated off)
On a slug-targeted run with CONTENT_CACHE_FAST_PATH=true, fetch-tutorials
reuses the previously-generated .md + sidecar nav/author rows for non-target
slugs, skipping composeTutorial/fetchRulesVr/AI-quiz/writeHugoPage — the bulk
of Phase 3's ~56s. Two gates, both required, else full regen:
- actions/cache KEY = parser-source hash (a parser change → cache miss → the
per-slug existsSync guard falls through to recompose)
- runtime feed fingerprint over CAP catalog + tag-labels (nav/mission/tags);
co-completions excluded (empty on warm-cache runs, recommendations are
client-hydrated)
Sidecar (navEntries + authorRows + fingerprint) written to .content-cache/,
cached by the same parser-hashed key as hugo/content/tutorials +
image_dimensions.json. Fail-open throughout; flag defaults OFF so default
behavior is byte-identical to today.
Enabling on DEV (input content-cache=true) + a byte-identical diff-verify
(task 3.6) is the follow-up before flipping the default.
Helper unchanged behavior; content-cache tests green (15).
…n B schema, Workstream D) Mutable current-content table (one row per slug, no version column) + append-only per-(version,slug) history carrying the BLOB for self-contained rollback replay (design.md D1/D2). Mirrored into the QA namespace. Verified: full model compiles + deploys to sqlite::memory:. NEXT (gated, not in this commit): .hdbmigrationtable staging for the two new HANA tables (db/src/ + db/last-dev/ regen via the project's cds-build-staging procedure) + hybrid deploy verify — the version-counter-hazard step. Then write path (dual-write, flag), readers, rollback.
… (Workstream D, flag-gated) commitSession now mirrors freshly-published slugs into the mutable ContentCurrent (UPSERT, one row per slug) + append-only ContentHistory (WRITTEN rows) when CONTENT_DELTA_WRITE_ENABLED=true, alongside the legacy ContentFiles write. Chunked BLOB read mirrors carryForwardUnchanged (raw db.run on HANA for the LOB-locator path, CQL on SQLite). Fail-SAFE: never throws into the commit tx — legacy ContentFiles stays the source of truth until the reader cutover. Tests (SQLite): test/unit/content-delta-dualwrite.test.js (4) — one-row-per-slug, UPSERT on republish, history append per version, flag-off writes neither. Existing commit-path tests still green (carryforward-memory + guard, 12). HANA LOB path is covered by the hybrid publish→rollback test (task 7.4).
feat(rebuild): generated-content fast path — Workstream C wiring (flag-gated, draft)
…h-schema feat(db): Option B schema — ContentCurrent + ContentHistory (Workstream D, draft)
rules.vr answer options like [ ] no / [x] yes rendered as false/true (and a yes/no correctAnswer flipped to a boolean) because the frontmatter serializer emits bare yes/no/on/off tokens under YAML 1.2, which Hugo's YAML 1.1 parser then reads as booleans. jsonify ships those booleans into <script id="tutorial-data"> and the quiz island renders "false"/"true". Add scripts/lib/hugo-yaml.ts (hugoFrontmatterStringify, version: '1.1') so the writer quotes every token a 1.1 reader would misparse, and route all Hugo-frontmatter serializers through it (render-frontmatter, fetch-tutorials, fetch-advocates, author-pages-writer). Regression test round-trips through a YAML 1.1 read.
fix(#2023): stop Hugo coercing yes/no quiz options to true/false
The canonical notify-workflow template (docs/authors/tutorial-repo-dispatch.yml) triggered only on `branches: [main]`. Source repos are a MIXTURE of default branches — most default to `main`, but some (e.g. sap-tutorials/Tutorials) default to `master`. On a master-default repo a main-only trigger never fires, so edits never dispatch `tutorial-updated` and PROD silently goes stale (root cause of the cp-* content-drift incidents; Tutorials was hand-patched by sap-tutorials/Tutorials#24184 but the template itself was never updated — so a future `install-notify-workflows` run would REVERT that fix). - Template now triggers on `[master, main]` (GitHub ignores a listed branch that doesn't exist, so it's safe on every repo regardless of default). - Add a regression guard in the installer unit test asserting the template fires on both branches and never ships a main-only trigger. Propagate to source repos with `npm run install-notify-workflows -- --execute`.
Pre-existing DEV breakage inherited by this PR's CI merge ref: the feature-flag registry drift guard (test/unit/feature-flags-registry.test.js) fails because srv/lib/content-publish-session.js reads process.env.CONTENT_DELTA_WRITE_ENABLED (Workstream D Option-B dual-write) without a matching FEATURE_FLAGS entry. Register it (env, true-enables, default OFF, dev-only) so the guard passes.
fix(notify): dispatch source-repo rebuilds on both master and main
…lag-gated, draft) (#2028) * feat(serve): Option B read cutover for serveStoredSlug (Workstream D, flag-gated) serveStoredSlug serves from the mutable ContentCurrent (WHERE slug=?, no version join) when CONTENT_DELTA_READ_ENABLED=true AND the slug is present, else falls back to the legacy version-pinned ContentFiles snapshot. Per-slug fallback keeps a partially-populated ContentCurrent (pre-full-seed) from 404-ing slugs still in ContentFiles. LOB BLOB read stays on raw db.run for HANA (new hanaCurrentTableName helper). X-Content-Source: db-current | db. Covers the hot path + pages/author/advocate (all route through serveStoredSlug). Special-slug/catalog/embedding readers still use ContentFiles (follow-up 6.2-6.5) — consistent because ContentFiles stays fully populated + authoritative. Tests: test/unit/content-delta-read.test.js (3); content-store legacy suite green (48). * feat(rollback): Option B ContentCurrent clear on rollback (Workstream D, flag-gated) During the dual-write migration window, rollbackHandler now clears ContentCurrent after the manifest flip (when CONTENT_DELTA_WRITE_ENABLED=true) so every read falls back to the just-restored authoritative ContentFiles(target.version) — correct-by-fallback, no per-slug BLOB replay needed. ContentHistory is retained (append-only) for the post-ContentFiles-retirement full history-replay (task 8.4). Fail-safe: never breaks the legacy rollback. Test: content-delta-read.test.js rollback case (publish v1+v2 → rollback v1 → ContentCurrent cleared → serve falls back to ContentFiles D1). 4 tests total green.
…2030) (#2031) Add query-string state sync to the Devtoberfest sessions grid so search, week/track filters, edition, and an open session panel are all shareable and restorable from the URL: /devtoberfest/sessions/?q=<text>&week=<w>&track=<name>&edition=<id>&session=<id> - New pure url-state.ts (parse/serialize + validation), mirroring the calendar's #2006 url-state.ts and the concepts-filter precedent; fail-open (empty/unknown params -> null, never throws). Unit-tested (round-trip, empty/whitespace rejection, unknown-param drop). - App.vue: parse on mount (q/week/track applied synchronously; session applied once the feed loads), history.replaceState on state change, popstate re-sync. App-level integration tests cover restore-from-URL, session-panel open, write-back, and unknown-session fail-open. Closes #2030
… (Workstream D 6.2-6.5, 8.4) (#2032) * feat(serve): migrate __404__ + __nav__ single-slug readers to ContentCurrent (Workstream D 6.2) Adds shared resolveContentBlob helper (ContentCurrent-first when read flag on + present, else legacy ContentFiles active snapshot; raw db.run LOB on HANA). serveNotFound (__404__) + navHandler (__nav__) now route through it — same per-slug fallback as serveStoredSlug, safe on a partially-seeded ContentCurrent. Enumeration readers (nav-fallback/hashes/source-hashes/embeddings) deliberately NOT migrated yet — they enumerate ALL slugs and must wait for the full ContentCurrent seed (task 4.3), else they'd return only the dual-written subset. Tests: content-store (48) + content-delta-read (4) green. * feat(serve): migrate hashesHandler + navHandlerFallback to ContentCurrent (Workstream D 6.3) Both enumerate the active slug set (metadata-only, no BLOB) — switch to ContentCurrent (no version) when CONTENT_DELTA_READ_ENABLED, else legacy ContentFiles active snapshot. Correct once ContentCurrent is fully seeded (task 4.3, running now via a full force rebuild). Tests: 52 green. * feat: complete Option B reader sweep — all readers off ContentFiles-version (Workstream D 6.2-6.5) Migrates the remaining readers to ContentCurrent (read-flag gated, per-slug or enumeration fallback to legacy ContentFiles): - chrome-shell __shell__ (ContentCurrent-first + version fallback; version cache key kept — self-invalidates as manifest version still bumps per publish) - sourceHashesHandler, getTutorialSource (source-column reads) - embedding readContentBuffer + active-slug-set reads (embedding-stats, embedding-reconciliation, cleanup.pruneOrphanEmbeddings, admin seedEmbeddings) No cache re-key needed (6.5): version-keyed caches invalidate on each publish's manifest-version bump regardless of read source. With ContentCurrent seeded (2362 slugs), every reader now serves from it under the read flag. This is the prerequisite for removing carryForwardUnchanged (8.4). Tests: content-store + delta-read + dualwrite + carryforward (58) green. * feat(publish): Option B carry-forward removal + rollback replay (Workstream D 8.4 — the payoff) CONTENT_DELTA_SKIP_CARRYFORWARD (default off) makes publish O(changed slugs): commitSession skips carryForwardUnchanged, so ContentFiles(newVersion) holds only the freshly-published slugs. ContentCurrent (mutable, dual-written) stays complete and serves everything; the ~95s PROD carry-forward disappears. Coupled rollback rework: with carry-forward off, ContentFiles(V) is no longer a complete snapshot, so rollbackHandler now REPLAYS ContentHistory into ContentCurrent (per-slug latest version <= V, chunked LOB read) instead of clear+fallback. detectReverts/history untouched. Adds invalidateContentCache() export (prod busts via cache-generation token on publish; ops/tests that drive publish directly use this). Tests: content-delta-carryforward-skip.test.js (2) — O(changed) publish (ContentFiles holds only changed slugs) + byte-correct multi-version rollback replay. Full content suite 70 green. Enable order (per env): read cutover deployed + ContentCurrent seeded, THEN flip CONTENT_DELTA_SKIP_CARRYFORWARD. * chore(feature-flags): register CONTENT_DELTA_READ_ENABLED + SKIP_CARRYFORWARD env flags Adds the two remaining Option B env flags to the feature-flag registry (CONTENT_DELTA_WRITE_ENABLED was already registered). Satisfies the feature-flags-registry drift guard. dev-only status, default OFF.
The canonical .deploy/mta.yaml approuter builder copied admin-ui, analytics-ui, explore-ui, scanner-ui and data-inspector-ui but never built or copied app/display-app, so static/display-app/ never shipped and the XSUAA-protected /display-app/ route 404'd in every env. Root mta.yaml had the step but that file is not what mbt build uses. Add npm install + vite build + copy of dist/ into static/display-app/, mirroring the analytics-ui pattern (app has base '/display-app/'). Note: the per-app deploy guards (Step 3.5 admin bundle, 3.6 QA navigator, explore-ui check) are hand-curated allowlists and none covers display-app/scanner-ui — an omitted route slips through.
…lags + rollout order) (#2033)
…icalized (#2036) The Option B reader sweep (#2032) added new ContentCurrent read paths in resolveContentBlob/serveStoredSlug (content-store.js) and readContentBuffer (embedding-pipeline.js) mirroring existing ContentFiles reads, but only the pre-existing lines carried the // slug-canonical: marker. The 8 new sibling lookups use the identical caller-canonicalized slug (documented contracts), so mark them consistently rather than adding redundant .toLowerCase(). Fixes check-slug-lookups build guard failure in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DEV→main release for the 1.19.0 PROD deploy. 28 commits ahead of main.
Headline: Option B content migration (#2017 / Workstream D)
Mutable
ContentCurrent+ append-onlyContentHistoryreplaces full-snapshot-per-versionContentFiles. Makes single-slug publish O(changed slugs) instead of O(corpus) (PROD carry-forward was ~95s @ 11k files → measured ~973ms on DEV).Safety: all three env flags default OFF — this deploy changes no PROD behavior until flags are flipped hands-on post-deploy.
PROD rollout (staged, hands-on, after this merges)
mbt build+ blue-greencf deployfrom fresh origin/main (flags off).CONTENT_DELTA_WRITE_ENABLED→ restart.CONTENT_DELTA_READ_ENABLED→ restart → verifyX-Content-Source: db-current.CONTENT_DELTA_SKIP_CARRYFORWARD→ restart → measure.cf restart.Already proven: DEV (all 3 flags, soaking since 2026-08-25) + QA (2218 slugs seeded, all 3 flags on).
Also bundled
#2030/#2031 devtoberfest deep-links · #2035 display-app into approuter static · #2036 Option B slug-lookup guard · plus other DEV commits since 1.18.3.