feat(#2104): migrate engagement_problem_completion_summary to the dimensional layer

[quazi-h]

What are the relevant tickets?

Closes #2104 (epic #2072)

Description (What does it do?)

engagement_problem_completion_summary built its course-title lookup directly off stg__mitxonline__app__postgres__courses_courserun and ..._courses_course, bypassing both the intermediate and dimensional layers. It now reads dim_course_run joined to dim_course on course_fk = course_pk, so the model references only dim_*/afact_* models. Layering baseline shrinks 198 → 196.

Two things in the diff worth knowing. The lookup keeps a platform = 'mitxonline' filter, because the staging pair read the MITx Online course tables only — course_title has always been null for every other platform's course run here, and without the filter 1,865 more course runs (1,321 mitxpro, 492 edxorg, 52 bootcamps) would suddenly get titles. Both dimension CTEs also carry the SCD2 fan-out guard from dim_product.sql; the staging version leaned on course_id being a primary key to get exactly one title per run, and a bare group by doesn't reproduce that bound.

How can this be tested?

Run from the repo root on this PR's branch (feat/2104-problem-completion-summary-dimensional). Needs AWS credentials.

Everything below uses a throwaway DuckDB under /tmp. ~/.ol-dbt/local.duckdb is shared by every worktree on the machine, and a locally-built dim_course_run/dim_course left there by another session shadows the Glue view and silently skews the diff — that happened while validating this PR and produced a false MISMATCH.

1. Register sources and build both sides.

export ISO=/tmp/iso-2686 REAL_HOME="$HOME"
mkdir -p $ISO/.ol-dbt
export HOME=$ISO AWS_CONFIG_FILE=$REAL_HOME/.aws/config \
       AWS_SHARED_CREDENTIALS_FILE=$REAL_HOME/.aws/credentials \
       AWS_REGION=us-east-1 AWS_DEFAULT_REGION=us-east-1

# --duckdb-path is required; HOME alone does not redirect it
for db in staging intermediate dimensional; do
  uv run --frozen ol-dbt local register --database "ol_warehouse_production_$db" \
    --duckdb-path $ISO/.ol-dbt/local.duckdb --quiet
done

M="$(git rev-parse --show-toplevel)/src/ol_dbt/models/reporting/engagement_problem_completion_summary"
git fetch origin main -q && git show origin/main:src/ol_dbt/models/reporting/engagement_problem_completion_summary.sql > "${M}_pre.sql"

DBT_PROFILES_DIR=src/ol_dbt uv run --frozen dbt run --project-dir src/ol_dbt \
  --select "engagement_problem_completion_summary engagement_problem_completion_summary_pre" \
  -t dev_local --full-refresh

Expect: 276, 159, 58 tables registered, Errors: 0 each. Then PASS=3 ERROR=0 SKIP=0, with every ref() logging Falling back to Glue view.

2. Test the changed model.

DBT_PROFILES_DIR=src/ol_dbt uv run --frozen dbt test --project-dir src/ol_dbt \
  --select "engagement_problem_completion_summary engagement_problem_completion_summary_pre" \
  -t dev_local --indirect-selection=cautious

Expect: PASS=2 WARN=0 ERROR=0 — the compound-uniqueness test.

3. Diff old against new.

uv run --frozen ol-dbt diff \
  --old engagement_problem_completion_summary_pre \
  --new engagement_problem_completion_summary \
  -k user_email,course_title,courserun_readable_id,subsection_title,subsection_block_index,chapter_block_fk

Expect: MATCH, exit 0.

Row counts: ..._pre=3377077  ...=3377077  Δ=0
Summary: match — row Δ 0, 0 unmatched row-side(s), 0 column value mismatch(es).

Negative control — copy the built model, mutate one course run's course_title, and diff that with --old-raw. Expect MISMATCH, exit 1 (I get 28,420 unmatched sides from 14,210 altered rows).

4. Clean up.

rm -f "${M}_pre.sql" && rm -rf $ISO && export HOME="$REAL_HOME"

Expect: git status clean.

Note for other #2072 migrations: dim_course/dim_course_run cannot be built on dev_local at all — dim_course hits Binder Error: Cardinality can only operate on MAPs on Trino-only cardinality(), and three ancestors use Trino-only JSON SQL. Both sides above read them as Glue views, which is why the registration has to be frozen for the comparison to mean anything.

🤖 Generated with Claude Code

[github-actions]

🔎 ol-dbt impact — column-level blast radius

✅ No column-level downstream impact detected for the changed models.

_{Posted by ol-dbt impact (annotate-only — does not block merge).}

[Copilot]

🟢 Approval recommended

The migration satisfies the layering requirements and preserves the lookup’s cardinality and platform scope.

Pull request overview

Migrates the problem-completion summary to the dimensional layer while preserving MITx Online title behavior.

Changes:

• Replaces staging course lookups with guarded dim_course_run and dim_course lookups.
• Removes the resolved layering violations from the baseline.

File summaries

File	Description
engagement_problem_completion_summary.sql	Uses dimensional course models with SCD2 deduplication.
dimensional_layering_baseline.txt	Removes two obsolete staging dependencies.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The migration satisfies the layering requirements and preserves the existing lookup scope and cardinality.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced