Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
43 changes: 43 additions & 0 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1430,6 +1430,49 @@ jobs:
run: npm test
- name: Build the static bundle sqb server embeds
run: npm run build
- name: Install the layout test's browser
run: npx playwright install --with-deps chromium
- name: Layout test (every page with awkward mock data at 1440 and 1920 px)
run: npx playwright test --project=layout

ui_timings:
# The timing guards in ui/e2e/perf.spec.ts, with the CPU slowed 6x. They go red over budget and
# report their numbers, but stay out of Verify's needs: a slow hosted runner must not block a merge.
name: UI work timings (not required)
needs: changes
if: needs.changes.outputs.ui == 'true'
runs-on: ubuntu-24.04
defaults:
run:
working-directory: ui
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: ui/package-lock.json
- name: Install dependencies
run: npm ci
- name: Build the static bundle sqb server embeds
run: npm run build
- name: Install the timing test's browser
run: npx playwright install --with-deps chromium
- name: Time the slow paths (graph gestures, Checks reopen, plan sections)
shell: bash
run: |
set -o pipefail
npx playwright test --project=perf --no-deps --reporter=list 2>&1 | tee perf.log
- name: Report the timings
if: always()
shell: bash
run: |
{
echo "### UI work timings"
echo '```'
grep -E 'CPU slowed|second visit' perf.log || echo "No timings were recorded."
echo '```'
} >> "$GITHUB_STEP_SUMMARY"

verify:
# The required Verify of a Release Please pull request is the commit status that the trusted
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 8 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_execution_limits.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_explicit_references.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_expression_sources.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_ignored_audit_failures.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_lifecycle_commands.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_manifest_artifact_gating.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_materialization_type_change.py \
Expand Down Expand Up @@ -340,7 +341,7 @@ test-e2e-performance:
@log="/tmp/opencode/test-e2e-performance-$$(date +%Y%m%d-%H%M%S).log"; \
echo "Logging to $$log"; \
env PYTHONUNBUFFERED=1 SQLBUILD_CONCURRENCY=$(SQLBUILD_CONCURRENCY) uv run pytest tests/e2e \
-m "performance and not cold_compile_performance and not cache_compile_performance and not real_warehouse and not dbt" \
-m "performance and not cold_compile_performance and not cache_compile_performance and not preview_performance and not real_warehouse and not dbt" \
-vv --log-cli-level=INFO --color=yes 2>&1 | tee "$$log"; \
status=$${PIPESTATUS[0]}; \
echo "TEST_E2E_PERFORMANCE_EXIT=$$status (log: $$log)" | tee -a "$$log"; \
Expand Down Expand Up @@ -370,13 +371,17 @@ test-e2e-dense-compile-performance:
-n auto --dist loadfile \
-m cold_compile_performance -k "models_$(SQLBUILD_BENCHMARK_MODELS)_" \
-vv -rP --log-level=INFO --log-cli-level=INFO --color=yes
# DuckShop 10K runs after the dense guards, alone, so its compile never overlaps theirs.
# DuckShop 10K runs after the dense guards, alone, so its compile never overlaps theirs; then its
# plan preview guard, which serves the project on a location.
if [ "$(SQLBUILD_BENCHMARK_MODELS)" = "10000" ]; then \
env PYTHONUNBUFFERED=1 uv run pytest benchmarks/duckshop-10k/tests -q --color=yes && \
env PYTHONUNBUFFERED=1 SQLBUILD_CONCURRENCY=4 uv run pytest \
tests/e2e/src/sqlbuild/cli/commands/main/compile/test_duckshop_10k_compile.py \
-m cold_compile_performance -k "models_$(SQLBUILD_BENCHMARK_MODELS)_" \
-vv -rP --log-level=INFO --log-cli-level=INFO --color=yes; \
-vv -rP --log-level=INFO --log-cli-level=INFO --color=yes && \
env PYTHONUNBUFFERED=1 uv run pytest \
tests/e2e/src/sqlbuild/cli/commands/main/plan/test_duckshop_10k_preview_performance.py \
-m preview_performance -vv -rP --log-level=INFO --log-cli-level=INFO --color=yes; \
fi

test-e2e-cache-compile-performance:
Expand Down
25 changes: 24 additions & 1 deletion benchmarks/duckshop-10k/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ marketplace partners, carriers, warehouses, retail stores, suppliers, payment ga
networks, messaging, affiliates, support tools, ledgers, wholesale desks, returns portals and a
product catalogue. Every name and value is invented.

Only the generator, its tests and `checksum.txt` are committed. The generated project is written
Only the generators, their tests and `checksum.txt` are committed. The generated project is written
to a directory you choose.

## Usage
Expand Down Expand Up @@ -64,3 +64,26 @@ because their results depend on the engine or the session.

CI runs both in the 10,000-model cell of the dense compile job, under its 4 GiB memory limit, after
the dense guards have finished, so the DuckShop compile never shares the limit with them.

## Synced warehouse state

`synced_state.py` writes the warehouse state a location syncs after building the project: about 13K
relations, a fingerprint and two `--changed` executions per built model, an execution per seed and
cursor observations, as the metadata pages a location sends the server. No warehouse runs the
project, so the hashes come from the planner itself: a `SQLBUILD_PLANNER_CAPTURE_DIR` capture of
one plan of the published project gives each model's identity metadata and query, and an unedited
model then plans as unchanged. `--changed-share` of the built models record a definition one line
older, and the models outside `--built-share` were never built.

```bash
python benchmarks/duckshop-10k/synced_state.py control.sqlite3 001-plan_output.json state.json \
--built-share 0.95
```

`tests/e2e/.../plan/test_duckshop_10k_preview_performance.py` serves the project on a location,
mirrors that state into the server, and times a warm `--changed` plan preview of every node, with
every model built and with one in twenty unbuilt. It fails when the warm preview takes more than 0.45
of the same location's publish, a linear 10K compile on the same machine, so a lookup over every
model or relation shows however fast the runner is; it also checks that only the edited and unbuilt
models report their own change, and times the ad-hoc command check. It runs in the 10,000-model
cell after the DuckShop compile guard.
Loading
Loading