Skip to content

upstream_merge_guide.md

Latest commit

 

History

History
216 lines (171 loc) · 12.1 KB

upstream_merge_guide.md

File metadata and controls

216 lines (171 loc) · 12.1 KB
Raw

Upstream Merge & Sync Guide: dataform-co/dataform → sqlanvil

This guide outlines a highly robust, sandboxed process for pulling, merging, and resolving conflicts when syncing changes from Google's upstream dataform-co/dataform releases (such as 3.0.58) into the SQLAnvil codebase.

Run every command here from the root of the SQLAnvil/sqlanvil engine checkout, not from this docs repo. Paths like ./scripts/... refer to that repo.

0. Last Sync & The One Key Principle

Last synced: upstream 3.0.61 → main on 2026-07-02 (merge commit e12ee80f). This sync also adopted the bzlmod migration (upstream #2187, issue #41): Bazel 5.4.0 → 7.3.2, new MODULE.bazel + MODULE.bazel.lock, WORKSPACE trimmed to just build_bazel_rules_nodejs. All the df/_main workspace-name references were adapted to sa (MODULE.bazel module(name), tools/ts_library.bzl module_name, tools/ts_proto_library.bzl workspace_name fallback, protos/BUILD module_name, both webpack.config.js aliases, cli/index_test_base.ts runfiles path). macOS gotchas fixed: (1) upstream's sed -i in ts_proto_library.bzl broke native BSD sed → made portable (temp-file + mv); (2) bazel build //... needs --config=macos-cpp (ccache bypass + abseil -Wno-deprecated-builtins) to compile the C++ protoc tool — the TS deliverables (core/cli/packages/tests) build without it. Prior sync: 3.0.59 → main 2026-06-03 (dfeb1e5d); DF_VERSION had been bumped to 3.0.60 without a code merge. The local dataform branch mirrors the upstream release tag; each sync advances main's merge-base with upstream, so the next merge only sees new commits.

The #1 lesson: the Bazel build — not grep — is the source of truth for rename leftovers. Git auto-merges most upstream changes cleanly, but those auto-merged regions carry upstream dataform. / df/ / @dataform/ / __dataform_* tokens that produce no conflict markers yet still break the renamed fork. tsc (via Bazel) flags every one. Grep is unreliable here: macOS BSD grep silently no-ops on \bword\b and on -r given an explicit file list. Finish every sync by building, fixing what tsc reports, and rebuilding — never trust a clean grep alone.

1. Upstream Sync Process Architecture

To ensure your primary local main branch remains 100% stable during the merge, always perform the merge inside a temporary sandbox branch before merging back into main.

                   upstream/main (Google)
                         │
                         ├── (Tagged Release: e.g. 3.0.58)
                         ▼
             [1. Fetch tag over HTTPS]
                         │
                         ▼
            [2. Create sandbox branch]
           `upstream-sync/dataform-3.0.58`
                         │
                         ▼
          [3. Execute Merge & Resolve Conflicts]
          - Mechanical import renames (df/ → sa/)
          - Proto packages (dataform → sqlanvil)
                         │
                         ▼
              [4. Run Bazel Verification]
             `./scripts/docker-bazel test //...`
                         │
                         ▼
              [5. Merge back to main]

2. Step-by-Step Execution Guide

Step 1: Fetch Upstream Releases

Ensure your upstream remote is configured using HTTPS (to unblock sandbox egress) and fetch all tags:

# 1. Update upstream URL to HTTPS
git remote set-url upstream https://github.com/dataform-co/dataform.git

# 2. Fetch latest releases & tags
git fetch upstream --tags

Step 2: Create a Sandbox Sync Branch

Checkout a fresh sandbox branch from your local stable main branch:

git checkout main
git checkout -b upstream-sync/3.0.58

Step 3: Run the Merge

Attempt to merge the targeted release tag (e.g. 3.0.58) into the sandbox:

git merge 3.0.58

Step 4: Check dependency resolutions drift

Upstream adds security pins to the resolutions block in package.json via Dependabot PRs that land on their main between tagged releases. Because we take upstream selectively rather than merging every tag wholesale, these are easy to miss — a one-line change buried in a release diff, with nothing about a normal sync drawing attention to it. They are not cosmetic: yarn.lock is consumed by yarn_install in WORKSPACE, so whatever it resolves reaches the build.

./scripts/check_upstream_resolutions              # vs upstream/main
./scripts/check_upstream_resolutions 3.0.70       # vs a specific tag

Exit 0 means we cover everything upstream pins. Exit 1 lists what to add. Pins we carry that upstream lacks are reported but never fail — we ship adapters (pg, mysql) they do not, so we legitimately pin things they never see.

If it reports missing pins, add them to package.json, then:

yarn install    # regenerate yarn.lock; expect transitive packages to drop out

Read the advisory behind each one rather than assuming it is only a version bump, and re-run the verification in §4 — a resolution can force a major version on a transitive dep (e.g. brace-expansion 1.x → 5.x), which yarn will warn about and which only the build can vindicate.

Found the hard way on 2026-09-19: the fork was three pins behind (braces, brace-expansion, linkify-it), all three resolving versions with advisories. Upstream PR #2327 supplied only one of the three; the other two had been sitting upstream for longer. Hence a scripted check rather than an eyeball.

3. Anticipated Conflicts & Resolution Playbook

Since the Dataform $\rightarrow$ SQLAnvil rename touches namespaces and import paths, the merge will trigger a small number of predictable conflicts. Use this playbook to resolve them:

A. Conflict Type: Protobuf Packages (protos/core.proto)

Conflict: Upstream adds new protobuf fields inside package dataform; whereas SQLAnvil uses package sqlanvil;.

  • Resolution:
    • Keep SQLAnvil's package declaration: package sqlanvil;.
    • Copy the new fields added by Google (e.g., string jit_code = ... inside Assertion message) and insert them using SQLAnvil naming conventions.

B. Conflict Type: TypeScript Imports (df/ vs sa/)

Conflict: Upstream imports use df/, e.g.:

import { ActionBuilder } from "df/core/actions";

SQLAnvil uses sa/:

import { ActionBuilder } from "sa/core/actions/base";
  • Resolution:
    • Standardize all new/merged imports to use the sa/ prefix.

C. Conflict Type: Code References (dataform. vs sqlanvil.)

Conflict: Upstream TypeScript code references Google's generated proto namespace dataform.Assertion, while SQLAnvil uses sqlanvil.Assertion.

  • Resolution:
    • Globally replace the merged references to use sqlanvil. instead of dataform..

D. Conflict Type: Auto-merged regions reintroduce dataform tokens (the silent one)

Not a git conflict. Upstream code that auto-merges cleanly — new helper bodies, new test cases, files upstream rewrote wholesale — arrives carrying dataform.X, df/… imports, @dataform/*, or __dataform_current_file. No conflict markers, but it compile-breaks the fork.

  • Resolution:
    • After resolving the visible conflicts, build and rename every token tsc reports: ./scripts/docker-bazel build //core/... //cli/... //protos/... --jobs=2 --local_ram_resources=2048
    • When upstream rewrote a whole file's apparatus (e.g. cli/vm/compile.ts caller-file machinery in 3.0.59), don't resolve hunk-by-hunk — git checkout --theirs <file>, then re-apply the rename. Piecemeal resolution leaves auto-merged code referencing variables only the upstream side defines (e.g. coreBundlePath, needsCallerFileShim).

E. Specific landmines seen in the 3.0.59 sync

  • Caller-file global: the exposed sandbox global must stay __sqlanvil_current_file (read by core/utils.ts); rename upstream's __dataform_current_file to it. __df_enter/__df_exit/__df_current are internal helper names — fine to leave.
  • dataform.json clean break: never reintroduce the hasDataformJson / global.dataformJson handling upstream adds — SQLAnvil reads only workflow_settings.yaml.
  • Extracted helpers: when upstream moves logic into helpers (e.g. executionSql.createTableTasks/Operation/Assertion), the rename must follow into the auto-merged helper bodies; prior SQLAnvil behavior (e.g. disabled-action handling) is usually preserved inside them — verify rather than re-add.
  • CLI install-path tests: upstream tests that npm i @dataform/core@<ver> become @sqlanvil/core@<ver> (unpublished) — they compile but fail at runtime. Skip or adapt; don't let them block the sync.
  • df_ in generated SQL / test fixtures: watch for non-namespace leftovers like df_osc_, _df_temp_, df_integration_test → rename to sa_. Casing artifacts too (readsqlanvil… → readSqlanvil…).

4. Verification & Clean-Up

Once all conflicts are resolved, run the full validation suite. Run these natively — pass --config=macos-cpp, which supplies the clang action env the protobuf C++ compile needs.

# 1. Build everything affected — THIS is what catches reintroduced dataform tokens
bazel build //core/... //cli/... //protos/... --config=macos-cpp

# 2. Run core compiler tests (the authoritative signal; cli e2e suites are flaky)
bazel test //core/... //cli/... --config=macos-cpp

# 3. Run the newly updated integration tests (needs the local Docker DBs up)
PG_HOST=localhost PG_PORT=5432 bazel test //tests/integration:postgres.spec --config=macos-cpp \
  --test_env=PG_HOST --test_env=PG_PORT --test_env=PG_USER --test_env=PG_PASSWORD --test_env=PG_DATABASE
Fallback: scripts/docker-bazel

Native macOS Bazel used to fail with a wrapped_clang / dyld LC_UUID toolchain error compiling protobuf C++, and this guide routed everything through the container as a result. That is resolved — native with --config=macos-cpp is the primary path as of 2026-09-19. The container route is kept for when the native toolchain breaks again or you need to reproduce CI exactly.

Always pass --jobs=2 --local_ram_resources=2048: the in-container Bazel JVM gets OOM-killed (Socket closed, error 14) under default parallelism during webpack bundling. Note the integration tests need PG_HOST=host.docker.internal there rather than localhost.

./scripts/docker-bazel build //core/... //cli/... //protos/... --jobs=2 --local_ram_resources=2048
./scripts/docker-bazel test //core/... --jobs=2 --local_ram_resources=2048
PG_HOST=host.docker.internal PG_PORT=5432 ./scripts/docker-bazel test //tests/integration:postgres.spec \
  --test_env=PG_HOST --test_env=PG_PORT --test_env=PG_USER --test_env=PG_PASSWORD --test_env=PG_DATABASE

If the build completes and all tests pass:

# 3. Checkout main & merge the verified sync branch
git checkout main
git merge upstream-sync/3.0.58

# 4. Clean up the sandbox branch
git branch -d upstream-sync/3.0.58

# 5. Push updated main to origin (GitHub)
git push origin main

5. Local fixes reported upstream (converge on adoption)

Bugs we fixed locally that are also present upstream and that we've reported to dataform-co/dataform. On each sync, check whether upstream adopted the fix — if so, take their version and drop our local change so the file stops diverging (less future merge friction). If not, keep ours and re-apply over the merge.

File Local fix Upstream issue / PR Status
common/flags/index.ts Lenient arg parser — ignore non-flag tokens instead of throwing Arg neither flag name nor flag value (which crashed the CLI when a positional followed a flag). Extracted parseArgvFlags() + test. issue dataform-co/dataform#2198 → fixed by PR #2199 PR open, not merged (not on main, not tagged) — watch #2199 for the convergence trigger