Skip to content

Commit 86e97b6

Browse files
committed
test(native): refuse vanished fallbacks without native answers and fail closed on goldens
1 parent 89d5a54 commit 86e97b6

13 files changed

Lines changed: 265 additions & 27 deletions

File tree

‎scripts/compiler_differential/README.md‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,24 @@ sets. `--native-fallbacks check` runs in both make targets and fails CI in both
3939
- an entry, engine or corpus count that is not on the list fails;
4040
- a listed entry whose count changed or that no longer occurs also fails, and the message says to
4141
update or remove it, so the fallback list only shrinks;
42-
- a vanished native answer fails as work that now runs in Python.
42+
- a vanished native answer fails as work that now runs in Python;
43+
- a listed fallback that vanished is refused outright, with "fallback disappeared but native
44+
answers did not appear; the stage may be switched off", unless the same engine, stage and
45+
corpus report more `<stage>.native` answers than the list records. A stage with only fallback
46+
entries (`sql_test_glue`, `project_assembly`) would otherwise look like a finished port when it
47+
is switched off.
4348

4449
The counts hold for the make targets' corpus (`--seeds 12`); other seed ranges are refused.
4550

51+
Two stages the list cannot watch:
52+
- `type_system`: the compile corpus never reaches type normalization. Its callers are the planner
53+
and executor and the preview stages' Python fallbacks. `tests/integration/.../type_system/
54+
test_native_type_parity.py` proves instead that the public `normalize_type` asks native first
55+
under `native-preview`.
56+
- `discovery.native model_file_listings`: it is reported unconditionally inside the native model
57+
file discovery, which has no fallback to Python, so this count cannot detect anything. It stays
58+
only as a record that native discovery ran.
59+
4660
**How a PR updates the list and the goldens**
4761

4862
One command rewrites both from the current tree:
@@ -71,6 +85,9 @@ in the same PR as the change that caused it.
7185
in the PR body. Never add one just to make CI pass.
7286
- A removed or lower `<stage>.native` answer count means less native work. It needs a reason
7387
in the PR body.
88+
- A removed fallback entry needs new or higher `<stage>.native` answers for the same stage in the
89+
same diff: the port must report the work it now answers. A fallback that vanished with no new
90+
answers is a stage switched off, not a port; the gate refuses it.
7491
- Code: every new path that hands work to Python calls `report_native_fallback`, and every new
7592
native path calls `report_native_answer` where its result is used.
7693
- Goldens:
@@ -101,7 +118,11 @@ review. Cache facts are covered by the incremental-versus-no-cache tests instead
101118
changes on purpose, and review the diff like code.
102119
- `tests/goldens/compiler/seed_range.toml` records the seed range the goldens were written for.
103120
Every seed in that range, and every fixture, example and failure case, must have a golden.
104-
Seeds outside the range (ad hoc runs with more seeds) are skipped.
121+
Seeds outside the range (ad hoc runs with more seeds) are skipped. A missing `seed_range.toml`
122+
fails the check for every seed project rather than skipping them.
123+
- Dialect variants (`<seed>-postgres`, `<seed>-snowflake`) are generated for the first seed of a
124+
run only, so a golden is required only for the variants of the recorded `seed_start`. An ad hoc
125+
`--seed-start N` run checks its base seeds in the range and skips its variants.
105126
- Once the Python stages are deleted, goldens are rewritten from `native`.
106127

107128
### All-engine user-facing errors

‎scripts/compiler_differential/_helpers/goldens/goldens.py‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -62,9 +62,21 @@ def golden_differences(
6262
path.parent.mkdir(parents=True, exist_ok=True)
6363
_ = path.write_text(golden_text(payloads[0]), encoding="utf-8")
6464
return []
65+
seed_range: Path = golden_dir / GOLDEN_SEED_RANGE_FILE
66+
if path.parent.name == GOLDEN_SEED_CORPUS and not seed_range.is_file():
67+
return [
68+
Difference(
69+
project=project.name,
70+
artifact=f"golden {seed_range.name}",
71+
location="file",
72+
left="<missing>",
73+
right=GOLDEN_MISSING_HINT,
74+
labels=(GOLDEN_LABEL, "hint"),
75+
)
76+
]
6577
if not path.is_file():
6678
if path.parent.name == GOLDEN_SEED_CORPUS and not _seed_in_recorded_range(
67-
golden_dir=golden_dir, name=path.stem
79+
seed_range=seed_range, name=path.stem
6880
):
6981
return []
7082
return [
@@ -194,13 +206,15 @@ def write_golden_seed_range(*, golden_dir: Path, seed_start: int, seeds: int) ->
194206
)
195207

196208

197-
def _seed_in_recorded_range(*, golden_dir: Path, name: str) -> bool:
198-
path: Path = golden_dir / GOLDEN_SEED_RANGE_FILE
199-
if not path.is_file():
200-
return False
201-
recorded: dict[str, object] = tomllib.loads(path.read_text(encoding="utf-8"))
209+
def _seed_in_recorded_range(*, seed_range: Path, name: str) -> bool:
210+
"""Seeds in the recorded range need goldens; dialect variants only for its first seed."""
211+
212+
recorded: dict[str, object] = tomllib.loads(seed_range.read_text(encoding="utf-8"))
202213
start: int = int(str(recorded.get("seed_start", 0)))
203-
index: int = int(name.split("-", 1)[0])
214+
seed, _, dialect = name.partition("-")
215+
index: int = int(seed)
216+
if dialect:
217+
return index == start
204218
return start <= index < start + int(str(recorded.get("seeds", 0)))
205219

206220

‎scripts/compiler_differential/_helpers/running/native_fallbacks.py‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,10 @@ def fallback_problems(
114114
f"{allowed.seeds}; this run used --seed-start {run.seed_start} --seeds {run.seeds}"
115115
]
116116
corpora: frozenset[str] = _corpus_prefixes(run.corpora)
117+
observed_answers: Counter[tuple[str, str, str]] = _answer_totals(observed)
118+
allowed_answers: Counter[tuple[str, str, str]] = _answer_totals(
119+
{**allowed.counts, **allowed.max_counts}
120+
)
117121
problems: list[str] = []
118122
for key in sorted({*allowed.counts, *allowed.max_counts, *observed}):
119123
if key[0] not in run.engines:
@@ -128,6 +132,8 @@ def fallback_problems(
128132
actual=actual.get(corpus, 0),
129133
exact=None if exact is None else exact.get(corpus, 0),
130134
bound=None if bound is None else bound.get(corpus, 0),
135+
answers_rose=observed_answers[(key[0], key[1], corpus)]
136+
> allowed_answers[(key[0], key[1], corpus)],
131137
)
132138
if problem is not None:
133139
problems.append(problem)
@@ -183,7 +189,13 @@ def write_allow_list(
183189

184190

185191
def _corpus_problem(
186-
*, label: str, answer: bool, actual: int, exact: int | None, bound: int | None
192+
*,
193+
label: str,
194+
answer: bool,
195+
actual: int,
196+
exact: int | None,
197+
bound: int | None,
198+
answers_rose: bool,
187199
) -> str | None:
188200
if answer and exact is None and bound is None:
189201
return f"{label}: native answered {actual}, not on the allow-list; record it"
@@ -194,6 +206,12 @@ def _corpus_problem(
194206
)
195207
if exact is None and bound is None:
196208
return f"{label}: {actual} not on the allow-list; port it or list it with a reason"
209+
if actual == 0 and not answers_rose:
210+
return (
211+
f"{label}: fallback disappeared but native answers did not appear; the stage may be "
212+
"switched off. A port that removes a fallback must report native answers for the "
213+
"work it now does"
214+
)
197215
if actual == 0:
198216
return f"{label}: listed but no longer occurs; remove it from the allow-list"
199217
if bound is not None:
@@ -203,6 +221,18 @@ def _corpus_problem(
203221
return None
204222

205223

224+
def _answer_totals(
225+
entries: dict[FallbackKey, dict[str, int]],
226+
) -> Counter[tuple[str, str, str]]:
227+
"""Native answer counts summed by engine, stage and corpus."""
228+
229+
totals: Counter[tuple[str, str, str]] = Counter()
230+
for (engine, stage, site, _), by_corpus in entries.items():
231+
for corpus, count in by_corpus.items():
232+
totals[(engine, stage, corpus)] += count * site.endswith(NATIVE_FALLBACK_ANSWER_SUFFIX)
233+
return totals
234+
235+
206236
def _label(*, key: FallbackKey, corpus: str) -> str:
207237
engine, stage, site, kind = key
208238
return f"{engine} {stage} {site} {kind} ({corpus})"

‎scripts/compiler_differential/constants.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,9 @@
3737
NATIVE_FALLBACK_LIST_HEADER: str = (
3838
"# Every native-to-Python fallback and preview analysis deferral the CI corpus may still\n"
3939
"# reach, with its exact count per corpus. `--native-fallbacks check` fails on anything new,\n"
40-
"# changed or gone. Ports delete their entries; see scripts/compiler_differential/README.md."
40+
"# changed or gone. Ports delete their entries; see scripts/compiler_differential/README.md.\n"
41+
"# The corpus does not reach type normalization (type_system): its callers are the planner,\n"
42+
"# the executor and the preview fallbacks, so test_native_type_parity.py covers that stage."
4143
)
4244
RECORDS_DIRECTORY: str = "records"
4345
GOLDEN_DIRECTORY: str = "tests/goldens/compiler"

‎scripts/compiler_differential/native_fallbacks.toml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# Every native-to-Python fallback and preview analysis deferral the CI corpus may still
22
# reach, with its exact count per corpus. `--native-fallbacks check` fails on anything new,
33
# changed or gone. Ports delete their entries; see scripts/compiler_differential/README.md.
4+
# The corpus does not reach type normalization (type_system): its callers are the planner,
5+
# the executor and the preview fallbacks, so test_native_type_parity.py covers that stage.
46
seed_start = 0
57
seeds = 12
68

@@ -113,8 +115,8 @@ counts = { example = 12, failure = 286, fixture = 12, seed = 50 }
113115
engine = "native"
114116
stage = "macro_call_store"
115117
site = "macro_call_store.native"
116-
kind = "store_attachments"
117-
counts = { example = 12, failure = 231, fixture = 12, seed = 47 }
118+
kind = "served_calls"
119+
counts = { example = 18, fixture = 15, seed = 192 }
118120

119121
[[entry]]
120122
engine = "native"
@@ -407,8 +409,8 @@ counts = { example = 9, failure = 106, fixture = 6, seed = 22 }
407409
engine = "native-preview"
408410
stage = "macro_call_store"
409411
site = "macro_call_store.native"
410-
kind = "store_attachments"
411-
counts = { example = 12, failure = 231, fixture = 12, seed = 47 }
412+
kind = "served_calls"
413+
counts = { example = 18, fixture = 15, seed = 192 }
412414

413415
[[entry]]
414416
engine = "native-preview"

‎src/sqlbuild/compiler/compile/_helpers/macro_bridge/call_store.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@
77
from sqlbuild.compiler.compile._helpers.attachment.target import build_compile_target_context
88
from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs
99
from sqlbuild.compiler.frontier.main.native_stage_enabled import native_stage_enabled
10-
from sqlbuild.compiler.frontier.main.report_native_answer import report_native_answer
1110
from sqlbuild.compiler.frontier.types import NativeStage
1211
from sqlbuild.compiler.macro_bridge.classes.macro_bridge import MacroBridge
1312
from sqlbuild.compiler.macro_bridge.main._claim_first_compile import claim_first_compile
@@ -46,7 +45,6 @@ def target_context_with_macro_call_store(
4645
project_dir=project_dir,
4746
model_paths=_model_paths(discovered_inputs=discovered_inputs, project_dir=project_dir),
4847
)
49-
report_native_answer(stage=NativeStage.MACRO_CALL_STORE, kind="store_attachments")
5048
return target_context
5149

5250

‎src/sqlbuild/compiler/macro_bridge/main/run_with_macro_bridge.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,9 @@
99
from contextvars import Token
1010

1111
import sqlbuild._native as _native
12+
from sqlbuild.compiler.frontier.main.report_native_answer import report_native_answer
1213
from sqlbuild.compiler.frontier.main.report_native_fallback import report_native_fallback
13-
from sqlbuild.compiler.frontier.types import NativeFallbackSite
14+
from sqlbuild.compiler.frontier.types import NativeFallbackSite, NativeStage
1415
from sqlbuild.compiler.macro_bridge.classes.macro_bridge import MacroBridge
1516
from sqlbuild.compiler.macro_bridge.constants import ACTIVE_MACRO_BRIDGE, STORE_MODULE_PATHS
1617

@@ -34,6 +35,7 @@ def run_with_macro_bridge[T](*, stage: Callable[[], T]) -> T:
3435
STORE_MODULE_PATHS.update(bridge.store_module_paths())
3536
hits, misses, recorded = bridge.stats()
3637
store_hits, store_records = bridge.store_stats()
38+
report_native_answer(stage=NativeStage.MACRO_CALL_STORE, kind="served_calls", units=store_hits)
3739
logging.getLogger(__name__).debug(
3840
"Native macro bridge: %d memo hits, %d store hits, %d misses, %d recorded calls, "
3941
"%d stored calls",

‎tests/e2e/scripts/compiler_differential/test_native_fallback_gate.py‎

Lines changed: 50 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@
3131
"semantic_checks",
3232
)
3333
_ANSWER_VANISHED: str = "native no longer answers here, so this work now runs in Python"
34+
_FALLBACK_WITHOUT_ANSWERS: str = (
35+
"fallback disappeared but native answers did not appear; the stage may be switched off"
36+
)
3437
_VANISHED_ENTRY: str = (
3538
'\n[[entry]]\nengine = "native-preview"\nstage = "reference_extraction"\n'
3639
'site = "reference_extraction.scan"\nkind = "deferred"\ncounts = { project = 1 }\n'
@@ -48,14 +51,13 @@
4851
expected_lines=("Compiler differential passed: 1 projects identical",),
4952
),
5053
NativeFallbackGateTestCase(
51-
description="entry_that_no_longer_occurs_fails",
54+
description="entry_that_no_longer_occurs_without_new_native_answers_fails",
5255
perturbation="",
5356
appended_entries=_VANISHED_ENTRY,
5457
expected_exit_code=1,
5558
expected_lines=(
5659
"Native fallback allow-list: native-preview reference_extraction "
57-
"reference_extraction.scan deferred (project): listed but no longer occurs; "
58-
"remove it from the allow-list",
60+
f"reference_extraction.scan deferred (project): {_FALLBACK_WITHOUT_ANSWERS}",
5961
),
6062
),
6163
NativeFallbackGateTestCase(
@@ -142,5 +144,50 @@ def test_given_recorded_allow_list_when_native_defers_more_or_less_then_the_gate
142144
assert all(line in output for line in test_case.expected_lines), output
143145

144146

147+
@pytest.mark.parametrize(
148+
"test_case",
149+
[
150+
NativeFallbackGateTestCase(
151+
description="sql_test_glue_switched_off_is_not_a_finished_port",
152+
perturbation=stage_disable_sabotage(("sql_test_glue",)),
153+
appended_entries="",
154+
expected_exit_code=1,
155+
expected_lines=(
156+
"Native fallback allow-list: native-preview sql_test_glue sql_test_glue.assembly "
157+
f"deferred (fixture): {_FALLBACK_WITHOUT_ANSWERS}",
158+
"Native fallback allow-list: native-preview sql_test_glue sql_test_glue.planning "
159+
f"deferred (fixture): {_FALLBACK_WITHOUT_ANSWERS}",
160+
"Native fallback allow-list: 2 problems",
161+
),
162+
),
163+
],
164+
ids=lambda case: case.description,
165+
)
166+
def test_given_committed_allow_list_when_fallback_only_stage_is_switched_off_then_gate_refuses(
167+
test_case: NativeFallbackGateTestCase, tmp_path: Path, capsys: pytest.CaptureFixture[str]
168+
) -> None:
169+
sabotage: Path = write_native_perturbation(tmp_path / "sabotage", source=test_case.perturbation)
170+
_ = capsys.readouterr()
171+
172+
exit_code: int = run_compiler_differential(
173+
[
174+
"--corpus",
175+
"fixtures",
176+
"--jobs",
177+
"2",
178+
"--work-dir",
179+
str(tmp_path / "check"),
180+
"--native-fallbacks",
181+
"check",
182+
"--engine-env",
183+
f"native-preview:PYTHONPATH={sabotage}",
184+
]
185+
)
186+
187+
output: str = capsys.readouterr().out
188+
assert exit_code == test_case.expected_exit_code, output
189+
assert all(line in output for line in test_case.expected_lines), output
190+
191+
145192
if __name__ == "__main__":
146193
pytest.main([__file__, "-vv"])

‎tests/integration/src/sqlbuild/adapter/type_system/_test_types.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,15 @@ class DeepTypeTestCase:
3333
description: str
3434
type_sql: str
3535
expected_native: None
36+
37+
38+
@dataclass(frozen=True)
39+
class PublicNativeTypeTestCase:
40+
"""Types normalized through the public entry point under the preview engine."""
41+
42+
description: str
43+
engine: str
44+
dialect: str
45+
type_strings: tuple[str, ...]
46+
expected_native_calls: tuple[tuple[str, str], ...]
47+
expected_answered: int

0 commit comments

Comments
 (0)