Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/bump-sdk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Run it in CI's environment, or the goldens pick up local drift:
- Python: CI installs the managed interpreters with `./task install-pythons`; do the same.
- Go: the `toolchain` in `go.mod`. A newer local `go` also changes goldens and generated `gofmt` output; use `GOTOOLCHAIN=<pinned>` if they differ.

Before going on, account for every changed file outside the generated set, including the bumper's (its environment is not CI's either): a bump without fallout changes `cmd/**`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `python/databricks/bundles/**`, `acceptance/bundle/refschema/out.fields.txt` and help-text goldens (`acceptance/help/output.txt`, `acceptance/cmd/account/account-help/output.txt`).
Before going on, account for every changed file outside the generated set, including the bumper's (its environment is not CI's either): a bump without fallout changes `cmd/**`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `python/databricks/bundles/**`, `acceptance/bundle/refschema/output` and help-text goldens (`acceptance/help/output.txt`, `acceptance/cmd/account/account-help/output.txt`).
Revert any golden change you can't attribute to the new SDK or spec.

**5. Handle SDK breaking changes.**
Expand Down
16 changes: 8 additions & 8 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -799,7 +799,7 @@ tasks:
# .codegen/cli.json (no universe checkout needed). Refreshing cli.json
# itself from the OpenAPI spec is the separate `generate-clijson` task.
- task: generate-cligen
# Refreshes acceptance/bundle/refschema/out.fields.txt, which feeds
# Refreshes acceptance/bundle/refschema/output/*.txt, which feeds
# generate-direct-apitypes and generate-direct-resources below.
- task: generate-refschema
- task: generate-schema
Expand Down Expand Up @@ -970,12 +970,12 @@ tasks:
- task: generate-check
- task: test-update

# Refreshes out.fields.txt, which records the field paths / types emitted by
# Refreshes acceptance/bundle/refschema/output/*.txt, which record the field paths / types emitted by
# `bundle debug refschema` (see cmd/bundle/debug/refschema.go). It reflects
# over types reachable from bundle/, so any bundle change or SDK bump (go.mod
# / go.sum) can affect the output.
generate-refschema:
desc: Regenerate acceptance/bundle/refschema/out.fields.txt
desc: Regenerate acceptance/bundle/refschema/output/*.txt
sources:
- bundle/**/*.go
- exclude: bundle/**/*_test.go
Expand All @@ -985,7 +985,7 @@ tasks:
- go.sum
- "{{.EMBED_SOURCES}}"
generates:
- acceptance/bundle/refschema/out.fields.txt
- acceptance/bundle/refschema/output/*.txt
cmds:
- go test ./acceptance -run TestAccept/bundle/refschema -update

Expand Down Expand Up @@ -1038,11 +1038,11 @@ tasks:
sources:
- bundle/direct/tools/generate_apitypes.py
- .codegen/cli.json
- acceptance/bundle/refschema/out.fields.txt
- acceptance/bundle/refschema/output/*.txt
generates:
- bundle/direct/dresources/apitypes.generated.yml
cmds:
- "sh -c 'uv run -p \">=3.11\" --no-project python bundle/direct/tools/generate_apitypes.py .codegen/cli.json acceptance/bundle/refschema/out.fields.txt > bundle/direct/dresources/apitypes.generated.yml'"
- "sh -c 'uv run -p \">=3.11\" --no-project python bundle/direct/tools/generate_apitypes.py .codegen/cli.json acceptance/bundle/refschema/output > bundle/direct/dresources/apitypes.generated.yml'"

generate-direct-resources:
desc: Generate direct engine resources YAML
Expand All @@ -1052,11 +1052,11 @@ tasks:
- .codegen/cli.json
- bundle/direct/dresources/apitypes.generated.yml
- bundle/direct/dresources/apitypes.yml
- acceptance/bundle/refschema/out.fields.txt
- acceptance/bundle/refschema/output/*.txt
generates:
- bundle/direct/dresources/configs/*.generated.yml
cmds:
- "uv run --script bundle/direct/tools/generate_resources.py .codegen/cli.json bundle/direct/dresources/apitypes.generated.yml bundle/direct/dresources/apitypes.yml acceptance/bundle/refschema/out.fields.txt bundle/direct/dresources/configs"
- "uv run --script bundle/direct/tools/generate_resources.py .codegen/cli.json bundle/direct/dresources/apitypes.generated.yml bundle/direct/dresources/apitypes.yml acceptance/bundle/refschema/output bundle/direct/dresources/configs"

# pydabs-* tasks are defined in python/Taskfile.yml (included above).

Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/debug/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Usage:
databricks bundle debug [command]

Available Commands:
refschema Dump all relevant fields all bundle resources
refschema Dump all relevant fields of the bundle config and its resources
states Show available state files

Flags:
Expand Down
10 changes: 5 additions & 5 deletions acceptance/bundle/empty_string_dropped/gen_empty_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
how we avoid mutually-exclusive blocks (aws vs gcp attributes, git_source vs
none).

A field is settable/eligible when out.fields.txt types it as `string` (this
A field is settable/eligible when refschema output types it as `string` (this
skips enums, which are named types) with flag ALL or INPUT, and it is not
output_only (<resource_type>.generated.yml) or a bundle-framework field.

Expand All @@ -22,7 +22,7 @@

import yaml

FIELDS = Path("acceptance/bundle/refschema/out.fields.txt")
FIELDS = Path("acceptance/bundle/refschema/output")
CONFIGS = Path("bundle/direct/dresources/configs")
GENERATED_SUFFIX = ".generated.yml"
TESTDIR = Path("acceptance/bundle/empty_string_dropped")
Expand All @@ -35,14 +35,14 @@
def string_leaf_parents():
"""Map resource type -> {parent_schema_path: {leaf field names}}.

Schema paths mirror out.fields.txt: dotted segments, list elements marked
Schema paths mirror refschema output: dotted segments, list elements marked
with "[*]" (e.g. "job_clusters[*].new_cluster"). Only string leaves that are
real object fields are kept: names ending in "[*]" (string-array elements) or
"*" (map values) are skipped.
"""
pat = re.compile(r"^resources\.([a-z_]+)\.\*\.(.+)$")
result = {}
for line in FIELDS.read_text().splitlines():
for line in "\n".join(f.read_text() for f in sorted(FIELDS.glob("*.txt"))).splitlines():
parts = line.split("\t")
if len(parts) < 3:
continue
Expand Down Expand Up @@ -76,7 +76,7 @@ def output_only_fields():
def fill(node, schema_path, parents, excluded):
"""Recursively set settable string leaves under node to ""; recurse into blocks.

schema_path uses out.fields.txt syntax (list elements marked "[*]") to look up
schema_path uses refschema output syntax (list elements marked "[*]") to look up
eligible leaves for the current object.
"""
if isinstance(node, dict):
Expand Down
Loading
Loading