Repository navigation
feat!: remove the dlt and ingestr integrations - #1554
Merged
Merged
Conversation
Remove the declarative dlt (`dlt_sources`) and ingestr (a source's `ingestr` block) integrations, the loaders SQLBuild generated for them, and the external loader connection mode they used: loaders that ran before SQLBuild opened its warehouse connections, with their own scheduling checks. Every loader now runs on a SQLBuild connection. The `dlt` and `ingestr` extras, the `external_source_load` and `ingestr_command` lifecycle operations and the plan's `integration_kind` field are gone. A sources file that still declares `dlt_sources`, or a source that still declares `ingestr`, fails compile with an error that points to `@loader`. BREAKING CHANGE: the dlt and ingestr integrations are removed. A project that declares `dlt_sources` or a source `ingestr` block no longer compiles; declare the source with `managed: true` and load it with a Python `@loader` named after the source. The `dlt` and `ingestr` package extras no longer exist.
Drop the dlt and ingestr integration pages, their sidebar entries and screenshots, and every mention in the loader, source, adapter, configuration and feature comparison pages, and regenerate the skill references.
…ir pages History written while dlt or ingestr loads ran still names their operations, so the operation catalogue keeps both names for decoding. The removed docs pages redirect to the loaders page, the failure corpus covers both removed-integration errors, and the project assembly goldens record the digests without integration loaders.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The dlt (
dlt_sources) and ingestr (a source'singestrblock) integrations ran loaders outside SQLBuild's warehouse connections, with their own connection mode, checks and plan fields. A Python@loadercovers the same need on a SQLBuild connection.Changes
dltandingestrextras and the plan'sintegration_kindfield. Nothing emits theexternal_source_loadandingestr_commandoperations any more, but stored run history that names them still decodes.dlt_sources, or a source that still declaresingestr, fails compile with an error that names the file or source and points to@loader.dlt_sourceskey andingestrblock. The project assembly goldens are regenerated for the removed fields.BREAKING CHANGE: a project that declares
dlt_sourcesor a sourceingestrblock no longer compiles. Declare the source withmanaged: trueand load it with a Python@loadernamed after the source. Thedltandingestrextras no longer exist.The next integration pull request builds on this one.
Verification
cargo testfor the discovery and core crates; a full extension and CLI build.sqb load.make test: the only local failures were launcher tests that need an installed environment; they pass in CI on the branch built on this one.make check-cion the head.