feat: typed StandardLineage app builder for cross-connection lineage - #1010
Draft
mitshah-atlan wants to merge 2 commits into
Draft
feat: typed StandardLineage app builder for cross-connection lineage#1010mitshah-atlan wants to merge 2 commits into
mitshah-atlan wants to merge 2 commits into
Conversation
Standard Lineage (cross-connection lineage) had no typed builder, so callers had
to hand-build the raw `inputs` dict for `client.app.create/update`. Two details
make that unreasonable to ask of a caller, and neither is expressible from a UI
configmap — hence hand-written, and added to the generator's _HAND_WRITTEN set:
1. `cross_connection_qualified_names` is declared `str` in the app's input
contract but means a LIST of connection qualified names. Sending a native list
fails validateInputsAgainstContract server-side; it has to be json.dumps'd, and
Heracles parses it back into a list for the manifest placeholder. `connections()`
takes a List[str] and encodes it.
2. The defining operation is re-scoping an EXISTING workflow — adding a connection
as it is onboarded — which is an update against a slug. No generated builder
does updates; AppBuilder only has create()/run().
The re-scope path reads before it writes, and that is load-bearing rather than a
convenience: `client.app.update` is a full replace, and the workflow's own
connection entity is republished by the DAG's create-connection node on every run,
so sending a rebuilt or partial connection would overwrite the real one in Atlan
and strip its name and admins. `set_connections` therefore carries the persisted
`connection` and `run_role` over verbatim, read from `client.app.get(slug).dag`
(AppSummary tolerates unmodelled fields, so the DAG arrives as an extra).
Surface:
StandardLineage(client).connection(name=...).connections([...]).run() # create
StandardLineage(client).add_connections(slug, [...]) # onboard
StandardLineage(client).remove_connections(slug, [...]) # hand back
StandardLineage(client).set_connections(slug, [...]) # replace
StandardLineage(client).get_connections(slug) # read
add_connections/remove_connections are idempotent and return None without
publishing a version when nothing would change, so an onboarding portal can replay
safely. Validation is client-side where the error is actionable: the app requires a
non-empty, same-connector scope, and passing the workflow's OWN standard-lineage
connection as its scope — a natural mistake, since both are "connections" — is
rejected with a message that says which is wanted.
Note the two distinct connectors: the workflow's own connection is minted under
`standard-lineage` (_CONNECTOR_NAME), while the `connector` input names the
connector of the connections in scope and is derived from them.
Verified end-to-end against a live tenant (create path deliberately not exercised
there — it would mint a workflow and a connection): the read, add, remove,
idempotent no-op and empty-scope refusal all behave as specified, and the
re-rendered DAG came back with all 14 connection attributes intact, both
downstream nodes identical, and the Temporal workflow type unchanged.
22 new tests; app suite 535 passed, full unit suite 7110 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: mitshah-atlan <mit.shah@atlan.com>
Applies the code-comments guidance to this branch's own diff only. The builder itself needed no tightening -- it already carries contract-level docstrings with no drift hazards and matches the conventions of its neighbours in pyatlan/model/apps/. The one real gap was the opposite of over-writing: the test file had no file header at all. Matches test_asset_export_flows_handwritten.py, the closest analogue in the repo (hand-written tests for a hand-written v3 builder): records that the file survives the generator's regen because it carries no AUTO-GENERATED banner, and why most cases pin the re-scope path rather than create -- client.app.update is a full replace, so what is absent from the payload matters as much as what is present. 22 tests pass; ruff format and check clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: mitshah-atlan <mit.shah@atlan.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Description
Adds a typed
StandardLineagebuilder for the Standard Lineage (cross-connection lineage) app,which had none — callers had to hand-build the raw
inputsdict forclient.app.create/update.Standard Lineage builds lineage across several connections of one connector, from query history
their own miners already extracted. One workflow owns a set of connections, so the defining
operation is re-scoping an existing workflow — adding a connection as it is onboarded. That is
what this PR makes a one-liner:
Why hand-written rather than generated (added to the generator's
_HAND_WRITTENset) — twothings a UI configmap cannot express:
cross_connection_qualified_namesis declaredstrin the app's input contract but means alist of connection qualified names. A native list fails
validateInputsAgainstContractserver-side; it has to be
json.dumps'd, and Heracles parses it back into a list for themanifest placeholder.
connections()takes aList[str]and encodes it.AppBuilderonly has
create()/run().The re-scope path reads before it writes, and that is load-bearing rather than a convenience.
client.app.updateis a full replace, and the workflow's own connection entity is republished bythe DAG's
create-connectionnode on every run. So a rebuilt or partial connection does not merelylose fields in the payload — the next run writes it over the real connection in Atlan, stripping its
name, admin users, admin roles, category and row limit, with no error at any point.
set_connectionstherefore carries the persistedconnectionandrun_roleover verbatim, readfrom
client.app.get(slug).dag(AppSummarytolerates unmodelled fields, so the DAG arrives as anextra).
Surface
add_connections/remove_connectionsare idempotent and returnNonewithout publishing aversion when nothing would change, so an onboarding portal can replay safely.
Validation is client-side, where the error is actionable: the app requires a non-empty,
same-connector scope, and passing the workflow's own
standard-lineageconnection as its scope— a natural mistake, since both are "connections" — is rejected with a message saying which was
wanted.
Note the two distinct connectors: the workflow's own connection is minted under
standard-lineage(
_CONNECTOR_NAME), while theconnectorinput names the connector of the connections in scope andis derived from them.
🧩 Type of change