Skip to content

feat: typed StandardLineage app builder for cross-connection lineage - #1010

Draft
mitshah-atlan wants to merge 2 commits into
mainfrom
feat/standard-lineage-app-builder
Draft

feat: typed StandardLineage app builder for cross-connection lineage#1010
mitshah-atlan wants to merge 2 commits into
mainfrom
feat/standard-lineage-app-builder

Conversation

@mitshah-atlan

@mitshah-atlan mitshah-atlan commented Aug 17, 2026

Copy link
Copy Markdown

✨ Description

Adds a typed StandardLineage builder for the Standard Lineage (cross-connection lineage) app,
which had none — callers had to hand-build the raw inputs dict for client.app.create/update.

Standard Lineage builds lineage across several connections of one connector, from query history
their own miners already extracted. One workflow owns a set of connections, so the defining
operation is re-scoping an existing workflow — adding a connection as it is onboarded. That is
what this PR makes a one-liner:

StandardLineage(client).add_connections(slug, ["default/bigquery/1700000002"])

Why hand-written rather than generated (added to the generator's _HAND_WRITTEN set) — two
things a UI configmap cannot express:

  1. cross_connection_qualified_names is declared str in the app's input contract but means a
    list of connection qualified names. A native list fails validateInputsAgainstContract
    server-side; it has to be json.dumps'd, and Heracles parses it back into a list for the
    manifest placeholder. connections() takes a List[str] and encodes it.
  2. Re-scoping is an update against a slug. No generated builder does updates — AppBuilder
    only has create()/run().

The re-scope path reads before it writes, and that is load-bearing rather than a convenience.
client.app.update is a full replace, and the workflow's own connection entity is republished by
the DAG's create-connection node on every run. So a rebuilt or partial connection does not merely
lose fields in the payload — the next run writes it over the real connection in Atlan, stripping its
name, admin users, admin roles, category and row limit, with no error at any point.
set_connections therefore carries the persisted connection and run_role over verbatim, read
from client.app.get(slug).dag (AppSummary tolerates unmodelled fields, so the DAG arrives as an
extra).

Surface

StandardLineage(client).connection(name=...).connections([...]).run()   # create
StandardLineage(client).add_connections(slug, [...])                    # onboard a connection
StandardLineage(client).remove_connections(slug, [...])                 # hand one back
StandardLineage(client).set_connections(slug, [...])                    # replace the scope
StandardLineage(client).get_connections(slug)                           # read the scope

add_connections / remove_connections are idempotent and return None without publishing a
version when nothing would change, so an onboarding portal can replay safely.

Validation is client-side, where the error is actionable: the app requires a non-empty,
same-connector scope, and passing the workflow's own standard-lineage connection as its scope
— a natural mistake, since both are "connections" — is rejected with a message saying which was
wanted.

Note the two distinct connectors: the workflow's own connection is minted under standard-lineage
(_CONNECTOR_NAME), while the connector input names the connector of the connections in scope and
is derived from them.

🧩 Type of change

  • 🚀 New feature (non-breaking change that adds functionality)
  • 🐛 Bug fix
  • 🔄 Refactor
  • 🧹 Maintenance
  • 💥 Breaking change
  • 📦 Dependency upgrade/downgrade
  • 📚 Documentation updates

mitshah-atlan and others added 2 commits August 17, 2026 17:40
Standard Lineage (cross-connection lineage) had no typed builder, so callers had
to hand-build the raw `inputs` dict for `client.app.create/update`. Two details
make that unreasonable to ask of a caller, and neither is expressible from a UI
configmap — hence hand-written, and added to the generator's _HAND_WRITTEN set:

1. `cross_connection_qualified_names` is declared `str` in the app's input
   contract but means a LIST of connection qualified names. Sending a native list
   fails validateInputsAgainstContract server-side; it has to be json.dumps'd, and
   Heracles parses it back into a list for the manifest placeholder. `connections()`
   takes a List[str] and encodes it.

2. The defining operation is re-scoping an EXISTING workflow — adding a connection
   as it is onboarded — which is an update against a slug. No generated builder
   does updates; AppBuilder only has create()/run().

The re-scope path reads before it writes, and that is load-bearing rather than a
convenience: `client.app.update` is a full replace, and the workflow's own
connection entity is republished by the DAG's create-connection node on every run,
so sending a rebuilt or partial connection would overwrite the real one in Atlan
and strip its name and admins. `set_connections` therefore carries the persisted
`connection` and `run_role` over verbatim, read from `client.app.get(slug).dag`
(AppSummary tolerates unmodelled fields, so the DAG arrives as an extra).

Surface:

    StandardLineage(client).connection(name=...).connections([...]).run()   # create
    StandardLineage(client).add_connections(slug, [...])                    # onboard
    StandardLineage(client).remove_connections(slug, [...])                 # hand back
    StandardLineage(client).set_connections(slug, [...])                    # replace
    StandardLineage(client).get_connections(slug)                           # read

add_connections/remove_connections are idempotent and return None without
publishing a version when nothing would change, so an onboarding portal can replay
safely. Validation is client-side where the error is actionable: the app requires a
non-empty, same-connector scope, and passing the workflow's OWN standard-lineage
connection as its scope — a natural mistake, since both are "connections" — is
rejected with a message that says which is wanted.

Note the two distinct connectors: the workflow's own connection is minted under
`standard-lineage` (_CONNECTOR_NAME), while the `connector` input names the
connector of the connections in scope and is derived from them.

Verified end-to-end against a live tenant (create path deliberately not exercised
there — it would mint a workflow and a connection): the read, add, remove,
idempotent no-op and empty-scope refusal all behave as specified, and the
re-rendered DAG came back with all 14 connection attributes intact, both
downstream nodes identical, and the Temporal workflow type unchanged.

22 new tests; app suite 535 passed, full unit suite 7110 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: mitshah-atlan <mit.shah@atlan.com>
Applies the code-comments guidance to this branch's own diff only.

The builder itself needed no tightening -- it already carries contract-level
docstrings with no drift hazards and matches the conventions of its neighbours in
pyatlan/model/apps/. The one real gap was the opposite of over-writing: the test
file had no file header at all.

Matches test_asset_export_flows_handwritten.py, the closest analogue in the repo
(hand-written tests for a hand-written v3 builder): records that the file survives
the generator's regen because it carries no AUTO-GENERATED banner, and why most
cases pin the re-scope path rather than create -- client.app.update is a full
replace, so what is absent from the payload matters as much as what is present.

22 tests pass; ruff format and check clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: mitshah-atlan <mit.shah@atlan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant