EEADataLakehouse

Two main class domains for the EEA data lakehouse:

• eea_datalakehouse.dds_ingestion — notebook-side client for the Dremio Document Service (DDS) Ingest API (DI-8.4/8.5). FolderIngest transfers a folder of data files to S3 (via DDS-issued presigned URLs only — never an S3 SDK or S3 credentials) and registers it as a Dremio table, coordinated entirely through the DDS REST API.
• eea_datalakehouse.catalog — Dremio catalog operations once data has landed: converting a table to a view, copying/moving/deleting catalog entries, managing folders, wiki text and tags, and listing what's there. datacopy/datamove always run over Arrow Flight SQL; everything else runs over Dremio's REST SQL Jobs API (folders/wiki/tags have no SQL equivalent at all, so those go through Dremio's own catalog REST API directly) — and every operation treats a stalled call as a Dremio engine cold-starting rather than a failure, so it can be remembered and retried later instead of blocking.

Install

These badges are live — each one queries the GitHub API directly and always shows whatever tag is currently released for that branch, updating on its own every time main/staging cuts a new release (see Releasing a new version).

@main/@staging always installs whatever was most recently released for that branch — a floating tag sharing the branch's own name, moved forward to the latest release automatically each time one is cut, so there's nothing to look up or keep in sync yourself:

pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@main"       # latest stable
pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"    # latest early access

staging's latest release (early access) — pin to the tag the "staging" badge above shows

pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12-staging"

## Usage

Dremio credentials are read from the injected kernel env vars `_DREMIO_USER` /
`_DREMIO_PWD`, and the service URL from `DDS_BASE_URL`. None of these are ever
logged or printed.

```python
from eea_datalakehouse.dds_ingestion import FolderIngest

outcome = FolderIngest(
    folder="./my_data",
    target_catalog_path="biodiversity.uploads",
    data_format="parquet",      # one of parquet | csv | json
    intent="read_only",         # or "editable"
    conflict_mode="fail",
    parallelism=4,               # concurrent uploads (default 4)
).run()

print(outcome.commit.table_path, outcome.commit.record_count)

run() performs begin → upload(all files) → commit. Re-running the same session resumes by skipping files the server reports as already uploaded.

from eea_datalakehouse.catalog import Catalog, EngineStartingError

# base_url/token are Dremio's own (not DDS). `username` is only needed for
# datacopy/datamove (Arrow Flight's basic-auth handshake) — everything else
# runs over Dremio's REST SQL Jobs API regardless of EEA_CATALOG_TRANSPORT.
with Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME) as catalog:
    try:
        catalog.table2view(
            "bwd.consumer", "bwd.draft.bw_assessment", idempotency_key="bwd-v2025_1"
        )
    except EngineStartingError:
        # A cold Dremio engine looks like a stalled call, not a failure — the
        # attempt is remembered under its idempotency_key; call it again later:
        catalog.retry_pending("bwd-v2025_1")

    info = catalog.gettableitemsfrom("bwd.versions.v2025_1.assessments", idempotency_key="check-1")
    print(info.schema, info.row_count)             # schema + row count, no rows fetched

    catalog.gettablesfrom("bwd", idempotency_key="list-1")   # recurses into every subfolder
# `with` calls catalog.close() on exit, disposing the REST/Flight session(s).
# If you don't use `with`, call catalog.close() yourself when done — a
# process-exit/SIGTERM fallback covers a caller that forgets either way.

Every operation takes idempotency_key= (keyword-only). On EngineStartingError — a stalled call, most likely a cold-starting Dremio engine, not a real failure — the attempt is remembered under that key; call catalog.retry_pending(idempotency_key) later to pick it back up with the exact same arguments.

Catalog operations

Table/view lifecycle

• table2view(view_path, source_path, create_target_folder=True) — the one-time DROP TABLE + CREATE VIEW transition for a location that started as a physical table (straight off ingest) and is moving to being a view. Checks source_path exists first; by default also creates view_path's containing folder if missing.
• createview(source_path, target_path, overwrite=False, create_target_folder=False) — CREATE VIEW ... AS SELECT * FROM source_path, leaving source_path untouched (unlike datamove). No data of its own — a view is a saved query — so this runs over REST like every other metadata-only operation, not Arrow Flight. Same folder-append and overwrite behavior as datacopy/datamove.
• draft2version(draft_path, version_path) / publishversion(consumer_view_path, version_path) — not implemented yet (both raise NotImplementedError); promoting a draft table into a permanent version and repointing a consumer-facing view at it.
• deleteview(view_path) — DROP VIEW IF EXISTS.

Copying/moving data — always over Arrow Flight SQL, never REST, regardless of EEA_CATALOG_TRANSPORT (needs username= on Catalog(...) for the Flight auth handshake):

• datacopy(source_path, target_path, overwrite=False, create_target_folder=False) — CREATE TABLE ... AS SELECT. If target_path is an existing folder rather than a specific table/view path, the source's own name is appended to it (cp source dest/ semantics). overwrite=False (the default) raises if the (possibly folder-adjusted) target already exists; overwrite=True detects whatever is actually there — it might be a view, not a table — and drops it with the matching verb first.
• datamove(source_path, target_path, entry_type=None, overwrite=False, create_target_folder=False) — copies then drops the source. entry_type ("TABLE"/"VIEW") is auto-detected via INFORMATION_SCHEMA when not given, rather than trusting a caller to get it right. Same folder-append and overwrite behavior as datacopy. After the move, verifies target_path actually exists in the catalog before declaring success, rather than trusting a silent CREATE/DROP.

Discovery

• gettablesfrom(schema_path) — every table/view under schema_path, recursing into subfolders; returns full dot-separated paths.
• gettableitemsfrom(table_path) — one table/view's schema ({column: DATA_TYPE}) and row count, without fetching any actual rows.

Wiki & tags (Dremio's catalog collaboration API — REST-only, no SQL/Flight equivalent):

• getwikifrom(path) / setwikito(path, text, tags=None) / deletewiki(path) — read, create/overwrite, or clear the wiki text on a catalog entity. deletewiki is idempotent — a missing path, or one with no wiki at all, is a no-op, not an error (Dremio's collaboration API has no separate delete-wiki endpoint, so this clears the text to empty). tags, if given to setwikito, is a list of {"tag_name", "tag_value", "tag_title"} dicts rendered into a # Meta Data section appended to text (both a human-readable title : value line per tag and the same data as <meta><tag name=... value=... title=.../>...</meta>) — Dremio's wiki is plain markdown with no structured-metadata concept of its own, so this is embedded directly in the text.
• setmeta2wiki(path, tags=None, overwrite=True) / getmetafromwiki(path, tag_name=None, field=None) — folders only (raises CatalogOperationError on a table/view — those have Dremio's own tags/labels for this instead). setmeta2wiki updates just the # Meta Data section of the wiki already at path, keeping whatever text comes before it untouched: overwrite=True (the default) replaces the whole section with one built fresh from tags; overwrite=False merges tags into whatever tags are already there (parsed back out of the existing <meta> block), appended after them, with no deduplication. If path has no wiki yet, starts from empty base text rather than raising. getmetafromwiki reads it back: without tag_name (or if it doesn't match one there), returns every tag as a list; with a matching tag_name, returns a single {"tag_name", ...} dict instead — both tag_value and tag_title if field isn't given, or just the one field ("tag_value"/"tag_title") asks for.
• gettagsfrom(path) / settagsto(path, tags) / deletetags(path, tags) — tables/views only (the mirror image of setmeta2wiki/getmetafromwiki — raises CatalogOperationError on a folder, which has no Dremio tags/labels concept of its own). Read the full tag list; replace it wholesale; or remove just the given tags, leaving the rest untouched.

Folders (REST-only, idempotent — an already-there/already-gone folder is not an error):

• createfolder(path, create_parents=False) — create_parents=False (the default) raises if the parent is missing rather than creating a deep new path; create_parents=True creates every missing level.
• deletefolder(path, cascade=False) — cascade=False (the default) raises if the folder still has contents; cascade=True deletes every table/view and subfolder inside first, depth-first.

Retrying

• retry_pending(idempotency_key) — re-attempt whatever last stalled on an EngineStartingError, using the same arguments it was originally called with.

Lifecycle

• catalog.close() (or with Catalog(...) as catalog:) — disposes the REST/Flight session(s). Idempotent, and covered by a process-exit/SIGTERM fallback if you forget.

Notebook facade (%catalog / %ingest)

For interactive use in JupyterLab, eea_datalakehouse.notebook registers two line magics aimed at data custodians rather than application developers, thin wrappers over CatalogSession/IngestSession. %catalog runs each call immediately — no commit step to remember. %ingest still queues and needs an explicit commit(), since a catalog operation can't run before its target has actually been ingested. See docs/notebook-facade-for-data-scientists.md for the full design, and docs/notebooks/catalog_session_example.ipynb / ingest_session_example.ipynb for worked examples. Install the extra this needs once: pip install "EEADataLakehouse[notebook]".

import eea_datalakehouse.notebook  # registers %catalog/%ingest — no %load_ext needed

%catalog data_copy("draft.raw_2026", "bwd.reference.water_temperature")
%catalog set_tags("bwd.reference.water_temperature", ["reviewed"])

%ingest ingest(folder="./bw_2026", target_catalog_path="bwd.reference",
                data_format="parquet", table_name="water_temperature")
%ingest commit(retry=True)

%catalog help (or %catalog help()) renders every command as an HTML table — name, parameters, description — rather than a raw Python signature, since its audience is a data custodian, not necessarily a developer. %ingest help does the same for IngestSession's (fewer) methods — handy when you don't remember an exact parameter name mid-notebook.

A freshly created session already has a context — it starts at "catalog" (the catalog root), not empty — so a relative path works even before the first use() call. use(None)/use("") reset it back to "catalog" the same way; set_context(None) (the lower-level primitive use wraps, not normally called directly) is the one way left to clear it to no context at all.

%%catalog (the cell-magic form) sets the context once, with use(path) on its magic line, then runs every other line of the cell in order under that context, without repeating the full path on each line. use is the only way context ever changes — no other verb touches it as a side effect of running, even one whose own path fully resolves to something that could sensibly become the new context (create_folder, notably, used to leave its own path as the new context; it no longer does — every verb only reads context, never writes it). use's own path must always be a whole, absolute path — unlike every other verb, it's never resolved against whatever context already exists, and never accepts a leading ./../. It does make a live check that path actually exists in the catalog first, raising if it doesn't, rather than silently pointing context somewhere later calls would fail against anyway; use(None)/use("") skip that check and reset context to "catalog" instead. Every other verb's own path/source_path/ target_path — including data_copy/data_move/create_view's two, each resolved independently against that same context — still resolves against the current context once one is set — with or without a leading . ("2027" and ".2027" mean the same thing) — unless it already starts with catalog (this deployment's one real root source), in which case it's always taken literally as absolute rather than appended to the context, dot or not. That's what makes pairing a relative source_path with a genuinely unrelated absolute target_path in the same data_copy/data_move/create_view call unambiguous without needing a dot on the absolute side: a real absolute path here always starts with catalog. One or more leading ../ (or a bare ..) instead walks up that many levels of the context first, everywhere a relative path is understood except inside use itself — "../water_temperature" is a sibling of the context, "../../water_temperature" a level further up; "."/"./" alone (no name after either) mean the context itself. get_context() shows what the context currently is — always exactly what use/set_context last stored — and use(path) prints the same thing itself (context set to <path>), since there's nothing to commit — use never queues anything, so %catalog's usual auto-commit would otherwise have nothing to show:

%%catalog use("bwd.reference")
set_tags("water_temperature", ["reviewed"])  # bare, no dot — same as ".water_temperature"
create_folder(".2027")   # creates bwd.reference.2027 for real, but leaves context untouched

%catalog get_context()               # -> 'bwd.reference' — still what use() set, not create_folder
%catalog use("bwd.reference.2027")   # prints: context set to 'bwd.reference.2027'
%catalog get_context()               # -> 'bwd.reference.2027'
%catalog list(".")   # '.' alone -> the context itself: same as list("bwd.reference.2027")
%catalog set_tags("../water_temperature", ["archived"])  # ../ works for any verb except use()
%catalog get_tags("catalog.other_root.assessments")  # starts with 'catalog' — absolute, not appended
%catalog data_copy(".water_temperature", "catalog.other_root.archive.water_temperature_2027")
#                                          ^ starts with 'catalog' — absolute, not appended either
%catalog data_copy(".water_temperature", "archive")
#                                          ^ bare, no dot — relative too, appends to the context

get_wiki(path), get_tags(path), list(path) (every table/view under path, at any depth) and schema(path) (column types + row count) answer immediately, like get_context() — nothing to commit or undo. delete_view(path)/delete_table(path) queue like every other write, but — unlike the idempotent DROP ... IF EXISTS Catalog.deleteview/deletetable wrap — require path to already exist, and (like delete_folder) can never be undone. Every one of these raises CatalogOperationError if path doesn't exist; schema also requires it to be a table or view, not a folder. list's path may also be omitted (or "") to list the current context itself — raises CatalogSessionError if none is set yet:

%catalog get_wiki("bwd.reference.water_temperature")
%catalog get_tags("bwd.reference.water_temperature")
%catalog list("bwd.reference")      # -> ['bwd.reference.water_temperature', ...]
%catalog use("bwd.reference")
%catalog list()                     # same result — path omitted, lists the context itself
%catalog schema("bwd.reference.water_temperature")   # -> TableInfo(schema={...}, row_count=...)
%catalog delete_view("bwd.reference.old_view")

Layout

Path	Purpose
dds_ingestion/credentials.py	env-var creds + redacted DremioCreds
dds_ingestion/models.py	typed request/response models for the DDS ingest contract
dds_ingestion/client.py	thin, unit-testable HTTP client (IngestClient)
dds_ingestion/progress.py	tqdm progress bar with graceful fallback
dds_ingestion/folder.py	FolderIngest orchestration (scan/parallel/resume)
catalog/client.py	Catalog — two connections (REST + Flight), every operation as a method
catalog/operations.py	the operations themselves (table2view, datacopy, createfolder, ...), as functions taking an executor and/or a CatalogRestClient
catalog/sql.py	SqlExecutor protocol + REST/Flight implementations, resolve_executor() (transport env var)
catalog/rest.py	CatalogRestClient — Dremio's native /api/v3/catalog for folders, wiki, and tags (none of which have a SQL/Flight equivalent)
catalog/retry_state.py	persisted memory of stalled attempts, for retry_pending()
catalog/errors.py	CatalogOperationError, EngineStartingError

Development

pip install -e ".[dev]"
pytest

Releasing a new version

1. Update version in pyproject.toml (e.g. 0.2.0).

2. Commit and push to main.

3. The Release GitHub Actions workflow runs on every push to main: it reads the version from pyproject.toml, builds the package, and publishes a v0.2.0 GitHub Release (creating the matching tag automatically), with notes auto-generated from merged PRs since the previous release (categorized via .github/release.yml).

   Pushing to main without bumping the version re-publishes the release for the current version with the latest build artifacts.

Only one release/tag exists per branch at a time — main always has exactly one vX.Y.Z release, staging exactly one vX.Y.Z-staging prerelease. Each new release deletes its branch's previous release and tag first, so tags aren't permanent — pin to whatever the Install badge shows now, not to an old tag number, since it won't exist once a newer release replaces it.

A separate floating tag literally named main/staging always points at that branch's latest release — the workflow force-moves it (git tag -f, force-push) once the real release above succeeds. It deliberately shares its name with the branch: git resolves the ambiguity deterministically (a tag always wins over a same-named branch), so @main/@staging in an install command means "latest release," not "current branch tip" — expect (and ignore) a "refname is ambiguous" warning from git/pip when that happens.

The workflow also rewrites this README's pip/%pip install ...@vX.Y.Z[-staging] example lines to the version it just released, committing that change back to the branch ([skip ci], so it doesn't re-trigger itself) — so the examples above never go stale, without anyone having to remember to update them by hand.

Install in JupyterLab

Run this in a notebook cell — @main/@staging always resolves to whatever was most recently released for that branch (see Install above):

%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@main"       # latest stable
%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"    # latest early access

To pin to one specific release instead, use the exact tag the live badges under Install show (kept in sync automatically, see Releasing a new version):

%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12"

# staging's latest release (early access)
%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12-staging"

Use the %pip magic rather than !pip — it installs into the kernel the notebook is actually running on, instead of whatever pip happens to be first on PATH. After installing, restart the kernel (Kernel > Restart Kernel...) so the import below picks up the newly installed package:

from eea_datalakehouse.dds_ingestion import FolderIngest

Alternatively, download the wheel attached to the GitHub Release page for that tag and install the local file instead of pulling from git:

%pip install /path/to/EEADataLakehouse-<version>-py3-none-any.whl