Support file-based OAuth2 bearer tokens for the REST catalog (token-path) - #17389
Open
haydn-j-evans wants to merge 2 commits into
Open
Support file-based OAuth2 bearer tokens for the REST catalog (token-path)#17389haydn-j-evans wants to merge 2 commits into
haydn-j-evans wants to merge 2 commits into
Conversation
haydn-j-evans
force-pushed
the
support-token-files
branch
from
July 27, 2026 21:24
a026093 to
cb0b9ce
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A common request is for Spark and Trino pods running on Kubernetes to commonly authenticate to REST catalogs (Lakekeeper, Polaris, etc.) using the pod's own service account token, mounted from a projected/auto-mounted file (typically /var/run/secrets/kubernetes.io/serviceaccount/token).
Kubernetes rotates this token in place before it expires — but Iceberg's REST catalog OAuth2 config only supports a static token string set once at catalog creation. Once that token expires, there's no way to pick up the rotated value without restarting the process, which isn't practical for long-lived Spark/Trino/Kafka Connect deployments.
This was requested in #13550 (Kafka Connect + Lakekeeper, mounting a K8s service-account token), which was auto-closed as stale without a fix — the underlying need (read a token from a file, with rotation) still applies today across any REST-catalog engine.
What this adds
A new token-path property, usable anywhere token is today, that points at a file instead of an inline string. The session automatically re-reads the file and swaps in the new token ahead of expiry — parsed from the token's JWT exp claim where available — by default 5 minutes before expiry, configurable via token-path-refresh-buffer-ms. On a transient read failure (e.g. racing Kubernetes' atomic file rotation), it retries on a short backoff rather than giving up.
It reuses the existing OAuth2Util.AuthSession scheduled-refresh infrastructure (same shared thread pool, same cooperative-stop-on-close semantics as the existing HTTP-based token refresh), just swapping the refresh mechanism from "call the OAuth2 token endpoint" to "re-read the file" — no network calls are needed for the file-based refresh path.
Since Spark, Trino, Kafka Connect, and other engines pass REST catalog properties straight through to RESTSessionCatalog as an unfiltered map, this requires no changes outside core — token-path is usable immediately via spark.sql.catalog..token-path=..., iceberg.catalog.token-path=... (Kafka Connect), or the equivalent connector property.