Skip to content

Renew weave's W&B access token on each WIF identity token refresh - #7576

Draft
evgenyrp wants to merge 1 commit into
masterfrom
wandb-wif-token-renewal
Draft

evgenyrp wants to merge 1 commit into
masterfrom
wandb-wif-token-renewal

Conversation

@evgenyrp

@evgenyrp evgenyrp commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Under W&B Workload Identity Federation, weave keeps using the W&B access token it exchanged until the token's stated expiry, but W&B stops accepting it about an hour after the exchange. It answers with 401, then 403 Project not found. The runtime already rewrites the identity token file every 30 min, but weave doesn't exchange it again while its cached token looks valid.

This hits any agent process that runs longer than about an hour:

  • In a 9h dataset-eval run, the parent's final Weave writes were dropped, so the evaluation never closed and showed no metrics.
  • Example 6's agent trace lost its last ~45 min.
  • bug-fix hit the same 401s.

After each identity-token refresh, wandb_wif now drops weave's cached access token, in memory and in its credentials file. The next request then exchanges the fresh identity token. This relies on private weave internals, so it is a stopgap until W&B fixes the token lifetime. If those internals change, it logs an error and the token-file refresh keeps working.

To verify: a deployed run longer than 2h shows no 401/403 errors in Sentry and a finished evaluation in Weave.

fixes #7523

@suhaibmujahid

Copy link
Copy Markdown
Member

We have related WIP (#7247), whatever solution works here, we should apply it there.

@evgenyrp

Copy link
Copy Markdown
Contributor Author

We have related WIP (#7247), whatever solution works here, we should apply it there.

I'm asking W&B support to verify this. If it unblocks the evals, we should probably just land this.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals agent didn't log the metrics to Weave

2 participants