perf: remove synchronous LiteLLM free-model sync from verified-models writes - #402
Open
juanmichelini wants to merge 3 commits into
Open
juanmichelini wants to merge 3 commits into
juanmichelini wants to merge 3 commits into
Conversation
…writes The admin PUT/POST/DELETE on /api/admin/verified-models fanned out a synchronous, per-org, sequential sweep to the LiteLLM management API inside the request cycle. On prod-scale data (~49k teams) this hung the request until the client timed out, and the in-flight sweep kept rewriting free-tier team allowlists to a stale snapshot. Make the admin write a pure, fast DB mutation: - Drop _sync_litellm_free_model_allowlists calls (and the helper) from create/update/delete_verified_model, plus _list_openhands_enabled_free_model_names. - Remove LiteLLMSyncError and the router's 502 mapping. - Remove LiteLlmManager.sync_free_model_allowlists and its now-dead helpers (_is_free_tier_team_info, _get_org_ids). Free-model allowlist propagation now lives entirely in the infra litellm-team-models Action. Tests updated accordingly. Co-authored-by: openhands <openhands@all-hands.dev>
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||
Contributor
Author
|
@openhands-agent /codereview-roasted |
Collapse two calls/defs that ruff format wants on a single line so the Lint python pre-commit hook passes. Co-authored-by: openhands <openhands@all-hands.dev>
juanmichelini
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human
Why
OHE customers don't care about is_free sync, only SaaS does. Sync will be managed from infra repo instead.
Related request: https://allhandsai.slack.com/archives/C09VBG51GEM/p1789401544248149
Summary
Fixes #399.
PUT/POST/DELETE /api/admin/verified-models/...fanned out a synchronous, per-org, sequential sweep to the LiteLLM management API from inside the request cycle whenever the payload touchedis_free/is_enabled. On prod-scale data (~49k teams) this hung the request until the client timed out, emitted no access-log line, and — because the sweep was pinned to a stale in-memory snapshot — kept rewriting free-tier team allowlists long after the request gave up, fighting other reconciliation.This PR makes the admin write a pure, fast DB mutation. Free-model allowlist propagation moves entirely off the request path into the infra
litellm-team-modelsAction (companion issue OpenHands/infra#1698).Changes
server/verified_models/verified_model_service.py_sync_litellm_free_model_allowlistscalls fromcreate_verified_model/update_verified_model/delete_verified_model._sync_litellm_free_model_allowlistsand_list_openhands_enabled_free_model_nameshelpers.LiteLLMSyncErrorexception class.server/verified_models/verified_model_router.pyLiteLLMSyncErrorimport, the_litellm_sync_error_response502 mapping, and theexcept LiteLLMSyncErrorhandlers.storage/lite_llm_manager.pysync_free_model_allowlistsand its now-dead helpers_is_free_tier_team_infoand_get_org_ids.Acceptance criteria
ruffpasses; changed modules import cleanly.Testing notes
The unit suite spins up a Postgres testcontainer (Docker), which is not available in this environment, so the full suite was not executed here. Changed modules compile/import cleanly and
ruff checkpasses on all touched files. CI will run the full suite.This PR was created by an AI agent (OpenHands) on behalf of @juanmichelini.
Enterprise server image for this PR: