Skip to content

feat(router): add lmdeploy router - #4981

Open
caikun-pjlab wants to merge 30 commits into
InternLM:mainfrom
caikun-pjlab:feat/add_lmdeploy_router
Open

caikun-pjlab wants to merge 30 commits into
InternLM:mainfrom
caikun-pjlab:feat/add_lmdeploy_router

Conversation

@caikun-pjlab

@caikun-pjlab caikun-pjlab commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR integrates the Rust-based lmdeploy-router into the main LMDeploy repository, providing unified development, testing, packaging, documentation, and release management.

Key Changes

  • Migrated the complete Rust router implementation to lmdeploy/router, including source code, benchmark suite, Python launcher, wheel packaging configuration, and tests.
  • Replaced the legacy Python proxy with lmdeploy-router; lmdeploy serve proxy now remains as a compatibility entry point that launches the Rust router.
  • Kept lmdeploy-router as an independent Python distribution while colocating its source in the LMDeploy repository. The wheel provides the Python launcher, lmdeploy_router_rs extension, and standalone router binary.
  • Added unified router CI for:
    • cargo fmt
    • cargo clippy
    • Rust tests
    • Python unit tests
    • mock-based Python integration tests
  • Expanded router integration coverage for REST APIs, worker registration and removal, load-balancing policies, authentication, retries, circuit breaking, rate limiting, payload handling, and prefill-decode routing.
  • Added real-engine e2e tests for regular serving, and PD disaggregation. PD disaggregation ci test is skipped now, because machine does not support RDMA.
  • Updated the PR function-test workflow so existing lmdeploy serve api_server tests run through lmdeploy-router, ensuring REST traffic is exercised through the router in real CI environments.
  • Added a PD disaggregation e2e stage for Qwen/Qwen3.5-35B-A3B, deploying separate Prefill and Decode instances and routing requests through lmdeploy-router.
  • Integrated router construction into the main LMDeploy Docker image by default, with BUILD_ROUTER=0 available to skip Rust compilation when router support is not needed.
  • Updated English and Chinese documentation with installation, routing-policy, dynamic registration, PD deployment, migration, and compatibility guidance.
  • Added a manual router release workflow that builds wheel artifacts by default and only uploads to PyPI when explicitly enabled.

Compatibility

  • The lmdeploy serve proxy command is preserved for compatibility.
  • Legacy min_expected_latency and min_observed_latency routing values are accepted and mapped to cache_aware.
  • The old Python proxy implementation and its internal import paths are removed; users should use lmdeploy-router or the compatibility CLI entry point.
  • The router wheel remains independently versioned and is decoupled from the main LMDeploy wheel.

Testing

  • Rust formatting, static analysis, and tests run in router CI.
  • Python unit and mock integration tests run in router CI.
  • Real-model PR function tests now exercise requests through lmdeploy-router.
  • Docker image builds support both router-enabled and router-disabled modes.
  • Documentation formatting, Python linting, YAML parsing, and workflow shell syntax have been validated.

@caikun-pjlab
caikun-pjlab marked this pull request as draft September 17, 2026 02:56
@caikun-pjlab caikun-pjlab changed the title Feat/add lmdeploy router feat(router): add lmdeploy router Sep 17, 2026
@caikun-pjlab
caikun-pjlab marked this pull request as ready for review September 20, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant