LCORE-3520 Close two synthesis-mode gaps: LLM model registration and CLI access

[omkarjoshi0304]

Summary

While adopting unified synthesis mode (LCORE-2336) in the OpenStack Lightspeed operator, we ran into two gaps:

1. LLM models aren't registered in registered_resources.models. apply_high_level_inference() builds providers.inference entries from inference.providers, but never registers those models as resources. Without that registration, Llama Stack can only discover a model via list_models() at startup, which needs a live connection to the provider — so inference breaks if the endpoint is briefly unreachable during startup. We were working around this with a post-processing step in our own init container script.

2. The CLI only supports legacy enrichment mode. python llama_stack_configuration.py -c config.yaml calls generate_configuration(), which expects an already-built run.yaml. There's no CLI path to synthesize_to_file()/synthesize_configuration() (unified mode), so we had to import the module directly instead of using the documented script interface.

Changes

• apply_high_level_inference() now registers each provider's allowed_models as an LLM resource in registered_resources.models, deduped against anything already registered.
• Added a --synthesize flag to the CLI: when set, builds a complete config via synthesize_to_file() from -c alone (ignoring -i), instead of enriching an existing run.yaml.

Both changes are additive — default CLI behavior for existing legacy-mode consumers is unchanged.

Once merged, we'll be able to call the plain CLI directly (python -m llama_stack_configuration -c lightspeed-stack.yaml -o run.yaml --synthesize) from our init container and drop our custom wrapper script entirely.

Test plan

• uv run python -m pytest tests/unit/test_llama_stack_synthesize.py tests/unit/test_llama_stack_configuration.py -q — 112 passed
• uv run ruff check — clean
• uv run mypy src/llama_stack_configuration.py — clean
• uv run black --check --fast — clean
• Rebased cleanly onto latest upstream/main

Summary by CodeRabbit

• New Features

  • Added unified configuration synthesis through the --synthesize command-line option.
  • Automatically registers configured provider models for high-level inference.
  • Supports multiple allowed models while preventing duplicate registrations.
  • Preserves existing behavior when synthesis is not enabled.

• Bug Fixes

  • Improved handling for providers without specified models.
  • Prevented outdated provider model entries from remaining after replacement.
  • Updated command-line descriptions and legacy input guidance.

[coderabbitai]

Walkthrough

High-level inference synthesis now registers provider allowed_models as deduplicated LLM resources and removes models from replaced providers. The CLI adds --synthesize for configuration-only synthesis while preserving the legacy default flow. Tests cover both behaviors.

Changes

Inference synthesis

Layer / File(s)	Summary
Register allowed inference models src/llama_stack_configuration.py, tests/unit/test_llama_stack_synthesize.py	Provider helpers build and replace inference entries. Synthesis registers missing allowed_models as LLM resources, prevents duplicates, evicts superseded models, and preserves baseline models.
Add unified CLI synthesis mode src/llama_stack_configuration.py, tests/unit/test_llama_stack_synthesize.py	The --synthesize option builds a complete run configuration from the main configuration file. The default invocation retains legacy input enrichment. CLI tests cover both paths.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant MainConfig
  participant apply_high_level_inference
  participant ModelRegistry
  CLI->>MainConfig: read configuration
  CLI->>apply_high_level_inference: synthesize with --synthesize
  apply_high_level_inference->>ModelRegistry: register allowed_models
  ModelRegistry-->>CLI: return synthesized run configuration

Loading

Suggested reviewers: jdubrick, max-svistunov, asimurka

🚥 Pre-merge checks | ✅ 7

✅ Passed checks (7 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Performance And Algorithmic Complexity	✅ Passed	The new registrar uses a set for model deduplication and linear list work only for duplicate-provider eviction; no API/DB calls, pagination gaps, or unbounded state were introduced.
Security And Secret Handling	✅ Passed	Changed files add no endpoints, K8s Secret resources, command execution, or plaintext secrets; auth uses ${env.*} references, model IDs are not logged, and synthesized output is mode 0600.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely identifies both main changes: LLM model registration and CLI access for synthesis mode.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

✨ Simplify code

• Create PR with simplified code

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/llama_stack_configuration.py`:
- Around line 1053-1064: Update the provider registration flow surrounding the
allowed_models loop so that when a later high-level provider entry replaces an
existing emitted provider_id, model resources registered by the earlier
declaration are removed before registering the replacement’s allowed_models.
Track registrations made by this function, preserve registrations for other
providers, and add a regression test covering duplicate provider_id entries with
different allowed_models.
- Around line 1012-1017: Stop mutating the input ls_config in the
configuration-building flow around registered_models and existing_model_ids;
create a new configuration with copied registered_resources/models data, apply
all updates including the logic around lines 1056-1064 to that new structure,
and return it. Update the caller to use the returned configuration instead of
relying on in-place changes.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e2c3b03f-cc53-49c9-adb5-82bcaadedf37

📥 Commits

Reviewing files that changed from the base of the PR and between fa5d895 and 141f12c.

📒 Files selected for processing (2)

• src/llama_stack_configuration.py
• tests/unit/test_llama_stack_synthesize.py

📜 Review details

⏰ Context from checks skipped due to timeout. (9)

• GitHub Check: E2E: server mode / ci / group 1
• GitHub Check: E2E: server mode / ci / group 2
• GitHub Check: E2E: server mode / ci / group 3
• GitHub Check: E2E: library mode / ci / group 3
• GitHub Check: E2E: library mode / ci / group 2
• GitHub Check: E2E: library mode / ci / group 1
• GitHub Check: Pylinter
• GitHub Check: bandit
• GitHub Check: integration_tests (3.13)

🧰 Additional context used

📓 Path-based instructions (3)

**/*

📄 CodeRabbit inference engine (Custom checks)

**/*: Flag meaningful O(n^2)+ algorithms on non-trivial inputs, including handlers and Kubernetes list operations.
Flag N+1 patterns that list items and then query once per item, including Kubernetes API and database access.
Flag expensive work inside loops, including API calls, JSON parsing, and regex compilation.
Flag unbounded growth in caches, watchers, or buffers when eviction or limits are missing.
Flag missing pagination or limits on list operations and API endpoints.
Flag secrets or tokens logged in plaintext or hardcoded in source.
Flag API endpoints missing authentication or authorization.
Flag injection vulnerabilities, including SQL injection, command injection, and path traversal.
Flag sensitive data leaked in API responses, WebSocket messages, or logs.
Flag Kubernetes Secrets and Red Hat secrets missing OwnerReferences.

Files:

• tests/unit/test_llama_stack_synthesize.py
• src/llama_stack_configuration.py

tests/unit/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Use pytest for unit tests, shared fixtures in conftest.py, pytest-mock for mocks, pytest.mark.asyncio for async tests, and maintain at least 60% unit-test coverage.

Files:

• tests/unit/test_llama_stack_synthesize.py

src/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

src/**/*.py: Use absolute imports for internal modules and follow the prescribed FastAPI and Llama Stack import conventions.
All modules must begin with descriptive docstrings; use logger = get_logger(__name__) from log.py for module logging; package __init__.py files must contain brief package descriptions.
Define shared constants in the central constants.py module, add descriptive comments, and annotate constants with Final[type].
Use complete type annotations for function parameters, return types, class attributes, and type aliases; prefer specific types over Any, use modern union syntax, and use typing_extensions.Self for model validators.
All functions and classes require descriptive Google-style docstrings, including appropriate Parameters, Returns, Raises, and Attributes sections.
Use descriptive snake_case, action-oriented function names such as get_, validate_, and check_; use PascalCase class names with standard suffixes such as Configuration, Error/Exception, Resolver, and Interface.
Avoid modifying input parameters in place; return a newly constructed data structure instead.
Use async def for I/O operations and external API calls; API endpoints should raise FastAPI HTTPException with appropriate status codes and handle Llama Stack APIConnectionError.
Use from log import get_logger and standard logger levels: debug for diagnostics, info for general execution, warning for unexpected conditions or potential problems, and error for serious failures.
Configuration models must extend ConfigurationBase, set extra="forbid" to reject unknown fields, use Pydantic validators for custom validation, and use types such as Optional[FilePath], PositiveInt, and SecretStr where appropriate.
Abstract interfaces must use ABC and @abstractmethod decorators.
Never commit secrets or keys; use environment variables for sensitive data.

Files:

• src/llama_stack_configuration.py

🧠 Learnings (3)

📚 Learning: 2026-06-24T13:45:37.249Z

Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 1971
File: src/utils/markdown_repair.py:31-36
Timestamp: 2026-06-24T13:45:37.249Z
Learning: In the lightspeed-stack repository, docstrings must use the section header name "Parameters:" (not "Args:") for function arguments, even if the project references Google Python docstring conventions. Ensure docstrings follow the project’s established "Parameters:" header format for any documented function parameters.

Applied to files:

• tests/unit/test_llama_stack_synthesize.py
• src/llama_stack_configuration.py

📚 Learning: 2026-07-06T15:26:18.398Z

Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 2071
File: src/models/config.py:2416-2422
Timestamp: 2026-07-06T15:26:18.398Z
Learning: In this repo’s Python code under src/**, don’t treat differences in string concatenation style as a style inconsistency when Black has effectively forced (or made clearer) use of explicit `+` string concatenation in multi-line logger/string expressions. If adjacent-literal implicit concatenation is avoided/changed specifically to accommodate Black’s formatting in these call sites, accept the `+` usage and don’t recommend converting it solely for consistency with nearby blocks that use implicit concatenation.

Applied to files:

• src/llama_stack_configuration.py

📚 Learning: 2026-07-17T19:25:05.325Z

Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 2166
File: src/utils/saved_prompts.py:129-157
Timestamp: 2026-07-17T19:25:05.325Z
Learning: For any endpoint that handles saved prompts and calls `src/utils/saved_prompts.py::create_saved_prompt`, treat the endpoint as the validation boundary. Before calling `create_saved_prompt`, validate the incoming saved-prompt name and content, specifically using `validate_saved_prompt_name` and then persist (store) the normalized value it returns. Do not call `create_saved_prompt` with unvalidated/raw name/content.

Applied to files:

• src/llama_stack_configuration.py

🪛 ast-grep (0.45.0)

src/llama_stack_configuration.py

[warning] 1406-1406: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(args.config, "r", encoding="utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(open-filename-from-request)

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/llama_stack_configuration.py`:
- Around line 1490-1494: Update the help text for the --synthesize argument in
the argument parser to remove the internal ticket reference LCORE-2336 while
preserving the description of unified synthesis mode and its run.yaml behavior.
- Around line 1497-1505: Update the configuration-loading flow before the
args.synthesize branch so yaml.safe_load returns an empty mapping when the file
is empty or contains only comments. Ensure synthesize_to_file and
generate_configuration receive a mapping rather than None, while preserving the
existing parsed configuration for non-empty files.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9392f637-35b4-41cc-ab85-4398ccb36e8e

📥 Commits

Reviewing files that changed from the base of the PR and between 141f12c and 1a8b3b6.

📒 Files selected for processing (2)

• src/llama_stack_configuration.py
• tests/unit/test_llama_stack_synthesize.py

📜 Review details

⏰ Context from checks skipped due to timeout. (5)

• GitHub Check: unit_tests (3.13)
• GitHub Check: Pyright
• GitHub Check: E2E: library mode / ci / group 2
• GitHub Check: E2E: server mode / ci / group 1
• GitHub Check: E2E Tests for Lightspeed Evaluation job

⚠️ CI failures not shown inline (2)

GitHub Actions: PR Title Checker / 0_check.txt: Close two synthesis-mode gaps: LLM model registration and CLI access

Conclusion: failure

View job details

##[group]Run thehanimo/pr-title-checker@v1.4.3
 with:
   GITHUB_***REDACTED_SECRET_ASSIGNMENT***
   pass_on_octokit_error: false
   configuration_path: .github/pr-title-checker-config.json
 ##[endgroup]
 (node:2128) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
 (Use `node --trace-deprecation ...` to show where the warning was created)
 Using config file .github/pr-title-checker-config.json from repo lightspeed-core/lightspeed-stack [ref: 94fb012130d13feb970c4e890a49563c46cd4d14]
 (node:2128) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
 Creating label (title needs formatting)...
 Label (title needs formatting) already created.
 Adding label (title needs formatting) to PR...
 HttpError: Resource not accessible by integration
 ##[error]Failed to add label (title needs formatting) to PR

---

GitHub Actions: PR Title Checker / check: Close two synthesis-mode gaps: LLM model registration and CLI access

Conclusion: failure

View job details

##[group]Run thehanimo/pr-title-checker@v1.4.3
 with:
   GITHUB_***REDACTED_SECRET_ASSIGNMENT***
   pass_on_octokit_error: false
   configuration_path: .github/pr-title-checker-config.json
 ##[endgroup]
 (node:2128) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
 (Use `node --trace-deprecation ...` to show where the warning was created)
 Using config file .github/pr-title-checker-config.json from repo lightspeed-core/lightspeed-stack [ref: 94fb012130d13feb970c4e890a49563c46cd4d14]
 (node:2128) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
 Creating label (title needs formatting)...
 Label (title needs formatting) already created.
 Adding label (title needs formatting) to PR...
 HttpError: Resource not accessible by integration
 ##[error]Failed to add label (title needs formatting) to PR

🧰 Additional context used

📓 Path-based instructions (3)

**/*

📄 CodeRabbit inference engine (Custom checks)

**/*: Flag meaningful O(n^2)+ algorithms on non-trivial inputs, including handlers and Kubernetes list operations.
Flag N+1 patterns that list items and then query once per item, including Kubernetes API and database access.
Flag expensive work inside loops, including API calls, JSON parsing, and regex compilation.
Flag unbounded growth in caches, watchers, or buffers when eviction or limits are missing.
Flag missing pagination or limits on list operations and API endpoints.
Flag secrets or tokens logged in plaintext or hardcoded in source.
Flag API endpoints missing authentication or authorization.
Flag injection vulnerabilities, including SQL injection, command injection, and path traversal.
Flag sensitive data leaked in API responses, WebSocket messages, or logs.
Flag Kubernetes Secrets and Red Hat secrets missing OwnerReferences.

Files:

• tests/unit/test_llama_stack_synthesize.py
• src/llama_stack_configuration.py

tests/unit/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Use pytest for unit tests, shared fixtures in conftest.py, pytest-mock for mocks, pytest.mark.asyncio for async tests, and maintain at least 60% unit-test coverage.

Files:

• tests/unit/test_llama_stack_synthesize.py

src/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

src/**/*.py: Use absolute imports for internal modules and follow the prescribed FastAPI and Llama Stack import conventions.
All modules must begin with descriptive docstrings; use logger = get_logger(__name__) from log.py for module logging; package __init__.py files must contain brief package descriptions.
Define shared constants in the central constants.py module, add descriptive comments, and annotate constants with Final[type].
Use complete type annotations for function parameters, return types, class attributes, and type aliases; prefer specific types over Any, use modern union syntax, and use typing_extensions.Self for model validators.
All functions and classes require descriptive Google-style docstrings, including appropriate Parameters, Returns, Raises, and Attributes sections.
Use descriptive snake_case, action-oriented function names such as get_, validate_, and check_; use PascalCase class names with standard suffixes such as Configuration, Error/Exception, Resolver, and Interface.
Avoid modifying input parameters in place; return a newly constructed data structure instead.
Use async def for I/O operations and external API calls; API endpoints should raise FastAPI HTTPException with appropriate status codes and handle Llama Stack APIConnectionError.
Use from log import get_logger and standard logger levels: debug for diagnostics, info for general execution, warning for unexpected conditions or potential problems, and error for serious failures.
Configuration models must extend ConfigurationBase, set extra="forbid" to reject unknown fields, use Pydantic validators for custom validation, and use types such as Optional[FilePath], PositiveInt, and SecretStr where appropriate.
Abstract interfaces must use ABC and @abstractmethod decorators.
Never commit secrets or keys; use environment variables for sensitive data.

Files:

• src/llama_stack_configuration.py

🧠 Learnings (3)

📚 Learning: 2026-06-24T13:45:37.249Z

Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 1971
File: src/utils/markdown_repair.py:31-36
Timestamp: 2026-06-24T13:45:37.249Z
Learning: In the lightspeed-stack repository, docstrings must use the section header name "Parameters:" (not "Args:") for function arguments, even if the project references Google Python docstring conventions. Ensure docstrings follow the project’s established "Parameters:" header format for any documented function parameters.

Applied to files:

• tests/unit/test_llama_stack_synthesize.py
• src/llama_stack_configuration.py

📚 Learning: 2026-07-06T15:26:18.398Z

Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 2071
File: src/models/config.py:2416-2422
Timestamp: 2026-07-06T15:26:18.398Z
Learning: In this repo’s Python code under src/**, don’t treat differences in string concatenation style as a style inconsistency when Black has effectively forced (or made clearer) use of explicit `+` string concatenation in multi-line logger/string expressions. If adjacent-literal implicit concatenation is avoided/changed specifically to accommodate Black’s formatting in these call sites, accept the `+` usage and don’t recommend converting it solely for consistency with nearby blocks that use implicit concatenation.

Applied to files:

• src/llama_stack_configuration.py

📚 Learning: 2026-07-17T19:25:05.325Z

Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 2166
File: src/utils/saved_prompts.py:129-157
Timestamp: 2026-07-17T19:25:05.325Z
Learning: For any endpoint that handles saved prompts and calls `src/utils/saved_prompts.py::create_saved_prompt`, treat the endpoint as the validation boundary. Before calling `create_saved_prompt`, validate the incoming saved-prompt name and content, specifically using `validate_saved_prompt_name` and then persist (store) the normalized value it returns. Do not call `create_saved_prompt` with unvalidated/raw name/content.

Applied to files:

• src/llama_stack_configuration.py

🪛 ast-grep (0.45.0)

src/llama_stack_configuration.py

[warning] 1496-1496: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(args.config, "r", encoding="utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(open-filename-from-request)

🔇 Additional comments (3)

src/llama_stack_configuration.py (2)

253-254: LGTM!

Also applies to: 284-290, 341-350, 551-579, 666-673

---

985-1103: LGTM!

Also applies to: 1120-1127, 1143-1154

tests/unit/test_llama_stack_synthesize.py (1)

9-13: LGTM!

Also applies to: 26-26, 419-543, 789-789, 922-991