Skip to content

LCORE-3496: add file processor to default config #2402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tisnik merged 1 commit into from
Aug 11, 2026
+26 −5
Merged

LCORE-3496: add file processor to default config #2402

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions src/data/default_run.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@
#
# It is intentionally thinner than the repo-root run.yaml: it carries only the
# APIs and providers needed to boot a minimal, queryable stack (inference,
# vector_io, responses, tool_runtime, files) plus the storage backends
# those providers reference. tool_runtime includes file-search (file_search /
# RAG) and model-context-protocol (MCP) so those capabilities work when
# operators enable them. Operators extend it via high-level sections or
# `native_override`.
# vector_io, responses, tool_runtime, files, file_processors) plus the storage
# backends those providers reference. tool_runtime includes file-search
# (file_search / RAG) and model-context-protocol (MCP) so those capabilities
# work when operators enable them. file_processors (inline::pypdf) is required
# for vector-store file attach / Notebook indexing after OGX removed the
# legacy PyPDF fallback. Operators extend it via high-level sections or
# `native_override` (e.g. swap pypdf for inline::docling).
#
# NOTE: `external_providers_dir` carries a default (`:=~/.llama/providers.d`)
# so the baseline resolves even when EXTERNAL_PROVIDERS_DIR is unset — see the
Expand All @@ -23,6 +25,7 @@ apis:
- responses
- conversations
- files
- file_processors
- inference
- tool_runtime
- vector_io
Expand All @@ -47,6 +50,13 @@ providers:
storage_dir: ${env.SQLITE_STORE_DIR:=~/.llama/storage/files}
provider_id: meta-reference-files
provider_type: inline::localfs
# required for /v1/vector-stores to work with files properly
file_processors:
- provider_id: pypdf
provider_type: inline::pypdf
config:
default_chunk_size_tokens: 800
default_chunk_overlap_tokens: 400
tool_runtime:
- config: {}
provider_id: model-context-protocol
Expand Down
11 changes: 11 additions & 0 deletions tests/unit/test_llama_stack_synthesize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,17 @@ def test_load_default_baseline_includes_mcp_tool_runtime() -> None:
assert "model-context-protocol" in ids


def test_load_default_baseline_includes_file_processors() -> None:
"""Default stack ships file_processors so vector-store file attach works."""
baseline = load_default_baseline()
assert "file_processors" in baseline["apis"]
processors = baseline["providers"]["file_processors"]
assert any(
p.get("provider_id") == "pypdf" and p.get("provider_type") == "inline::pypdf"
for p in processors
)
Comment on lines +126 to +134

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the configured chunk values.

The test passes when pypdf has the correct provider type but incorrect chunking values. Assert default_chunk_size_tokens == 800 and default_chunk_overlap_tokens == 400 to cover the configuration added on Lines 57-59 of src/data/default_run.yaml.

Proposed test update 
     processors = baseline["providers"]["file_processors"]
-    assert any(
-        p.get("provider_id") == "pypdf" and p.get("provider_type") == "inline::pypdf"
-        for p in processors
+    pypdf = next(
+        (p for p in processors if p.get("provider_id") == "pypdf"),
+        None,
     )
+    assert pypdf is not None
+    assert pypdf["provider_type"] == "inline::pypdf"
+    assert pypdf["config"]["default_chunk_size_tokens"] == 800
+    assert pypdf["config"]["default_chunk_overlap_tokens"] == 400
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def test_load_default_baseline_includes_file_processors() -> None:
"""Default stack ships file_processors so vector-store file attach works."""
baseline = load_default_baseline()
assert "file_processors" in baseline["apis"]
processors = baseline["providers"]["file_processors"]
assert any(
p.get("provider_id") == "pypdf" and p.get("provider_type") == "inline::pypdf"
for p in processors
)
def test_load_default_baseline_includes_file_processors() -> None:
"""Default stack ships file_processors so vector-store file attach works."""
baseline = load_default_baseline()
assert "file_processors" in baseline["apis"]
processors = baseline["providers"]["file_processors"]
pypdf = next(
(p for p in processors if p.get("provider_id") == "pypdf"),
None,
)
assert pypdf is not None
assert pypdf["provider_type"] == "inline::pypdf"
assert pypdf["config"]["default_chunk_size_tokens"] == 800
assert pypdf["config"]["default_chunk_overlap_tokens"] == 400
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_llama_stack_synthesize.py` around lines 126 - 134, Update
test_load_default_baseline_includes_file_processors to locate the pypdf provider
entry and assert its default_chunk_size_tokens is 800 and
default_chunk_overlap_tokens is 400, while preserving the existing provider_id
and provider_type checks.



# ---------------------------------------------------------------------------
# deep_merge_list_replace
# ---------------------------------------------------------------------------
Expand Down
Loading