feat: split Studio search into separate course and library indexes

[blarghmatey]

Description

Splits the Studio Meilisearch index in two: course blocks stay in studio_content, and Libraries V2 content (blocks, containers, collections) moves to a new studio_library_content index.

Library authoring waits synchronously on Meilisearch, and indexing cost grows with index size. On large instances studio_content is almost entirely course blocks (one instance: 1,319,937 documents, 99.93% course blocks), so every library write pays for content it never touches. That is the root of #38993, where the library commit endpoint times out publishing a few blocks because each synchronous library index update takes seconds on a 500K+ document index. Course indexing is async and has no reason to share an index with libraries. This implements @bradenmacdonald's proposal in #38993 (comment) and replaces #39041 (disable course indexing) as the quick fix, without losing Studio course search.

• Writes are routed by document type or context key. Rebuild locks and _new temp indexes are per index, and post_migrate reconciliation creates and configures both.
• rebuild_index rebuilds and swaps in the library index before starting courses. Course indexing is otherwise unchanged.
• GET /api/content_search/v2/studio/ returns course_index_name and library_index_name. The tenant token's search rules cover both indexes with the existing access filter. index_name stays for one release and points at the course index.
• New reindex_studio --libraries-only, backed by a new rebuild_library_index task.
• Decision record: openedx/core/djangoapps/content/search/docs/decisions/0002-separate-course-and-library-indexes.rst.

Roles affected: Operator (upgrade steps below), Course Author (faster library authoring on large instances; no UI change).

Supporting information

• Issue: [Ulmo] Library v2 commit endpoint timeout with large Meilisearch indexes #38993
• Proposal: [Ulmo] Library v2 commit endpoint timeout with large Meilisearch indexes #38993 (comment)
• Supersedes: feat: add MEILISEARCH_COURSE_INDEXING to scope Studio course indexing #39041
• Frontend: feat: query separate course and library search indexes frontend-app-authoring#3248

Testing instructions

Upgrading an instance that already has a populated studio_content:

1. ./manage.py cms migrate. Reconciliation creates the empty studio_library_content index.
2. ./manage.py cms reindex_studio --libraries-only. This rebuilds the library index, then deletes type != "course_block" documents from studio_content. Courses are not reindexed.
3. With the paired frontend, open a library: components, collections and units list and search. Open the Studio search modal from a course: course content still searches.
4. Create a library component and confirm it appears in studio_library_content and not in studio_content.

Automated: pytest -c cms/pytest.ini openedx/core/djangoapps/content/search passes, 128 tests. New tests assert which index each write, rebuild, swap and cleanup targets, and that the token rules cover both indexes.

The unit tests mock the Meilisearch client, so I also replayed the calls this change relies on against a real Meilisearch v1.36.0, using the index settings from index_config.py: temp index then swap for the library index, the type != "course_block" delete-by-filter (leaves only course blocks, and is a no-op on rerun), and a tenant token scoped to both indexes (both searchable with the access filter applied, multi-search across both works, any other index returns 403). All passed. Not yet run end to end on a full Studio deployment.

Deadline

Wanted in Verawood per the discussion on #39041. Backport: #39122.

Other information

• Deploy together with the frontend change. A frontend that only reads index_name searches the course index, so it stops finding library content once step 2 has run.
• STUDIO_INDEX_NAME is replaced by STUDIO_COURSE_INDEX_NAME and STUDIO_LIBRARY_INDEX_NAME, and reset_index() now takes an index name. reconcile_index() is renamed reconcile_indexes(), and reconcile_index(index_name) now reconciles a single index. Nothing else in this repo uses them; an external plugin importing them would need updating.
• The Meilisearch API key Studio uses must be allowed to manage studio_library_content and studio_library_content_new (keys scoped by MEILISEARCH_INDEX_PREFIX already are).
• One data migration, search.0003, deletes library rows from IncrementalIndexCompleted so an interrupted pre-upgrade incremental run can't skip libraries. No schema changes.

[openedx-webhooks]

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[Copilot]

🟡 Changes recommended

Legacy incremental checkpoints can omit libraries, and partial rebuild failures can still trigger destructive cleanup.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Splits Studio search into dedicated course and Libraries V2 Meilisearch indexes to reduce synchronous library-indexing latency.

Changes:

• Routes course and library documents to separate indexes.
• Adds per-index rebuilding, reconciliation, cleanup, and token rules.
• Adds a libraries-only upgrade command, tests, and decision record.

File summaries

File	Description
api.py	Implements index routing, rebuilding, cleanup, and API response changes.
tasks.py	Adds the library-only rebuild task.
management/commands/reindex_studio.py	Adds --libraries-only.
docs/decisions/0002-separate-course-and-library-indexes.rst	Documents the architecture and upgrade process.
tests/test_api.py	Tests routing, rebuilding, cleanup, and index selection.
tests/test_handlers.py	Verifies event writes target the correct index.
tests/test_reconcile.py	Tests reconciliation of both indexes.
tests/test_reindex_cmd.py	Tests the new command and task.
tests/test_views.py	Tests response fields and dual-index tenant rules.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[bradenmacdonald]

Do you think any changes are needed to tutor? It seems not but I wanted to ask.

[bradenmacdonald]

Nice work! I think the things copilot pointed out are worth addressing but otherwise this worked really well in my testing.

[blarghmatey]

@bradenmacdonald No Tutor changes needed. cms migrate creates and configures the empty library index through post_migrate reconciliation. Tutor's init job then runs reindex_studio, which fills the library index, removes library documents from the course index, and reindexes courses. 019008f also makes sure libraries from an interrupted pre-upgrade run aren't skipped. The prefix-scoped API key already covers the new index name.

[bradenmacdonald]

@ormsbee Do you want to look at this or do you want anyone else to look at it before I merge it?