Skip to content

improve the performance of STAC based back-end #972

Open

Description

@HansVRP
image

I ran some tests to asses the wall-time and the total time spent in the database for different backends.

The STAC one is underperforming.

Likely candidates:

  • Results seem come back 10 at a time. get_by_status and get_by_indices call client.search(...) without a limit, so the server's default page size applies We can consider increasing this one

  • Counting downloads every job. count_by_status calls get_by_status, downloads every matching item over those pages, then counts them in Python.

  • A full count runs every loop (job_db.count_by_status()). The f-string is built even when INFO logging is off, so each loop downloads the whole collection just to write a log line that never appears.

  • Every save first lists all collections. persist() calls exists(), which pages through every collection on the server.

.

Activity

  1. omaraldulimay commented on Nov 14, 2026

    @omaraldulimay

    Hi, I've opened #973 for this. It sets a page size on the searches, uses numberMatched for count_by_status (falling back to the current approach if the server doesn't report it), looks up the collection directly in exists(), and skips the histogram count when INFO logging is off. Happy to adjust anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions