You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
improve the performance of STAC based back-end #972
I ran some tests to asses the wall-time and the total time spent in the database for different backends.
The STAC one is underperforming.
Likely candidates:
Results seem come back 10 at a time. get_by_status and get_by_indices call client.search(...) without a limit, so the server's default page size applies We can consider increasing this one
Counting downloads every job. count_by_status calls get_by_status, downloads every matching item over those pages, then counts them in Python.
A full count runs every loop (job_db.count_by_status()). The f-string is built even when INFO logging is off, so each loop downloads the whole collection just to write a log line that never appears.
Every save first lists all collections. persist() calls exists(), which pages through every collection on the server.
Hi, I've opened #973 for this. It sets a page size on the searches, uses numberMatched for count_by_status (falling back to the current approach if the server doesn't report it), looks up the collection directly in exists(), and skips the histogram count when INFO logging is off. Happy to adjust anything.
Activity
omaraldulimay commentedon Nov 14, 2026
Hi, I've opened #973 for this. It sets a page size on the searches, uses numberMatched for count_by_status (falling back to the current approach if the server doesn't report it), looks up the collection directly in exists(), and skips the histogram count when INFO logging is off. Happy to adjust anything.