Repository navigation
Protect query execution and support recoverable 10M-row exports - #1
Merged
Merged
Conversation
RAHoebe
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Concurrent exports could evict active data, collide in DuckDB spill files, or leave incomplete cache entries after a crash. This release adds leases, bounded admission, quota accounting, isolated execution and durable cache recovery, while retaining the existing HTTP/CSV contracts and normal limits. An optional deployment profile supports 10,000,000 rows, 2 GiB CSV results and 900-second queries with four query slots in the tested 2 GiB container.
The worker remains read-only and OMERO-agnostic. Additive capabilities provide exact result provenance, versioned cache identity, authenticated metrics, correlation IDs, overlapping keyring credentials and an optional private mTLS profile. Analysis owns authorization, writes and interrupted-promotion reconciliation in the paired PR.
Validation: 58 worker tests on Windows and Linux; lint, format and strict typing; container smoke/security and CI pass. The actual Analysis HTTP version matrix passes all 17 cases. Real OMERO query/download/save and browser loading pass at 10M rows for DuckDB, SQLite and CSV. Exact 10M/2-GiB boundaries and overflow rejection pass. Eighteen real engine-OOM/restart cases, six VM cache-publication power cuts, ten promotion power cuts and interrupted cleanup/reconciliation pass. The completed three-repeat 10M matrix covers all 540 requests: 468 successes, 72 expected admission rejections and zero unexpected failures. All 72 single-client 1M/4M regressions pass. Normal runs have no kernel OOM kills and remain within cache quotas; additional first-journal-directory and durable cache-entry power cuts pass.
Deployment: retain both images and configurations; upgrade worker first, verify current Analysis, then deploy the paired Analysis and opt into larger limits or mTLS. Preserve cache, OMERO annotations and persistent Analysis state/audit/signing key on rollback. See the recovery runbook for commands, observed OMERO search-index recovery and target-environment acceptance requirements. Local test success does not authorize a production rollout on untested hardware.
Completed release evidence records timings, memory, cache usage and hashes of raw evidence.