Enforce sequence coordinate bounds - #663
Open
theferrit32 wants to merge 10 commits into
Open
theferrit32 wants to merge 10 commits into
theferrit32 wants to merge 10 commits into
Conversation
models is a module, so models[var["type"]] raised TypeError for every input and the "vrs" format was unusable. Resolve the class via VrsType, returning None for unknown types as intended.
SeqRepo silently truncates out-of-range fetches, so translators minted permanent VRS identifiers for locations that do not exist on their sequence, or reported them as a misleading reference mismatch against ''. Zero-width insertions past the end got no diagnostic at all, and the CNV and vrs paths never fetch sequence. Add _DataProxy.validate_location_bounds, which raises DataProxyValidationError (unconditionally; there is no require_validation escape hatch) when any defined start/end value lies outside [0, len]. start and end are checked independently so circular start > end remains valid, pos == len is a valid insertion point, and undefined Range endpoints are skipped. Call it from _create_allele (hgvs, spdi, beacon, gnomad), from _from_gnomad before validate_ref_seq, from CnvTranslator._from_hgvs, and from _from_vrs. The original input accession is threaded through as sequence_id and namespace-coerced with the new coerce_accession_namespace (shared with derive_refget_accession) so the length lookup is a metadata cache hit, adding no requests.
Second layer for routes the translator checks do not cover: normalize() called directly on a hand-built Allele, the annotator, and denormalize/renormalize round trips. Uses the length SequenceProxy already fetched, so it adds no I/O, and runs before the definite-range early return so those locations are checked too. Fails before bioutils can compute on truncated sequence.
Parameterized cases for validate_location_bounds (stub proxy), the normalize guard (local test SeqRepo), and every translator input path: hgvs g./n./c./p. (including p.Ter<len+1>del), spdi (including a zero-width insertion past the end), gnomad (bounds message rather than reference mismatch, with and without require_validation), beacon, vrs, and CNV hgvs. In-bounds edge cases (terminal residue, insertion at end, circular start > end, indefinite ranges) are accepted. Each translator test uses a fresh REST dataproxy so its cassette is self-contained regardless of test order.
- Import Range at runtime instead of under TYPE_CHECKING; there is no import cycle. Use isinstance(pos, Range) rather than duck-typing .root. - Fix validate_location_bounds docstring: every defined Range member is checked, not just the largest. - Drop the warning log before raising; the error is always raised. - Make the accession coercion helper private. - Remove the passthrough _Translator._validate_location_bounds wrapper and call the dataproxy method directly. - Trim tests that duplicated coverage across the helper, normalize, and translator levels, along with their cassettes.
Call bioutils coerce_namespace directly, which already leaves namespaced identifiers unchanged, and restore derive_refget_accession to its original form. Fold _defined_values into _check_location_bounds, its only caller.
Look up the sequence length by the refget accession already in the allele values instead of threading the input accession through every translator. This is the same metadata lookup normalize makes next, so it adds no request on the default path. Out-of-bounds errors from _create_allele now name the ga4gh:SQ id. test_from_beacon (do_normalize=False) gains the ga4gh metadata request in its cassette.
- Leave _from_vrs input as-is: like the rest of that path, it is not validated or normalized. Test the VrsType lookup fix directly, including the unknown and missing type cases. - Validate CNV bounds against the refget accession, matching the allele paths. - Call validate_location_bounds from normalize instead of importing the private helper, and fold the helper into the method. - Drop redundant bounds test rows and their cassettes; add type hints.
Out-of-bounds errors now name the sequence as given (e.g. GRCh38:1) followed by the refget accession it resolved to, taken from the aliases in the same metadata lookup. Pass the input accession from every translator path again so errors are consistent across formats, and so the length lookup reuses the cached input-id metadata. This restores test_from_beacon.yaml to its original form; bounds cassettes are re-recorded.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some recent work done on clinvar validation surfaced a small set of discrepancies which informed this change.
htslib does not enforce coordinates are in bounds for the sequence. This seems like intentional and defined behavior. And pysam mostly delegates to htslib for fetches (but disallows negative coords and
start>end), and seqrepo delegates to pysam, and vrs-python currently delegates to seqrepo.Entry points that now check bounds:
reference comparison, so the error isn't reported as a reference mismatch.
At a high level these are the layers and their bounds handling (prior to this branch which changes vrs-python):
(begin ai)
htslib (
faidx_fetch_seq) (converted to interbase for ease of comparison)The only signal that anything was clamped is the length htslib reports back for what it
actually fetched. The
samtools faidxcommand prints a "Truncated sequence" warning tostderr but still exits 0.
pysam (
FastaFile.fetch), compared with htslibseqrepo (
SeqRepo.fetch), compared with pysamNo differences: start/end are passed straight through.
vrs-python
main, local proxy (SeqRepoDataProxy.get_sequence), compared with seqrepoNo differences:
get_sequencepasses start/end straight through, and nothing else onmainchecks location bounds.The one related check,
validate_ref_seq, runs only on the gnomAD (VCF-style) inputpath. It compares the fetched reference with the input REF.
than a bounds error.
require_validation=False(thevrs-annotatedefault), that failure is only alogged warning, and an Allele is still returned.
HGVS, SPDI and beacon inputs never compare against the reference, and neither do
normalize()ortranslate_to. An out-of-bounds location from those paths is acceptedsilently and given a permanent identifier.
That is why the ClinVar issue went unnoticed on SeqRepo.
NP_001346993.1:p.Ter194del(a protein of 193 aa) parses to a deletion at
[193, 194), normalization's fetch returns'', and aga4gh:VAAllele is produced. It only surfaced because RefgetStore raised anerror on the same fetch. The comparison between the two backends then showed different
object types:
ga4gh:VAon SeqRepo, and on RefgetStore the harness's CNV fallback,ga4gh:CX.