Skip to content

fix(zarr-metadata): draw the same codecs on every derandomized run - #4635

Merged
d-v-b merged 4 commits into
zarr-developers:mainfrom
d-v-b:fix/zarr-metadata-derandomized-codec-draws
Nov 14, 2026
Merged

d-v-b merged 4 commits into
zarr-developers:mainfrom
d-v-b:fix/zarr-metadata-derandomized-codec-draws

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Ensure correctly derandomized codec pipelines for hypothesis strategies that draw them.

🤖 AI text below 🤖

Strategies in zarr_metadata.testing.strategies that draw codec pipelines, such as array_documents and pipelines, now draw the same examples on every derandomized Hypothesis run. A codec was sampled from options that hold strategies, which Hypothesis labels by their memory address, so the examples its mutator produced varied from one process to the next.

Cause

_codec in testing/_draw.py did st.sampled_from(options) over (definition, configurations) pairs, where configurations is a strategy. Hypothesis labels a sampled_from by hashing its elements, and a strategy hashes by id, so these spans were labeled by where the strategies sat in memory. During generation, Hypothesis's mutator copies choices between spans that share a label, so which spans it grouped, and so which examples a derandomize=True, database=None run drew, depended on memory layout. Machine load and other processes move that.

The agent traced every test case across parallel identical runs: the first real divergence was the same choice sequence producing different mutator groups (46 vs 47). Comparing span labels across two runs in one process pinned it to two transpose draws in one pipeline, which share a label only when the second pass's strategy reused the first's address. It is not timing logic in Hypothesis's engine, and not PYTHONHASHSEED (runs still diverged with the seed fixed). A standalone script reproduces it on Hypothesis 6.170.15; an upstream report is drafted, not filed.

_codec now samples definition names, as the chunk grid and data type draws already do. That draws the same integer choice, and keys the label on the option set: an earlier revision sampled an index, which gave every option list of the same length one label. An instrumented run over the array, group and consolidated-group document strategies found no other sampled_from labeled by an address.

Share tests
  • tests/testing/_shares.py gains examples_of, cached per strategy, which draws twice and fails if the runs differ. It is a tripwire, not a proof: on the old code a second run in the same process differed in some processes and not others.
  • across_seeds and bounds put the bound rule in code: each bound lies past the most extreme of 100 seeds by a quarter of that measure's distance from the median, rounded outward to 0.01. Every bound is reset by it. Two old bounds sat inside the seed range: the "no axes" ceiling (0.25) and the "an axis past 32 bits" floor (0.1). Editing the lambda line in _shares._draw reseeds every share, since Hypothesis derandomizes from its source.
  • The two first-data-type checks (array documents, struct fields) run over 1000 examples, each with a ceiling of 0.13. Over 300, bool's share across seeds reached 0.13 to 0.15, too near the 0.2 the checks guard against.

#4636 was stacked on this PR's first two commits and has merged, so main already holds their index-sampling version; this PR's diff against main is the name sampling, the bound tooling, and bounds re-measured on main with #4636's rectilinear change.

just check in packages/zarr-metadata passes (lint, pyright, tests on 3.11 and 3.14, docs-check). A naive-agent review and roborev found nothing blocking; the review's points are folded in.

🤖 Generated with Claude Code

`_codec` drew from `st.sampled_from` over `(definition, configurations)`
pairs whose configurations are strategies built during the draw. Hypothesis
labels such a `sampled_from` by hashing its elements, and a strategy hashes
by address, so these spans were labeled by memory layout. Hypothesis's
mutator copies choices between spans that share a label, so which spans it
grouped, and so which examples a derandomized run drew, changed with
whatever moved the allocator: another process, machine load. `_codec` now
samples an index into the options, labeled by a `range`.

The share tests measured this drift. `examples_of` now draws twice and
fails if the runs differ, which catches an address-labeled strategy in the
same process; the array-document targets share one draw. Bounds are reset
from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the
seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the
seed minimum (0.06).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package labels Nov 14, 2026
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.90%. Comparing base (22c8741) to head (2eeb6bc).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4635   +/-   ##
=======================================
  Coverage   94.90%   94.90%           
=======================================
  Files          96       96           
  Lines       14051    14051           
=======================================
  Hits        13335    13335           
  Misses        716      716           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 14, 2026
The share tests in `tests/testing` flaked on this branch: "an axis past
32 bits" below its floor, and "no axes" above its ceiling on Python
3.14. zarr-developers#4635 fixes the cause, a `sampled_from` that Hypothesis labeled by
memory address. It also resets the bounds from 100 seeds. The conflict
in `_DOCUMENT_SHARES` takes zarr-developers#4635's bounds.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ible

`_codec` samples definition names rather than an index. An index into
options of the same length shares one label with every other, so the
mutator could copy a choice between unrelated codec slots; names key the
label on the option set, as the chunk grid and data type draws already do.

`examples_of` is cached per strategy, so the share tests that count one
strategy draw it once, and its docstring calls the second-run check what
it is, a tripwire. `across_seeds` and `bounds` put the rule the bounds
follow into code, so a bound can be reset rather than re-derived; every
bound is reset by it. The two first-data-type checks run over 1000
examples: over 300, `bool`'s share across seeds reached 0.13 to 0.15,
too near the 0.2 they guard against. The changelog fragment no longer
says the options' strategies are built during the draw: some are cached.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b added a commit that referenced this pull request Nov 14, 2026
…types (#4636)

* fix(zarr-metadata): draw the same codecs on every derandomized run

`_codec` drew from `st.sampled_from` over `(definition, configurations)`
pairs whose configurations are strategies built during the draw. Hypothesis
labels such a `sampled_from` by hashing its elements, and a strategy hashes
by address, so these spans were labeled by memory layout. Hypothesis's
mutator copies choices between spans that share a label, so which spans it
grouped, and so which examples a derandomized run drew, changed with
whatever moved the allocator: another process, machine load. `_codec` now
samples an index into the options, labeled by a `range`.

The share tests measured this drift. `examples_of` now draws twice and
fails if the runs differ, which catches an address-labeled strategy in the
same process; the array-document targets share one draw. Bounds are reset
from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the
seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the
seed minimum (0.06).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): add changelog fragment for #4635

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): refuse an empty rectilinear edge list

zarr-extensions#78 (cadd1dab6) made the array form of a dimension's
chunk edge lengths non-empty, even for an axis of length 0, which
writes a bare integer to have no chunks. The rectilinear chunk grid's
rules now report `chunk_shapes: [[]]`, its JSON Schema says
`minItems: 1` for the list, and `chunk_grid_configurations` always
draws at least one edge. Spec links are re-pinned to cadd1dab6.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): read MinLen and MaxLen on arrays in the TypedDict checker

The checker, the JSON Schema writer and the pydantic schema now read
annotated-types' `MinLen`, `MaxLen` and `Len` on an array, as pydantic
does. One table of constraints now drives parsing, the shape check and
the schema keywords, in place of five parallel tables. An array outside
its bounds is an `invalid_value` problem whose `ctx` holds `min_length`
or `max_length`. A length bound on anything but an array raises
`TypeError`.

`codecs` and each rectilinear dimension's edge list now declare
`MinLen(1)` on their types. This removes the two hand-patched
`minItems: 1` schema fragments and the rectilinear `_rules` function.
`chunk_shapes: [[]]` and `codecs: []` are still refused at the same
`loc`. `check` now also refuses `codecs: []`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): declare sharding's codec pipelines non-empty

The `sharding_indexed` codec's `codecs` and `index_codecs` each hold
exactly one array -> bytes codec, so neither is ever empty. Both now
declare `MinLen(1)` on their types, as the array document's `codecs`
does. An empty one is reported by the type check at the same `loc` as
before, with `ctx={"min_length": 1}`. Before, the pipeline's order check
reported it as "expected an array -> bytes codec, got none".

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(zarr-metadata): move constant data type rules onto their types

`null_terminated_bytes` and zarr-python's `raw_bytes` checked
`length_bytes >= 1` in a rule, and v3 `struct` checked that it has a
field. The types now declare `Ge(1)` and `MinLen(1)`, the two rules that
did only this are gone, and the struct rule keeps its name and field-type
checks. The problems keep their `loc` and kind. The JSON Schema of each
type now says so too: it accepted `length_bytes: 0` and `fields: []`,
which the reader refuses.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): read MultipleOf on integers in the TypedDict checker

The checker, the JSON Schema writer and the pydantic schema now read
annotated-types' `MultipleOf` on an integer, as pydantic does under
`multiple_of`. JSON Schema's keyword is `multipleOf`. The step must be a
positive integer, and a type takes one. Two steps on a NewType and its
annotation merge to their least common multiple. A float is refused,
because validators compare floats inexactly.

Each constraint in the table now names the shapes it applies to, with
the JSON Schema keyword for each, and a slot (below, above or step) in
place of a side.

`fixed_length_utf32`'s `length_bytes` and raw `r<N>` `bits` now declare
`Gt(0), MultipleOf(8 or 4)` on their types. The two rules that checked
only this are gone. A test whose own `r*` definition accepts any size
now uses its own configuration type.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): read MinLen and MaxLen on strings in the TypedDict checker

`MinLen`, `MaxLen` and `Len` now also bound a string, as pydantic's
`min_length` does. The JSON Schema writes `minLength` and `maxLength`.
Python and JSON Schema both count code points, so the checker and the
schema agree on what a string's length is. Messages count characters:
"expected a string with at least 1 character, got \"\"". A length bound
on an object still raises `TypeError`.

The v3 `struct` field name now declares `MinLen(1)` on its type, and
the struct rule no longer checks for an empty name. The problem keeps
its `loc` and kind.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(zarr-metadata): number the changelog fragments by this PR

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(zarr-metadata): loosen the floor on documents with an axis past 32 bits

The derandomized share stream changes with any change to what the
strategies draw, and with machine load: base main measures 0.103
against a floor of 0.1 under load, and the rectilinear fix measures
0.047 to 0.133 across runs. Over seeds 0-5 at 500 examples both trees
average about 0.17, from 0.07 to 0.29. A floor of 0.03 still asks that
a few hundred examples meet several such axes.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): hold a length bound beside an unknown key

`constrained` skipped a type's constraints whenever the inner parse
reported any problem. An `unknown_key` problem leaves the value readable,
because the key is dropped and the value is still returned. So
`check` returned an array of closed TypedDicts longer than its `MaxLen`,
and a definition's rules ran on it, although the rules run only on a
configuration within its bounds. The constraints are now skipped only
for a problem other than an unknown key. Number bounds were never
affected, because a number holds no keys.

The docs now list `multiple_of`, `min_length` and `max_length` among the
`ctx` keys, and `constraints_of` says a type takes one step too.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): name the member a carried name's type check refuses

A raw `r<N>` name's size is now checked by its type. For
`"data_type": "r12"` that gave "expected an integer > 0 that is a
multiple of 8, got 12", about a string the document wrote, with no
mention of bits. A type-check problem with a configuration that a name
encodes now names the member and the name: `bits in "r12": ...`. A
rule's message is left as written, because such rules already describe
the name, as the v2 typestr rules do.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(zarr-metadata): pin that the types refuse empty pipelines and edge lists

The sharding tests for an empty `codecs` or `index_codecs` checked only
the `loc` and kind, which the pipeline's order check also reports. They
now also assert `ctx={"min_length": 1}`, which only the type check gives.
The rectilinear strategy test checked the grid's rules, which no longer
look at empty lists. It now reads the configuration with the definition,
which checks the type. The changelog no longer says the JSON Schema of a
raw `r<N>` name carries its multiple of 8: that schema is the name
pattern.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zarr-developers#4636 was stacked on this PR's first two commits and squash-merged, so
main holds their index-sampling version; each conflict keeps this
branch's newer version. zarr-developers#4636's rectilinear change (always at least one
edge) moves the array-document shares, so their bounds are re-measured
with `bounds(across_seeds(...))`: `bool`'s seed maximum over 1000
examples rose to 0.108, and its ceiling to 0.13.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@read-the-docs-community

Copy link
Copy Markdown

@d-v-b
d-v-b marked this pull request as ready for review November 14, 2026 21:32
@d-v-b
d-v-b merged commit 1ab86a0 into zarr-developers:main Nov 14, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant