Skip to content

test: pin the behavior 100% coverage was not pinning - #1404

Open
raman325 wants to merge 1 commit into
mainfrom
test/property-codec-coverage
Open

test: pin the behavior 100% coverage was not pinning#1404
raman325 wants to merge 1 commit into
mainfrom
test/property-codec-coverage

Conversation

@raman325

Copy link
Copy Markdown
Owner

Proposed change

Line coverage reached 100% in #1403, which is exactly the point at which it stops telling you anything. Two harder measurements say otherwise:

  • Branch coverage is 99%, not 100% — 57 partial branches sit behind the full-line number.
  • A mutation-testing sample scored 76.2% across four pure-logic modules (resilience.py, credentials.py, _util.py, config.py). Roughly a quarter of deliberately-introduced defects were executed by the tests without being detected.

resilience.py and _util.py both scored 100% — their assertions are genuinely strong. credentials.py scored 41.7%.

Three functions had no direct assertions at all

Each reached 100% line coverage purely through its callers:

Function Why it matters
LockCapabilities.bounded_slot_count The gate for the out-of-range slot check added in #1401. Every mutation of its condition survived (orand, isis not, <=>, 01). Wrong here either blocks legitimate writes or admits an impossible slot.
WriteResult.changed Decides whether the seam refreshes the coordinator. Truth table never asserted.
build_slot_unique_id Produces the entity registry key. Not referenced by a single test, so the | separator every existing entity depends on was unpinned.

EntryConfigDiff.has_changes was tested only at the extremes — every existing test changes slots and locks together, or neither — so mutating any individual or to and survived. It gates the Lovelace re-render, so a collapsed disjunct means a user who only adds a slot silently gets a stale dashboard.

Property tests for the identifier codecs

The tag codec in providers/_util.py has had round-trip and totality properties since the property suite was created. The structurally identical codec in domain/config.py had none — and that asymmetry cost a real bug: #1402's parse_slot_device_identifier gated on str.isdigit(), rejecting the negative slot its own builder emits, which made the device invisible to both the orphan sweep and the removal hook.

Reverting that fix against the new property:

E  AssertionError: assert None == -1
E  Falsifying example: test_device_identifier_round_trips(slot=-1)

Under a second, minimal counterexample. The example-based tests that shipped alongside the builder do not notice, because nobody writes slot -1 down by hand. That is the whole argument for the property.

PIN masking

A round trip over secret material, previously covered only by examples. The properties pin what actually matters: the mask is deterministic (one PIN reads as one token throughout a log), constant-width (a variable-width token discloses PIN length, meaningful for a 4–8 digit code), salted per slot (a log cannot reveal that two slots share a PIN), and deobfuscate_pins is total on arbitrary pasted text.

Deliberately not asserted: that the token never contains the PIN as a substring. An 8-hex-char token can contain 1234 by chance, so that property would flake — the sound formulation is the constant-width one above.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature (which adds functionality)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue:
  • This PR is related to issue:

Mutation score over the sampled modules: 76.2% → 92.6% (credentials.py 41.7% → 80.6%, config.py 79.5% → 94.9%), re-measured with the same harness after these tests. Every remaining survivor is either a slots=True dataclass toggle — no observable effect when frozen=True already blocks the writes the tests attempt — or an error-message string.

Tests only; no production code touched. Suite: 1457 passed, 100% line coverage retained, ~59s. Verified at HYPOTHESIS_PROFILE=ci (200 examples) as well as the dev default.

🤖 Generated with Claude Code

Line coverage hit 100% in #1403, which is exactly the point at which it
stops being informative. Branch coverage is 99% (57 partial branches), and
a mutation-testing sample over four pure-logic modules scored 76.2% --
meaning roughly a quarter of deliberately-introduced defects were executed
by the tests without being detected.

Three functions turned out to have NO direct assertions at all, reaching
100% line coverage purely through their callers:

- `LockCapabilities.bounded_slot_count` -- the gate for the out-of-range
  slot check. Every mutation of its condition survived. Getting it wrong
  either blocks legitimate writes or admits an impossible slot.
- `WriteResult.changed` -- decides whether the seam refreshes the
  coordinator. Its truth table was never asserted.
- `build_slot_unique_id` -- produces the entity registry key. No test
  referenced it, so the pipe separator that every existing entity depends
  on was unpinned.

Also adds property tests for the two identifier codecs in `config.py`.
Their sibling in `providers/_util.py` has had round-trip properties since
the property suite was created; these had none, and that asymmetry cost a
real bug -- #1402's parser gated on `str.isdigit()` and so rejected the
negative slot its own builder emits. Reverting that fix makes the new
round-trip property fail in under a second with the minimal counterexample
`slot=-1`; the example-based tests that shipped with the builder do not
notice, because nobody writes slot -1 down by hand.

PIN masking gets properties too. It is a round trip over secret material:
the mask has to be deterministic (one PIN reads as one token across a log),
constant-width (a variable-width token discloses PIN length), salted per
slot (so a log cannot reveal two slots share a PIN), and total on arbitrary
pasted text.

Mutation score over the sampled modules: 76.2% -> 92.6%. Every remaining
survivor is a `slots=True` dataclass toggle with no observable effect
under `frozen=True`, or an error-message string.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 9d178f84aeb0
Copilot AI lite review requested due to automatic review settings August 11, 2026 05:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added python Pull requests that update Python code code-quality Pull requests that improve code quality labels Aug 11, 2026
@codecov

codecov Bot commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.95%. Comparing base (68492f1) to head (5eafaa0).
✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1404   +/-   ##
=======================================
  Coverage   98.95%   98.95%           
=======================================
  Files          53       53           
  Lines        6598     6598           
  Branches      470      470           
=======================================
  Hits         6529     6529           
  Misses         69       69           
Flag Coverage Δ
python 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

code-quality Pull requests that improve code quality python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants