Skip to content

fix(corekit): give the MultiDict _missing sentinel @final and __bool__ - #667

Closed
JarryShaw wants to merge 1 commit into
mainfrom
fix/missing-sentinel-final-bool
Closed

JarryShaw wants to merge 1 commit into
mainfrom
fix/missing-sentinel-final-bool

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Sep 22, 2026

Copy link
Copy Markdown
Owner

Site 2 of the three sentinel sites the sweep in #661 left open. Site 1 (NoValue) needs nothing
and is untouched. Site 3 (_NOT_FOUND) is a decision rather than a change, and the reasoning is
posted on #661 rather
than acted on here.

What was short of the convention

_missing was already what the review comment on #640 asked for — an instance of a purpose-built
class, not a bare object(). Measured against NoValueType at pcapkit/corekit/fields/field.py:26-36,
which is the package's convention for a sentinel of this kind, it lacked exactly two things:
@final and __bool__.

  • It answered True to bool(). A marker meaning no default was supplied that is truthy
    states the opposite of what it means. NoValueType defines __bool__ returning False for
    precisely that reason.
  • _Missing was subclassable. The convention marks a singleton marker @final; nothing
    subclasses it, and nothing should.

The issue's own recommendation was @final only, on the grounds that __bool__ "would also be
a live behaviour change rather than a cosmetic one, since this instance is reachable by callers".
That reservation is worth taking seriously, and the measurements below are what settle it — the
deciding fact is that _missing can never escape through the public API: pop() returns
default only on the branch where default is not _missing, so no caller can ever receive it,
including one that passes _missing itself as default (that still raises). The only routes to its
truthiness are deliberate introspection — inspect.signature(MultiDict.pop).parameters['default'].default,
or plain MultiDict.pop.__defaults__, which needs no import at all. On those routes False is the
more correct answer, because if default: currently reports "there is a default" when there is
not. If you would still rather have @final alone, drop the __bool__ method and the three
assertions naming it; the rest of the change stands unchanged.

_missing is a private name by every normal measure — underscore-prefixed, absent from __all__,
previously undocumented. The claim here is narrower than "it is public": it is that its value is
handed out by a public signature, which is what makes its truthiness worth being correct.

What was measured before touching it

All four _missing sites, and none tests truthiness. Two are parameter defaults
(multidict.py:373, :592 pre-change) and two are identity tests
(:393, :596 — both if default is not _missing:). Nothing anywhere does if default:, and there
is no fifth reference in pcapkit/, tests/, docs/ or examples/.

Nothing subclasses _Missing, package-wide or under tests/. The only references outside its own
definition are the four sites above plus two assertions in tests/corekit/test_multidict.py:34-35.

The pickle round trip still works, at every protocol__reduce__ returning the bare string
_missing is pickle-by-name, and it is the reason the singleton survives a process boundary. The
emitted bytes are byte-identical before and after:

pickle proto 0: b'cpcapkit.corekit.multidict\n_missing\np0\n.'          back is _missing -> True
pickle proto 4: b'\x80\x04\x95*\x00\x00\x00\x00\x00\x00\x00\x8c\x19pcapkit.corekit.multidict\x94\x8c\x08_missing\x94\x93\x94.'
                                                                        back is _missing -> True

And across a genuine process boundary, which is the case the docstring claims and the unit test
cannot reach. Dumped in one interpreter, loaded in a second that had not imported the module:

B: multidict imported before loads? False
B: multidict imported after loads?  True
B: loaded obj is B-own _missing -> True
B: repr = no value | bool = False

A before/after probe of every observable differs on exactly three lines — the three that are the
point of the change:

- bool(_missing)   = True        + bool(_missing)   = False
- has __bool__     = False       + has __bool__     = True
- __final__ attr   = <absent>    + __final__ attr   = True

repr, __reduce__, the pickle bytes at protocols 0 through 5, both pop() signature defaults, and
every pop() result — including the falsy defaults None, False, 0, '', [], and a second
_Missing() instance that is falsy but not identical — are unchanged.

The @final assertion is not vacuous

typing.final only began recording __final__ on the decorated class in Python 3.11, and
pcapkit.utilities.compat.final is typing.final on everything from 3.8 up. A bare
assertTrue(hasattr(_Missing, '__final__')) would therefore pass vacuously on 3.8 through 3.10,
which are inside the declared support range. The test probes the decorator with a throwaway class
and skips with a named reason where it cannot record the mark, rather than asserting nothing.

Docs

docs/source/pcapkit/corekit/multidict.rst had no directive for _Missing at all, so the new
docstrings would have been unreachable by any doc build — a self-contradiction, since the class
docstring says it follows NoValueType, which is wired in at
docs/source/pcapkit/corekit/fields/field.rst:15-19. It now carries an Auxiliaries section
mirroring that one. The .. automethod:: entries for the two dunders follow this package's own
idiom — docs/source/pcapkit/corekit/context.rst:75 already documents a __bool__ that way.

The whole-project docs build takes over nine minutes, which is too slow to gate a four-line rst
addition, so the section was validated with a minimal single-file Sphinx project under -W (warnings
as errors): exit 0, with the class docstring, both dunder docstrings and the #: comment on
_missing all rendering and every role resolving.

Verification

Failing first, then passing, pytest's exit code read from a file rather than off a pipeline, and
pcapkit.__file__ asserted into the worktree with __editable__* stripped from sys.meta_path
before any other import. (Those finders are classes, so type(f).__module__ is builtins and a
filter written that way silently misses them.)

run result exit code
new tests, source at 0c7f2b7c9 2 failed, 9 passedAssertionError: True is not False, then AttributeError: type object '_Missing' has no attribute '__final__' 1
new tests, source changed 11 passed, 20 subtests passed 0
tests/corekit/ in full 169 passed, 398 subtests passed 0
tests/project/ in full 96 passed, 469 subtests passed 0

Coverage does not go backwards — pcapkit/corekit/multidict.py is at 100% statement and branch
on both sides, 284 → 288 statements with 0 missed and 0 partial branches either way. The four added
statements are the import, the @final line, def __bool__, and its return False.

pylint, mypy, isort, vermin and bandit on the changed file report the same findings before
and after
: mypy's 10 pre-existing unused-ignore/override errors map one-to-one onto the new line
numbers shifted by exactly the 41 lines added, pylint's 4 findings are all outside the diff hunks
(lines 187/429/444/508), vermin's single finding is the untouched typing_extensions import, and
bandit is clean. No new finding was introduced.

A pre-existing failure found on the way, not caused by this change

Running tests/corekit/test_multidict.py and tests/project/ in the same process fails three
tests/project/test_public_api.py tests. Confirmed pre-existing: reproduced at untouched
origin/main (0c7f2b7c9) in a throwaway worktree, same three failures, exit 1 — and reproduced on
this branch with both new tests --deselected. The cause is that bootstrap_core_modules()
ensure_package() in tests/_support.py installs bare stub pcapkit, pcapkit.corekit and
pcapkit.utilities modules into sys.modules and nothing restores them, so a later test that
introspects the real public surface sees the stubs (these public packages declare no __all__). The
full suite is green because an intervening test directory re-imports the real package, which is why
this only surfaces when those two paths are run adjacently. Reported rather than fixed here, since
it belongs in its own change.

Label

fix — not feat (no new capability), not docs (there is a code change), not test (the test is
evidence, not the point). breaking was considered and rejected: _missing is not in __all__, and
pop() structurally cannot return it, so no public contract changes. The one thing that genuinely
flips is bool() of the sentinel itself on the two introspection routes above — obscure, but real,
so breaking is additive and can be added alongside fix if you read that reachability differently.

Refs #661

@JarryShaw JarryShaw added the fix Pull requests that fix a defect (fix: subject prefix) label Sep 22, 2026
JarryShaw added a commit that referenced this pull request Sep 22, 2026
The bullet #667 would otherwise have carried, kept here so that #667 touches only
`pcapkit/corekit/multidict.py` and `tests/corekit/test_multidict.py`.

One bullet, because it is one convention gap in one class: `_Missing` behind
`MultiDict.pop` and `OrderedMultiDict.pop` lacked the `@final` and the falsy
`__bool__` that `NoValueType` in `pcapkit.corekit.fields.field` sets as the
package's convention for a marker of this kind.

The bullet says plainly that no behaviour changes, and says why rather than
asserting it: both `pop()` implementations decide by identity, never by
truthiness, and `pop()` structurally cannot return the marker -- it returns
`default` only on the branch where `default is not _missing`. It also names the
one way the old truthiness was observable, which is what justifies touching it at
all: `inspect.signature(MultiDict.pop).parameters['default'].default` hands the
marker to any caller who asks, and `if default:` on it reported "a default was
supplied" where none had been.

It closes by recording the disposition of the other two sites from the #640
sweep, so the entry is the whole story: site 1 needed nothing, and `_NOT_FOUND`
in `pcapkit.utilities.compat` stays a bare `object()` deliberately, being a
verbatim line of CPython's `functools.cached_property` inside a
`sys.version_info < (3, 8)` branch no supported interpreter reaches. The
reasoning behind that one is on #661, not here.

`:obj:` roles had to come out: `util/changelog_md.py` rejects them with
`ResidualMarkupError`, since its six conversion rules do not cover interpreted
text and `CHANGELOG.md` would carry the role through as literal text. Double
backticks instead, which is what the rest of the entry file uses.

20 lines added to the entry file; `CHANGELOG.md` regenerated with
`util/changelog_md.py`, not edited. `--check` exits 0 and `tests/project/` is
green at 96 passed, 469 subtests.

Committed from a detached HEAD on e55ba36 and pushed to the branch ref, because
`docs/changelog-1.5.0` is checked out in another agent's worktree at a stale
f846523 and could not be taken here. Note e55ba36, not the 69a6e13 I was
given: the branch had already moved on with #665's and #651's entries.

Refs #661
…bool__` (#661)

The sweep in #661 found `_missing` already a purpose-built class rather than a
bare `object()`, but short of the convention `NoValueType` sets at
`pcapkit/corekit/fields/field.py:26-36`.

* A marker meaning *no `default` was supplied* answered `True` to `bool()`,
  stating the opposite of what it means. It is a private name, but its value is
  not hidden: it is the runtime default of `MultiDict.pop` and
  `OrderedMultiDict.pop`, so both
  `inspect.signature(MultiDict.pop).parameters['default'].default` and plain
  `MultiDict.pop.__defaults__` hand it to a caller who asks. Now defines
  `__bool__` returning `False`.
* `_Missing` was subclassable, which the convention forbids for a singleton
  marker. Now carries `@final`, from `pcapkit.utilities.compat`.
* Documented the class and all three dunders, including why `None` cannot serve
  as the marker and why `__reduce__` returns a bare name -- and wired them into
  `docs/source/pcapkit/corekit/multidict.rst`, which had no directive for
  `_Missing` at all, so the new docstrings would have been unreachable by any doc
  build. The `Auxiliaries` section mirrors the one
  `docs/source/pcapkit/corekit/fields/field.rst:15-19` gives `NoValueType`/`NoValue`.

Both `pop()` implementations decide on identity (`is not _missing`), never on
truthiness, so the falsy marker changes no behaviour: `repr`, `__reduce__`, the
pickle bytes at every protocol 0-5, both signature defaults, and every `pop()`
result including the falsy defaults `None`/`False`/`0`/`''`/`[]` are unchanged.
A fresh process that has not imported the module still unpickles the marker to
its own `_missing`.

`tests/corekit/`: 169 passed, 398 subtests; `tests/project/`: 96 passed, 469
subtests. `multidict.py` stays at 100% statement and branch coverage (284 -> 288
statements, 0 missed either side). The added rst builds clean under `-W`.

Refs #661
@JarryShaw
JarryShaw force-pushed the fix/missing-sentinel-final-bool branch from e101851 to 064ad15 Compare September 22, 2026 19:19
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict

GOOD TO GO — with one required change, which has been made.

That is the reviewer's verdict verbatim: it opened GOOD TO GO, with one required change: and then
named a blocking item. Since a named required change is not a clean pass, it was treated as blocking
and fixed in 064ad1594 before this comment; the verdict is reproduced here rather than paraphrased
so the qualification is on the record.

Cross-review ran on Sonnet (this change was authored on Opus), briefed to falsify rather than
bless, read-only, on the nine load-bearing claims the PR body asserts. No model substitution was
needed. 80 tool calls, ~20 minutes.

The required change, and what was done

docs/source/pcapkit/corekit/multidict.rst had no directive for _Missing/_missing, so the new
class, __bool__ and __reduce__ docstrings were unreachable by any doc build — while the class
docstring claims to follow NoValueType, which is wired in at
docs/source/pcapkit/corekit/fields/field.rst:15-19. The reviewer called this "a real,
self-contradicting gap, not a hypothetical one", and it is right.

Fixed: an Auxiliaries section mirroring field.rst's, with .. autoclass:: for _Missing,
.. autodata:: with :no-value: for _missing, and .. automethod:: for the two dunders whose
contracts matter. The dunder entries follow this package's own idiom rather than inventing one —
docs/source/pcapkit/corekit/context.rst:75 already documents a __bool__ that way, and
__index__/__repr__/__post_init__ appear the same way 30, 6 and 17 times across docs/.
Validated with a minimal single-file Sphinx project under -W, exit 0, all four docstrings rendering
— the whole-project build takes over nine minutes and timed out, so it is not a usable gate on four
lines of rst.

What the reviewer independently reproduced

All eight remaining claims verified TRUE, each with evidence it obtained itself rather than
re-reading this PR:

  • Four _missing sites, no fifth, and none truth-tests.
  • Failing-then-passing reproduced from scratch in its own throwaway worktree at 0c7f2b7c9:
    2 failed, 9 passed, exit 111 passed, 20 subtests, exit 0.
  • Coverage re-measured with coverage run (no pytest-cov): 288 stmts, 0 miss, 110 branch, 0 partial, 100% after; 284/0/110/0, 100% before. Reproduced exactly.
  • Pickle at every protocol, with the two quoted byte strings reproduced byte-identically — and
    the fresh-process case, confirming 'pcapkit.corekit.multidict' not in sys.modules immediately
    before pickle.loads, then result is that_module._missingTrue.
  • @final breaks nothing: it subclassed _Missing at runtime to confirm the decorator is a
    static marker only, and instantiated it a second time as the test's falsy-but-not-identical case
    needs. mypy on both trees: 99 errors in 34 files either side, the only difference being the ten
    multidict.py line numbers shifted by exactly +41.
  • The 3.11 __final__ claim verified against real interpreters rather than from memory:
    hasattr(_Probe, '__final__') is False on 3.8.20, 3.9.25 and 3.10.21, True on 3.11.15 and
    3.12.13. It also tried and failed to construct a case where _Probe and _Missing could diverge
    on the mark, so it judged the probe-and-skip sound.
  • pylint / bandit / isort / vermin identical before and after, using the Makefile's real flag
    sets — it noted that bare pylint defaults give a misleading 15 findings because they do not disable
    design/invalid-name, which is worth knowing for anyone re-running it.

Where it disputed the PR, and what changed as a result

Two things, both accepted:

  1. "It is not private in practice" was overstated rhetoric. _missing is private by every
    normal measure — naming, absent from __all__, previously undocumented — and is reachable only by
    deliberate introspection. The commit message and PR body now say that plainly instead: the name is
    private, but its value is handed out by a public signature, which is the narrower claim that
    actually justifies the change.
  2. A second escape route the PR had not disclosed: MultiDict.pop.__defaults__(no value,),
    plain function introspection with no inspect import at all. It called "only two ways" an
    undersell and preferred "one deliberate-introspection family of ways". Both are now named in the
    PR body and the commit message. This strengthens rather than weakens the case for __bool__,
    since the value is slightly easier to reach than claimed.

It also pushed back on the breaking judgement without overturning it: fix is right, but
"genuinely reachable and its truthiness genuinely changes" is a real if obscure fact, and it thought
that deserved to be on record given how confidently the PR argues unobservability. The PR body now
says so.

What it could not verify

A full sphinx-build of the whole project. It confirmed :func:~typing.final`` points at a real
intersphinx target by reading docs/source/conf.py:84-85, but did not execute a package-wide build.
Partly closed since: the minimal `-W` build above exercises every role in the new block, though not
the rest of the package.

Nits it raised and I did not act on

  • :func:~typing.final`` is "slightly imprecise, not wrong" — the class is decorated via the
    pcapkit.utilities.compat shim, not `typing` directly. Left as-is: the role resolves, and `typing`
    is where the decorator's semantics are actually documented.
  • The changelog bullet on docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657 retains the original "not private in practice" phrasing. Left alone
    deliberately rather than force-pushed: that branch is long-lived and shared with other agents'
    worktrees, and the bullet states the precise mechanism in the very next clause, so it is not
    misleading in context.

One thing the reviewer observed that was me

It reported this worktree's HEAD moving under it mid-review — briefly detached at e55ba367c with a
modified docs/source/changelog/1.5.0.rst, then back. That was deliberate and mine: the changelog
entry for this work goes on the shared docs/changelog-1.5.0 branch for #657, which is checked out in
another agent's worktree and so had to be committed from a detached HEAD here and pushed to the
branch ref. Good catch, and a real hazard of handing a reviewer the same worktree — it responded
correctly by moving its own before/after comparisons into a separate throwaway worktree, which it has
since removed.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Closing: _Missing is a verbatim werkzeug port, and the one doc change here is against convention

Two owner rulings, both against this change.

1. _Missing is a direct port, so it stays verbatim

In the owner's words: "OrderedDict was a direct port from werkzeug's implementation. might not worth updating it."

Verified rather than assumed — fetched src/werkzeug/_internal.py at three refs, main, 2.3.8 and 1.0.1:

class _Missing:
    def __repr__(self) -> str:   return "no value"
    def __reduce__(self) -> str: return "_missing"

_missing = _Missing()

Byte-identical to pcapkit/corekit/multidict.py apart from this package's string annotations, and unchanged on werkzeug's main today. Same principle that closed #661 for _NOT_FOUND at pcapkit/utilities/compat.py:73, which is verbatim CPython v3.8.0 Lib/functools.py:927.

One honest distinction, recorded because it argued the other way and was considered: multidict.py as a whole is not a frozen copy. It says "inspired and based on" at :9-11, raises MissingKeyError / UnsupportedCall from pcapkit.utilities.exceptions instead of werkzeug's, is Generic[_KT, _VT], and removes setlistdefault outright at :575. So the file has diverged deliberately. The rule applies to the specific block that still matches upstream character for character, which _Missing does.

2. The doc section this PR adds is against the private-member convention

A convention the owner has now stated: "we'd normally not include _xxx private members in the doc by default unless it carries something that's cross referenced in the public member's docstring/docs, e.g., TypeVar, attributes, etc."

This PR adds an Auxiliaries section to docs/source/pcapkit/corekit/multidict.rst with autoclass:: pcapkit.corekit.multidict._Missing and autodata:: …._missing. Nothing public cross-references either name — checked, and the only _missing hits under docs/source/ are changelog prose about the unrelated enum _missing_ hook. A name appearing as a signature default in pop(key, default=_missing) is not a cross-reference.

So the cross-review's one required change — "multidict.rst had no directive for _Missing" — was itself against convention. Recording that plainly: the reviewer was right that the docstrings were unreachable by any doc build, and the correct resolution was to not add the docstrings, not to add the directive.

What is being given up

The change was zero-behaviour either way: all four _missing sites are identity tests (:373, :393, :592, :596), so __bool__ was purely defensive and @final is a static marker only. What is lost is docstrings on the sentinel — and under convention 2 those would not have been rendered anyway.

Nothing merged from this branch. #661 remains closed, its site 1 and site 2 both now resolved as "no change".

A post-board sweep to remove existing private-member directives elsewhere in the docs is being filed separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant