Skip to content

Native YAML decoding on the stock parser (WIP) - #26

Closed
reuvenharrison wants to merge 78 commits into
masterfrom
spike/native-yaml-stock-goyaml
Closed

reuvenharrison wants to merge 78 commits into
masterfrom
spike/native-yaml-stock-goyaml

Conversation

@reuvenharrison

Copy link
Copy Markdown

Draft, not close to merge. Opening it so the work is reviewable and the open problems are visible. Status is stated honestly below; nothing here is finished.

What this is

kin decodes a YAML document by converting it to map[string]any, re-serialising to JSON text and parsing it again, because these types implement UnmarshalJSON and not UnmarshalYAML. Positions cannot survive that, so they are smuggled through as synthetic __origin__ nodes and reapplied afterwards by a reflection walk.

This replaces that with UnmarshalYAML on every type, decoding straight from the node, against unmodified go.yaml.in/yaml/v3 — no fork, no patches.

Why it is worth doing

Measured on real types with their real hooks, decoding 400 responses:

ns/op B/op allocs
round trip (today) 5.84M 5.33 MB 90,381
json.Unmarshal + hooks 3.02M 2.76 MB 48,401
native UnmarshalYAML 2.30M 2.32 MB 37,129

2.54x faster than today, and faster than the JSON fast path, because the hooks re-serialise: Responses.UnmarshalJSON re-marshals every child back to JSON and re-parses it once per entry, and SchemaRef.UnmarshalJSON parses the same bytes up to four times.

It also removes ~495 lines of origin transport across three repos, since a node already carries Line/Column.

Shape of the change

The 41 types fall into four groups, which is what makes this tractable:

  • 24 shadow-struct types, generated. The JSON versions restate the known field set as a list of deletes — Schema.UnmarshalJSON is 91 lines, ~60 of them — which silently misfiles a field added to the struct and forgotten in the list. Here it comes off the struct tags and cannot drift.
  • 9 $ref wrappers, one helper plus thin methods.
  • 3 maplike collections, one generic helper.
  • 4 special: Header defers to Parameter; Types, BoolSchema and ExclusiveBound are union-typed scalars.

No end positions are used. A block's extent is derivable from start positions — the next key or sequence item at the same or shallower indentation — which agrees with recorded ends on ~99.98% of ~11.9M spans. That is what removes the need for a patched parser.

Open problems

1. YAML 1.1 implicit resolution. The old path normalised every scalar through JSON's type system on the way through the round trip. Decoding natively keeps YAML 1.1 semantics, so implicit resolution now applies:

  • a date-shaped example became a time.Time and failed validation — fixed here by retagging before decode, reproducing the DisableTimestamps option our yaml fork has and stock go-yaml does not
  • a spec with the key 18_24 loads it as the integer 1824
  • examples and defaults whose type shifts stop being format-checked

Timestamps were one instance of a family. The family needs a considered answer, not another special case. This is the largest open item.

2. 24 failures in openapi3. 13 are tests asserting EndLine/EndColumn, which is the deliberate design change. The other 11 are real: the arbitrary-top-level-key $ref path, nil-versus-empty on maplike collections (an absent responses no longer fails validation), a changed error message, and origin coverage gaps.

3. Duplicate keys. JSON resolves a repeated key last-one-wins; YAML rejects it. The json path is kept as a fallback so such documents still load, without origins.

4. originFileVar is package-level. UnmarshalYAML receives a node and nothing else, so the file name cannot be threaded through. This follows the precedent of IncludeOrigin and inherits its concurrency characteristics. Making both per-Loader is worth doing and is a separate change.

Validation so far

  • Every full document in testdata (18) decodes identically to the JSON path.
  • Origins match the current path on Key.Line/Column, Fields and Sequences, compared against applyOrigins on the same documents.
  • The apis-guru corpus is what surfaced the timestamp problem, and is the reason problem 1 is stated rather than assumed absent.

Two bugs the suite caught that would otherwise have shipped: origins were produced when the caller had not asked for them (the gate read the package global rather than the flag the Loader passes, and since NewLoader seeds itself from that global, test pollution made it look correct); and a parameter inside a sequence had no Key, because stamping walked mappings only.

@reuvenharrison

Copy link
Copy Markdown
Author

openapi2 is untouched, and is a possible follow-up

This migrates openapi3 only. openapi2 still decodes by converting the document to JSON text and parsing it again, so github.com/oasdiff/yaml and github.com/oasdiff/yaml3 remain in go.mod even though openapi3 no longer references either.

Dropping those dependencies entirely needs openapi2 to move as well. It looks considerably smaller than this change: 9 types with UnmarshalJSON against 41, already dual-tagged with yaml alongside json, and no Origin to carry, so the methods reduce to the shadow-decode-plus-extensions shape with nothing else in them. The two odd cases are SchemaRef, which is the same $ref-or-value shape as its openapi3 counterpart, and openapi2/marsh.go, which needs the same switch as openapi3/marsh.go.

Also still on the wrapper: cmd/validate/main.go and a handful of tests in openapi2, openapi2conv and openapi3.

Deliberately out of scope here. Worth doing as its own change, where it can be reviewed against openapi2's own behaviour rather than buried in this one.

fenollp and others added 29 commits October 22, 2026 13:00
…y version

The Reference Object of OpenAPI 3.0, 3.1 and 3.2 alike says "any
properties added SHALL be ignored". From 3.1 on only summary and
description mean something there, and the loader already applies them;
anything else was still reported as extra sibling fields, failing
documents the specification calls valid.

So validateExtras now only reports the x- extensions next to a $ref, under
ProhibitExtensionsWithRef. That option was disabled for OpenAPI 3.1
schemas, only because their sibling keywords were reported together with
those extensions: it now applies in every version, as documented.
ExtraSiblingFieldsError is still returned for the unknown fields of inline
objects.

validateExtras also reported the fields next to a $ref once the $ref was
gone: InternalizeRefs replaces a component that is a $ref by what it
refers to, which leaves those fields out of the document, but not out of
what Validate checked.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
… schema

In JSON Schema 2020-12, on which OpenAPI 3.1 bases its Schema Object,
the keywords next to a $ref apply alongside it: an instance must satisfy
both the referenced schema and its siblings. They were instead copied
over the referenced schema's keywords of the same name, so a sibling
maxLength: 10 relaxed a referenced maxLength: 3, a sibling required list
dropped the referenced one, and sibling properties replaced the
referenced ones.

Annotations, which describe rather than constrain, still override the
referenced ones. Every other sibling keyword now goes into one more allOf
entry of the copy the loader makes, so Value keeps the shape of the
referenced schema. getkin#1271 reaches the same validation results by making
the siblings the Value and the referenced schema an allOf entry, which
leaves Value without the type or properties of the referenced schema
whenever a $ref has even just a description next to it.

The $refs within the siblings are now resolved before they are applied,
since applyRefSiblings also runs from the callback that ends a cycle.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
They were dropped. An OpenAPI 3.1 schema's sibling keywords, which
constrain the referenced schema, were lost that way, and so was every
other field next to a $ref: a document did not survive a round trip.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
The loader records the $refs it is in the middle of resolving, to close
cycles: a $ref met again is filled in with the value once it is resolved.
It recorded them by $ref alone, so the same $ref written for a path item
and for one of its responses handed the path item to the response, which
panicked with "interface conversion: interface {} is *openapi3.PathItem,
not *openapi3.Response". They are now recorded by kind of object and $ref.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
…t points to

A Path Item Object's $ref is no Reference Object: the fields written next
to it belong to the same path item, and every version of the specification
leaves the result undefined only "in case a Path Item Object field appears
both in the defined object and the referenced object".

Any such field made the loader skip the $ref, silently dropping all the
operations of the referenced path item, and letting a $ref to nothing go
unnoticed. Both are now merged; a field set on both sides is taken from
next to the $ref, as a Reference Object's summary and description are,
which is what OpenAPI 3.1 says this $ref is likely to be aligned with.

Marshaling writes back the $ref with the fields that were next to it,
instead of the $ref alone.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
…ma $ref

Running the test cases of getkin#1271 against the previous commits shows four
ways the keywords written next to an OpenAPI 3.1 schema's $ref still go
wrong:

* InternalizeRefs made a component of a $ref's Value, which is a copy of
  the referenced schema with the siblings applied, so they leaked into
  every other use of that component. It now uses the referenced schema,
  which SchemaRef keeps for this. And the external $refs within the
  siblings were rewritten in memory but marshaled as they were first read,
  so the internalized document still pointed outside. Marshaling now takes
  each sibling from what the parsed siblings now hold, and falls back to
  what was read for the fields a Schema leaves out, such as pattern: "".

* A file of bare schemas has no openapi field, so the siblings of the
  $refs within it were ignored as in OpenAPI 3.0, even when an OpenAPI 3.1
  document referred to it. Such a file now follows the version of the
  document whose refs the loader resolves.

* Siblings that do not form a schema, such as allOf: {type: integer},
  were silently dropped. OpenAPI 3.1 reads them as a Schema Object, so
  loading now fails there. OpenAPI 3.0 still ignores them.

* The allOf entry made of the assertion siblings had no Origin, so the
  errors it raised, such as for an invalid pattern, could not be located.
  It now shares the $ref's.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
These are the test cases of getkin#1271, which makes the keywords next to an
OpenAPI 3.1 schema's $ref the Value and the referenced schema an allOf
entry of it. Here Value keeps the shape of the referenced schema, so the
assertions on where each keyword ends up now check what the schema
accepts, or the shape it has here, instead:

* Fields next to a $ref round-trip as written, where getkin#1271 writes them
  out as an ordinary Schema would and so drops their zero values.
* In OpenAPI 3.0, where these fields are ignored, the loader does not
  follow the $refs within them, so InternalizeRefs leaves them as written.
  getkin#1271 internalizes them.
* A cyclic Go value is no longer validated: it overflows the stack
  against any recursive schema, siblings or not, and decoding JSON
  cannot produce one.

The four cases of $refs that only lead to one another are skipped. Their
Value is nil, with or without siblings, as on master: JSON Schema
2020-12 leaves such infinite recursion undefined, and getkin#1271 resolves them
to the conjunction of their siblings.

Adapted by Pierre Fenoll from getkin#1271

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Found by seeding FuzzToV3 with the APIs-guru corpus.
…AML path

Seeds:
- Walk testdata trees rather than globbing their top level, so specs in
  origin/, issue*/, relativeDocs/ and the like seed the fuzzers too:
  FuzzLoadFromData goes from 51 seeds to 161.
- Add the APIs-guru corpus documents under 16 KiB (about 280 OpenAPI 3
  and 550 Swagger 2) when the TestV{2,3}ApisGuruOpenapiDirectory tests
  have downloaded it, outside -short. FuzzLoadFromData reaches 449.
- Add YAML-specific seeds: anchors and aliases, merge keys, tags, block
  scalars, YAML 1.1 implicit resolution, flow style and CRLF.

Targets:
- FuzzLoadWithOrigins: asking for origins changes nothing about the
  document, and every recorded position lies inside the input.
- FuzzYAMLMatchesJSON: a JSON document decodes the same through the
  YAML path, and any document survives being re-encoded as YAML.
- FuzzUnmarshalFromData: an openapi2 document loads back to itself from
  its own JSON and YAML forms.

internal/fuzzing holds what targets in several packages share: finding
seeds, and telling which documents the YAML emitter cannot write back.
The fuzz workflow runs the new targets and restores the corpus from the
go workflow's cache, fetching it when that is missing.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…#1370)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
fenollp and others added 14 commits November 1, 2026 02:02
The registered body decoders never panic and return only a primitive,
[]any or map[string]any.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither router panics, and when both match a request they agree on the
route and on its path parameters, compared decoded: gorillamux leaves
every value escaped, legacy only "%2F".

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
setup-go keys its module and build cache on go.sum. Run before the
checkout it found none, warned "Dependencies file is not found" and
cached nothing, so every job downloaded every module again.

Also move actions/cache from v3, which actionlint reports as too old to
run, to v4.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: connabyss <connabyss.code@proton.me>
)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…to (getkin#1400)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@reuvenharrison
reuvenharrison force-pushed the spike/native-yaml-stock-goyaml branch from 5d30fc3 to 686176e Compare November 5, 2026 20:26
@reuvenharrison
reuvenharrison force-pushed the spike/native-yaml-stock-goyaml branch from 7b73f31 to f57c238 Compare November 7, 2026 09:45
reuvenharrison and others added 3 commits November 7, 2026 11:50
The loader read a YAML document by converting it to map[string]any,
re-serialising that to JSON text and parsing it again, because these
types implement UnmarshalJSON and not UnmarshalYAML. Positions cannot
survive that trip, so they were smuggled through as synthetic __origin__
nodes and reapplied by a reflection walk afterwards.

The UnmarshalYAML methods are already here. This points the loader at
them: a document is parsed once, the types are built from its nodes, and
the origins come off those same nodes. The __origin__ consumer layer
goes with the round trip, and the loader decodes against go-yaml v4.

internal/yamlconv keeps the conversion to JSON for the types that still
reach a document that way. A column is a byte count in v4 where the fork
counted characters, so internal/nodeindex measures a line the same way.

Types.UnmarshalYAML now takes a *yaml.Node, the shape every other type
here already uses. That is a breaking change for a caller that invokes
it directly rather than through the yaml package.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
openapi2 reads a document by converting YAML to JSON and decoding that,
since its types implement UnmarshalJSON and not UnmarshalYAML. That is
what internal/yamlconv does, on the stock parser, so the last use of the
fork goes and both fork modules go with it.

The conversion retags what the fork handled behind a DecodeOpts: a
date-shaped scalar, since a JSON value has no timestamp, and a mapping
key, since a member is named by the text that spells the key.

openapi2.UnmarshalFromData exports the decode. The v3 side has Loader
for this and openapi2 had no entry point of its own.

The goldens move because a decode error is now the json package's own,
without the wrapper the fork put in front of it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestConcurrentDecodeDoesNotLeakOrigins loads the same document from
several goroutines and asserts each one gets its own origins. The state
it covers is per-decode, so a leak between decodes shows up as a data
race before it shows up as a wrong position.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@reuvenharrison
reuvenharrison force-pushed the spike/native-yaml-stock-goyaml branch from f57c238 to 8e02f32 Compare November 7, 2026 09:52
@reuvenharrison

Copy link
Copy Markdown
Author

Superseded by getkin#1238, merged and released in v0.153.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.