Repository navigation
Native YAML decoding on the stock parser (WIP) - #26
reuvenharrison wants to merge 78 commits into
Conversation
openapi2 is untouched, and is a possible follow-upThis migrates Dropping those dependencies entirely needs Also still on the wrapper: Deliberately out of scope here. Worth doing as its own change, where it can be reviewed against openapi2's own behaviour rather than buried in this one. |
…y version The Reference Object of OpenAPI 3.0, 3.1 and 3.2 alike says "any properties added SHALL be ignored". From 3.1 on only summary and description mean something there, and the loader already applies them; anything else was still reported as extra sibling fields, failing documents the specification calls valid. So validateExtras now only reports the x- extensions next to a $ref, under ProhibitExtensionsWithRef. That option was disabled for OpenAPI 3.1 schemas, only because their sibling keywords were reported together with those extensions: it now applies in every version, as documented. ExtraSiblingFieldsError is still returned for the unknown fields of inline objects. validateExtras also reported the fields next to a $ref once the $ref was gone: InternalizeRefs replaces a component that is a $ref by what it refers to, which leaves those fields out of the document, but not out of what Validate checked. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
… schema In JSON Schema 2020-12, on which OpenAPI 3.1 bases its Schema Object, the keywords next to a $ref apply alongside it: an instance must satisfy both the referenced schema and its siblings. They were instead copied over the referenced schema's keywords of the same name, so a sibling maxLength: 10 relaxed a referenced maxLength: 3, a sibling required list dropped the referenced one, and sibling properties replaced the referenced ones. Annotations, which describe rather than constrain, still override the referenced ones. Every other sibling keyword now goes into one more allOf entry of the copy the loader makes, so Value keeps the shape of the referenced schema. getkin#1271 reaches the same validation results by making the siblings the Value and the referenced schema an allOf entry, which leaves Value without the type or properties of the referenced schema whenever a $ref has even just a description next to it. The $refs within the siblings are now resolved before they are applied, since applyRefSiblings also runs from the callback that ends a cycle. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
They were dropped. An OpenAPI 3.1 schema's sibling keywords, which constrain the referenced schema, were lost that way, and so was every other field next to a $ref: a document did not survive a round trip. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
The loader records the $refs it is in the middle of resolving, to close
cycles: a $ref met again is filled in with the value once it is resolved.
It recorded them by $ref alone, so the same $ref written for a path item
and for one of its responses handed the path item to the response, which
panicked with "interface conversion: interface {} is *openapi3.PathItem,
not *openapi3.Response". They are now recorded by kind of object and $ref.
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
…t points to A Path Item Object's $ref is no Reference Object: the fields written next to it belong to the same path item, and every version of the specification leaves the result undefined only "in case a Path Item Object field appears both in the defined object and the referenced object". Any such field made the loader skip the $ref, silently dropping all the operations of the referenced path item, and letting a $ref to nothing go unnoticed. Both are now merged; a field set on both sides is taken from next to the $ref, as a Reference Object's summary and description are, which is what OpenAPI 3.1 says this $ref is likely to be aligned with. Marshaling writes back the $ref with the fields that were next to it, instead of the $ref alone. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
…ma $ref Running the test cases of getkin#1271 against the previous commits shows four ways the keywords written next to an OpenAPI 3.1 schema's $ref still go wrong: * InternalizeRefs made a component of a $ref's Value, which is a copy of the referenced schema with the siblings applied, so they leaked into every other use of that component. It now uses the referenced schema, which SchemaRef keeps for this. And the external $refs within the siblings were rewritten in memory but marshaled as they were first read, so the internalized document still pointed outside. Marshaling now takes each sibling from what the parsed siblings now hold, and falls back to what was read for the fields a Schema leaves out, such as pattern: "". * A file of bare schemas has no openapi field, so the siblings of the $refs within it were ignored as in OpenAPI 3.0, even when an OpenAPI 3.1 document referred to it. Such a file now follows the version of the document whose refs the loader resolves. * Siblings that do not form a schema, such as allOf: {type: integer}, were silently dropped. OpenAPI 3.1 reads them as a Schema Object, so loading now fails there. OpenAPI 3.0 still ignores them. * The allOf entry made of the assertion siblings had no Origin, so the errors it raised, such as for an invalid pattern, could not be located. It now shares the $ref's. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
These are the test cases of getkin#1271, which makes the keywords next to an OpenAPI 3.1 schema's $ref the Value and the referenced schema an allOf entry of it. Here Value keeps the shape of the referenced schema, so the assertions on where each keyword ends up now check what the schema accepts, or the shape it has here, instead: * Fields next to a $ref round-trip as written, where getkin#1271 writes them out as an ordinary Schema would and so drops their zero values. * In OpenAPI 3.0, where these fields are ignored, the loader does not follow the $refs within them, so InternalizeRefs leaves them as written. getkin#1271 internalizes them. * A cyclic Go value is no longer validated: it overflows the stack against any recursive schema, siblings or not, and decoding JSON cannot produce one. The four cases of $refs that only lead to one another are skipped. Their Value is nil, with or without siblings, as on master: JSON Schema 2020-12 leaves such infinite recursion undefined, and getkin#1271 resolves them to the conjunction of their siblings. Adapted by Pierre Fenoll from getkin#1271 Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
Found by seeding FuzzToV3 with the APIs-guru corpus.
…AML path
Seeds:
- Walk testdata trees rather than globbing their top level, so specs in
origin/, issue*/, relativeDocs/ and the like seed the fuzzers too:
FuzzLoadFromData goes from 51 seeds to 161.
- Add the APIs-guru corpus documents under 16 KiB (about 280 OpenAPI 3
and 550 Swagger 2) when the TestV{2,3}ApisGuruOpenapiDirectory tests
have downloaded it, outside -short. FuzzLoadFromData reaches 449.
- Add YAML-specific seeds: anchors and aliases, merge keys, tags, block
scalars, YAML 1.1 implicit resolution, flow style and CRLF.
Targets:
- FuzzLoadWithOrigins: asking for origins changes nothing about the
document, and every recorded position lies inside the input.
- FuzzYAMLMatchesJSON: a JSON document decodes the same through the
YAML path, and any document survives being re-encoded as YAML.
- FuzzUnmarshalFromData: an openapi2 document loads back to itself from
its own JSON and YAML forms.
internal/fuzzing holds what targets in several packages share: finding
seeds, and telling which documents the YAML emitter cannot write back.
The fuzz workflow runs the new targets and restores the corpus from the
go workflow's cache, fetching it when that is missing.
…eless const/enum schemas Closes getkin#1367
…btests and assert errors
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…#1370) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…idator.Middleware (getkin#1389)
…kin#1405) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The registered body decoders never panic and return only a primitive, []any or map[string]any. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither router panics, and when both match a request they agree on the route and on its path parameters, compared decoded: gorillamux leaves every value escaped, legacy only "%2F". Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
setup-go keys its module and build cache on go.sum. Run before the checkout it found none, warned "Dependencies file is not found" and cached nothing, so every job downloaded every module again. Also move actions/cache from v3, which actionlint reports as too old to run, to v4. Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: connabyss <connabyss.code@proton.me>
…#1413) Signed-off-by: Aryan <aryangupta050903@gmail.com>
…to (getkin#1400) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
5d30fc3 to
686176e
Compare
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
…1430) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…1431) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ap both errors (getkin#1432) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
7b73f31 to
f57c238
Compare
The loader read a YAML document by converting it to map[string]any, re-serialising that to JSON text and parsing it again, because these types implement UnmarshalJSON and not UnmarshalYAML. Positions cannot survive that trip, so they were smuggled through as synthetic __origin__ nodes and reapplied by a reflection walk afterwards. The UnmarshalYAML methods are already here. This points the loader at them: a document is parsed once, the types are built from its nodes, and the origins come off those same nodes. The __origin__ consumer layer goes with the round trip, and the loader decodes against go-yaml v4. internal/yamlconv keeps the conversion to JSON for the types that still reach a document that way. A column is a byte count in v4 where the fork counted characters, so internal/nodeindex measures a line the same way. Types.UnmarshalYAML now takes a *yaml.Node, the shape every other type here already uses. That is a breaking change for a caller that invokes it directly rather than through the yaml package. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
openapi2 reads a document by converting YAML to JSON and decoding that, since its types implement UnmarshalJSON and not UnmarshalYAML. That is what internal/yamlconv does, on the stock parser, so the last use of the fork goes and both fork modules go with it. The conversion retags what the fork handled behind a DecodeOpts: a date-shaped scalar, since a JSON value has no timestamp, and a mapping key, since a member is named by the text that spells the key. openapi2.UnmarshalFromData exports the decode. The v3 side has Loader for this and openapi2 had no entry point of its own. The goldens move because a decode error is now the json package's own, without the wrapper the fork put in front of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestConcurrentDecodeDoesNotLeakOrigins loads the same document from several goroutines and asserts each one gets its own origins. The state it covers is per-decode, so a leak between decodes shows up as a data race before it shows up as a wrong position. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f57c238 to
8e02f32
Compare
|
Superseded by getkin#1238, merged and released in v0.153.0. |
Draft, not close to merge. Opening it so the work is reviewable and the open problems are visible. Status is stated honestly below; nothing here is finished.
What this is
kin decodes a YAML document by converting it to
map[string]any, re-serialising to JSON text and parsing it again, because these types implementUnmarshalJSONand notUnmarshalYAML. Positions cannot survive that, so they are smuggled through as synthetic__origin__nodes and reapplied afterwards by a reflection walk.This replaces that with
UnmarshalYAMLon every type, decoding straight from the node, against unmodifiedgo.yaml.in/yaml/v3— no fork, no patches.Why it is worth doing
Measured on real types with their real hooks, decoding 400 responses:
json.Unmarshal+ hooksUnmarshalYAML2.54x faster than today, and faster than the JSON fast path, because the hooks re-serialise:
Responses.UnmarshalJSONre-marshals every child back to JSON and re-parses it once per entry, andSchemaRef.UnmarshalJSONparses the same bytes up to four times.It also removes ~495 lines of origin transport across three repos, since a node already carries
Line/Column.Shape of the change
The 41 types fall into four groups, which is what makes this tractable:
Schema.UnmarshalJSONis 91 lines, ~60 of them — which silently misfiles a field added to the struct and forgotten in the list. Here it comes off the struct tags and cannot drift.$refwrappers, one helper plus thin methods.Headerdefers toParameter;Types,BoolSchemaandExclusiveBoundare union-typed scalars.No end positions are used. A block's extent is derivable from start positions — the next key or sequence item at the same or shallower indentation — which agrees with recorded ends on ~99.98% of ~11.9M spans. That is what removes the need for a patched parser.
Open problems
1. YAML 1.1 implicit resolution. The old path normalised every scalar through JSON's type system on the way through the round trip. Decoding natively keeps YAML 1.1 semantics, so implicit resolution now applies:
examplebecame atime.Timeand failed validation — fixed here by retagging before decode, reproducing theDisableTimestampsoption our yaml fork has and stock go-yaml does not18_24loads it as the integer1824Timestamps were one instance of a family. The family needs a considered answer, not another special case. This is the largest open item.
2. 24 failures in
openapi3. 13 are tests assertingEndLine/EndColumn, which is the deliberate design change. The other 11 are real: the arbitrary-top-level-key$refpath, nil-versus-empty on maplike collections (an absentresponsesno longer fails validation), a changed error message, and origin coverage gaps.3. Duplicate keys. JSON resolves a repeated key last-one-wins; YAML rejects it. The json path is kept as a fallback so such documents still load, without origins.
4.
originFileVaris package-level.UnmarshalYAMLreceives a node and nothing else, so the file name cannot be threaded through. This follows the precedent ofIncludeOriginand inherits its concurrency characteristics. Making both per-Loaderis worth doing and is a separate change.Validation so far
testdata(18) decodes identically to the JSON path.Key.Line/Column,FieldsandSequences, compared againstapplyOriginson the same documents.Two bugs the suite caught that would otherwise have shipped: origins were produced when the caller had not asked for them (the gate read the package global rather than the flag the
Loaderpasses, and sinceNewLoaderseeds itself from that global, test pollution made it look correct); and a parameter inside a sequence had noKey, because stamping walked mappings only.