xpkg build is non-deterministic: package.yaml key order varies per process, breaking multi-platform packages

[kaessert]

What happened?

crossplane xpkg build is not deterministic. Building the same, unmodified package directory several times produces package.yaml files that differ in mapping key order — same byte length, different content, different digest.

The instability is not global: it only affects mappings whose key set happens to break the comparator that sigs.k8s.io/yaml sorts keys with (details below). Most packages never notice. When a package does contain such a key set, the resulting base layer digest changes on every build.

This breaks multi-platform packages. The standard build/makelib/xpkg.mk flow builds one .xpkg per platform in its own process and then composes an index from them:

build.artifacts.platform: do.build.xpkgs        # one `crossplane xpkg build` per platform
xpkg.release.publish: crossplane xpkg push --package-files .../linux_amd64/x.xpkg \
                                            --package-files .../linux_arm64/x.xpkg  ...

Because the two builds are separate processes, they disagree, and the pushed index carries two different io.crossplane.xpkg: base layers for the same package. The Upbound registry rejects such packages outright:

status: rejected
reason: base layer content is inconsistent across images

Five consecutive releases of a provider were silently rejected this way before we traced it. Nothing in the build, the push, or the CLI output signals a problem — the failure only shows up in the registry afterwards.

How can we reproduce it?

Self-contained, ~40 lines. No provider runtime image needed.

package/crossplane.yaml:

apiVersion: meta.pkg.crossplane.io/v1
kind: Configuration
metadata:
  name: repro
spec:
  crossplane:
    version: ">=v1.14.0"

package/crds/xwidgets.yaml:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xwidgets.example.org
spec:
  group: example.org
  names:
    kind: XWidget
    plural: xwidgets
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              interval:
                type: object
                properties:
                  interval12Hours: {type: object}
                  interval15Mins: {type: object}
                  interval1Day: {type: object}
                  interval1Hour: {type: object}
                  interval1Min: {type: object}
                  interval30Mins: {type: object}
                  interval30Secs: {type: object}
                  interval5Mins: {type: object}
                  interval6Hours: {type: object}

Then:

for i in 1 2 3 4 5; do
  crossplane xpkg build --package-root=package -o /tmp/repro$i.xpkg
done
# extract package.yaml from each .xpkg and hash it

Result — five builds of identical input, four distinct package.yaml hashes (all 1428 bytes):

1 1a5ee444a8f15499
2 a5708ff6a57cdb87
3 1a5ee444a8f15499
4 8c4d5b2b333aab44
5 73dc09b744fb3f27

diff between two of them shows only interval* keys moving.

The same reproducer fails identically with up xpkg build, which shares the code path.

Root cause

crossplane-runtime/pkg/xpkg/build.go → encode() re-serializes every package object through the apimachinery JSON serializer in YAML mode:

do := json.NewSerializerWithOptions(json.DefaultMetaFactory, objScheme, objScheme,
    json.SerializerOptions{Yaml: true})

In YAML mode that serializer does json.Marshal(obj) followed by sigs.k8s.io/yaml.JSONToYAML(...) (k8s.io/apimachinery/pkg/runtime/serializer/json/json.go, doEncode). JSONToYAML decodes the JSON into map[string]interface{} and re-marshals it with gopkg.in/yaml.v2.

yaml.v2 sorts mapping keys with keyList.Less, a "natural sort" that treats a run of digits as a number. That comparator is not a strict weak ordering. For the key set above:

interval12Hours < interval1Day  == true
interval1Day    < interval5Mins == true
interval12Hours < interval5Mins == false

sort.Sort over a non-transitive Less returns a result that depends on the input permutation — and the input permutation here comes from iterating a Go map, which is randomized per process. Same input, same key set, different output ordering each run.

Marshalling just those nine keys 200 times through sigs.k8s.io/yaml yields 56 distinct orderings. Zero-padding the digit runs (interval01Day, interval05Mins, …) makes every digit run the same width, restores a total order, and yields exactly 1.

Note this is inherent to the key names, not to package size or complexity: scanning every sibling-key set in two real shipped packages gave 4/575 unstable for the affected provider and 0/427 for provider-aws-ec2 v2.4.0. Large, long-standing packages are unaffected purely by luck of naming.

Suggested fix

Preserve the JSON document's key order instead of round-tripping through an unordered map. encoding/json already sorts map keys byte-lexicographically (a real total order) and emits struct fields in declaration order, so the JSON produced in step 1 is already deterministic — it is only the map[string]interface{} round-trip in JSONToYAML that discards and re-derives that order.

Converting JSON → YAML through an order-preserving representation (e.g. decoding into a yaml.v3 Node tree, or json.Decoder with UseNumber into an ordered structure) would make xpkg build byte-reproducible for all inputs, not just the lucky ones.

A cheaper interim mitigation, if the encoder is hard to change: have crossplane xpkg push verify that the base layer is byte-identical across the --package-files it is given, and fail loudly rather than publishing an index that registries will reject.

What environment did it happen in?

• Crossplane CLI version: v2.0.2 (also reproduced on crossplane/cli@main)
• crossplane-runtime: v2.4.0
• Also affects up xpkg build (upbound/up), same pkg/xpkg code path

Related

• xpkg build: GoTemplate function input kind field buried after large block scalar due to JSON key reordering #344 — different symptom (field order within RawExtension inputs), same encode() re-serialization in crossplane-runtime/pkg/xpkg/build.go.