[fix][broker] Keep accepting Avro named type references written as objects

[lhotari]

Motivation

#24617 upgraded Avro to 1.12.x. Avro 1.12.2 changed the schema parser so that a reference to a named type must be written as the bare name (AVRO-4176, apache/avro#3772). Up to Avro 1.12.1 the parser also accepted the reference written as an object whose type is the name, and resolved it to the named type while ignoring any other attribute of the object:

{
  "type": "record", "name": "Drawing", "namespace": "org.example.shapes",
  "fields": [
    {"name": "background", "type": {"type": "enum", "name": "Color", "namespace": "org.example.shapes", "symbols": ["RED", "BLUE"]}},
    {"name": "outline", "type": {"type": "org.example.shapes.Color"}}
  ]
}

With Avro 1.12.2 this fails with:

org.apache.avro.SchemaParseException: A schema "type" MUST be a primitive type or one of "enum", "fixed", "record", "error", "array" or "map".

Such schema definitions were accepted by the broker and stored in the schema registry before the upgrade, and clients that hand-write or generate schemas may still produce them. After the upgrade the broker fails to parse them when validating a schema, when looking up an existing version of a schema, and when checking compatibility, and clients fail to parse them when consuming from a topic whose stored schema uses this form. The upgrade must not start rejecting schemas that were valid before, in the same spirit as #25193 restored the previous name validation.

Modifications

• Add AvroSchemaCompat.normalizeNamedTypeReferences() to pulsar-common, which rewrites {"type": "name"} objects to the bare "name" before a schema definition is handed to the Avro parser, reproducing the pre-1.12.2 behaviour. The rewrite follows the schema structure (the schema itself, field types, array items, map values and union members), so field defaults and other JSON values are never touched, and it returns the input unchanged when there is nothing to rewrite or the input is not JSON.
• Add StructSchemaDataValidator.parseAvroSchema() as the single broker-side way of parsing a schema definition (compatible name validator + the rewrite above) and use it in StructSchemaDataValidator, SchemaRegistryServiceImpl, AvroSchemaBasedCompatibilityCheck and JsonSchemaCompatibilityCheck, which each created their own parser before.
• Apply the rewrite on the client side in SchemaUtil.parseAvroSchema() and GenericJsonRecord, which parse the schema definitions received from the broker.

The stored schema data is not modified; the rewrite only affects what is passed to the parser.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• AvroSchemaCompatTest: the rewrite in every schema position, nested records, a top-level reference, field defaults being left alone, and unchanged/invalid input being returned as is.
• SchemaDataValidatorTest: AVRO and JSON schema data using the legacy form is accepted and parses to the named type.
• BaseAvroSchemaCompatibilityTest (Avro and JSON compatibility checks): the legacy and the bare name form of a schema are fully compatible with each other.
• SchemaUtilTest: the client parses the legacy form to the same schema as the bare name form.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Schema definitions that use the pre-Avro-1.12.2 object form for named type references are accepted again by the broker and the client; nothing changes for other schemas.

Documentation

• doc-not-needed