Local models that use constrained generation (MLX, Core ML, and llama.cpp) always generate the first variant of an anyOf schema, because generateNode(_:) in StructuredGeneration.swift takes variants[0]. So a nullable union like anyOf: [person, .null] (#288) can never produce null, and a union of object types always produces the first one.
The model should choose the branch. One approach is to mask the next token to the start tokens of every variant, as itemStartTokens(for:) already does for array items, and continue with the variant whose start tokens include the sampled token. Variants that start the same way, like two object types, would need to look ahead to the first key.
Local models that use constrained generation (MLX, Core ML, and llama.cpp) always generate the first variant of an
anyOfschema, becausegenerateNode(_:)inStructuredGeneration.swifttakesvariants[0]. So a nullable union likeanyOf: [person, .null](#288) can never producenull, and a union of object types always produces the first one.The model should choose the branch. One approach is to mask the next token to the start tokens of every variant, as
itemStartTokens(for:)already does for array items, and continue with the variant whose start tokens include the sampled token. Variants that start the same way, like two object types, would need to look ahead to the first key.