[SPARK-58917][SQL] Respect inferSchema for variant ingestion in CSV parser

[pratham76]

What changes were proposed in this pull request?

This PR adds a new CSV read option variantRespectInferSchema that controls type inference behavior for variant ingestion in the CSV parser. When enabled along with inferSchema=false, scalar CSV values are preserved as strings inside Variants instead of being automatically inferred to numeric or boolean types.

Why are the changes needed?

Currently, the CSV to Variant parser (used by the singleVariantColumn option and explicit VariantType columns) always infers scalar types (long, decimal, date, timestamp, boolean) regardless of the inferSchema option. This means a value like "0001" is stored as the integer 1 rather than the string "0001", which may not be the desired behavior in all cases.

Users need a way to preserve the original string representation of CSV values when ingesting into Variants, particularly for:

• Leading zeros in numeric-looking strings (e.g., "0001", "00123")
• Data that should remain as strings for semantic reasons (e.g., zip codes, product codes)
• Preserving exact input format for downstream processing

Does this PR introduce any user-facing change?

Yes. This PR introduces a new CSV read option, variantRespectInferSchema (default: false)

• When false (default): Preserves existing behavior - types are always inferred for variant values
• When true and inferSchema=false: CSV values are preserved as strings in Variants
• When true and inferSchema=true: Types are still inferred (respects the inferSchema setting)

How was this patch tested?

Added comprehensive tests in CSVSuite, and added tests in CsvFunctionsSuite for the from_csv function

Was this patch authored or co-authored using generative AI tooling?

No

[ghanse]

@pratham76 I have already introduced this in #58175