[shantanutheone]

Which issue does this PR close?

• Closes COPY … STORED AS JSON (and INSERT INTO a JSON table) drops every MAP entry whose value is NULL: {color: red, size: NULL} is written as {"color":"red"} #26493.

Rationale for this change

COPY ... STORED AS JSON and INSERT INTO a JSON table drop every MAP entry whose value is NULL. map(['color','size'], ['red', NULL]) is written as {"color":"red"}, so reading the file back gives a different map: the size key is gone and cardinality drops from 2 to 1. A map with only NULL values, such as map(['flag'], [NULL]), comes back as {}.

The JSON sink uses arrow-json's writer with its default explicit_nulls = false, and arrow-json's MapEncoder skips NULL-valued entries in that mode. For a struct field a missing key still reads back as NULL when the table has a schema, but for a map the entry is the data, so leaving it out loses it.

What changes are included in this PR?

• datafusion/datasource-json/src/file_format.rs: a small ndjson_writer helper that builds the line-delimited writer with with_explicit_nulls(true) when the schema contains a Map at any depth (inside List, LargeList, FixedSizeList, ListView, LargeListView, Struct, Dictionary or RunEndEncoded). JsonSerializer uses it, which covers both COPY and INSERT INTO.
• datafusion/datasource-json/src/source.rs: plan_to_json uses the same helper, so both write paths behave the same.
• An upgrade guide note in 56.0.0.md for the output change.

Schemas without a Map are written exactly as before.

What is the testing strategy for this PR?

New round-trip cases at the end of copy.slt: write tables with NULL-valued MAP entries to JSON, read them back with a declared schema, and check the values and cardinality. One case has the map at the top level, the other has it inside a struct and inside a list, so the nested check is covered. Both cases fail without the source change (the maps read back as {} or lose their NULL keys) and pass with it.

Two small unit tests in file_format.rs cover what SQL round trips cannot see: contains_map finds a Map inside List/Struct, Dictionary and RunEndEncoded, and JsonSerializer writes {"id":null,"m":{"k":null}} when a Map is present but still omits the NULL ({}) when there is none.

Are there any user-facing changes?

Yes, for JSON files whose schema contains a Map. Explicit nulls are a writer-wide setting in arrow-json, so in those files every other NULL (top-level columns and struct fields) is now written as "field":null instead of being omitted. Reading back with a declared schema gives the same values. With schema inference, a column or struct field that is NULL in every sampled row is now inferred as type Null instead of being missing. Files without a Map are unchanged. No public API changes.

Not covered here: unbounded JSON stream tables (datafusion/catalog/src/stream.rs) build their own LineDelimitedWriter, so they still drop these entries, and the CLI's --format json / ndjson output is unchanged. datafusion-catalog does not depend on datasource-json, so sharing the helper means moving it (probably to datafusion-datasource). I can do that in a follow-up.

Open question: whether maintainers would prefer a format.explicit_nulls option on JsonOptions instead of tying this to Map columns.

[shantanutheone]

Opened #26500 for this. It turns on arrow-json's explicit_nulls only when the schema contains a Map (at any depth), so {color: red, size: NULL} is written as {"color":"red","size":null} and reads back with the same keys and cardinality. Since the setting is writer-wide, other NULLs in those files are now written as null too, which I noted in the 56.0.0 upgrade guide.