All pull requests

Pull requests

• feat: run user-defined generators inside the native generate operator

  area:expressionsenhancementrun-spark-4.1-tests
  #7253·andygrove opened Oct 26, 2026Member·

  ·
  1

• feat: run inline, inline_outer, and stack generators natively

  enhancementrun-spark-4.1-tests
  #7251·andygrove opened Oct 26, 2026Member·

  ·
  1

  4

• docs: describe the long-lived branch for upgrading DataFusion

  documentation
  #7250·andygrove opened Oct 26, 2026Member·

  ·

• deps: upgrade to DataFusion 56.0.0

  area:Deltaarea:expressionsarea:ffiarea:Icebergarea:joinsarea:scanarea:shuffledependenciesrun-all-spark-profilesrun-iceberg-tests
  #7249·andygrove opened Oct 26, 2026Member·

  ·

• fix: run flatten natively on arrays of binary and name the expression in fallback reasons

  area:expressionsbug
  #7248·andygrove opened Oct 26, 2026Member·

  ·

• docs: say that Spark 3.4 and 3.5 build a cache through rows under AQE

  documentation
  #7238·andygrove opened Oct 26, 2026Member·

  ·

  · 1.3.0

• perf: compare string bounds in the Arrow data buffer when gathering cache statistics

  enhancementperformance
  #7237·andygrove opened Oct 26, 2026Member·

  ·

  · 1.3.0

• refactor: move literal conversion and query context lookup out of create_expr

  enhancementrun-spark-4.1-tests
  #7236·andygrove opened Oct 26, 2026Member·

  ·

• perf: compare decimal bounds in the Arrow data buffer when gathering cache statistics

  enhancementperformance
  #7235·andygrove opened Oct 26, 2026Member·

  ·

  · 1.3.0

• perf: remove per-partition tail copy and flush from the native shuffle writer

  area:shuffleenhancementperformance
  #7234·andygrove opened Oct 26, 2026Member·

  ·
  1

  4

• fix: keep rows from imported Iceberg files by not pushing range and IN residuals on name-mapped tables

  area:Icebergarea:scanbugrun-iceberg-tests
  #7233·andygrove opened Oct 26, 2026Member·

  ·

• chore: add branch protection for branch-1.2

  enhancement
  #7232·andygrove opened Oct 26, 2026Member·

  ·

• fix: declare required child distribution on aggregate, window and join operators

  bugrun-spark-4.1-tests
  #7231·andygrove opened Oct 26, 2026Member·

  ·
  1

  2

• Avoid fetching remaining shuffle blocks when closing early

  area:shuffle
  #7229·dajiaohuang opened Oct 26, 2026·

  ·
  1

• fix: name the failed file in native Parquet read errors

  area:ffiarea:scanbug
  #7228·ump45nose opened Oct 26, 2026Contributor·

  ·
  3

• fix: reject Parquet timestamp columns read as non-timestamp types

  area:scanbug
  #7227·ump45nose opened Oct 26, 2026Contributor·

  ·
  1

  3

• feat: support native Iceberg Bloom writes on older runtimes

  area:Icebergarea:writerenhancement
  #7226·NikitaMatskevich opened Oct 25, 2026·

  ·

• fix: write a sliced batch to the in-memory cache without the rows before it

  bug
  #7225·andygrove opened Oct 25, 2026Member·

  ·

  · 1.3.0
  1

  7

• fix: spill a native PartialMerge whose input is sorted on some of its keys

  area:aggregationarea:memorybugrun-spark-4.1-tests
  #7222·andygrove opened Oct 25, 2026Member·

  ·
  1

  21

• fix: fall back to Spark when the driver cannot use the ABFS authentication

  area:scanbug
  #7221·andygrove opened Oct 25, 2026Member·

  ·
  1

  19

• fix: preserve float collect_set semantics in distinct aggregate chains

  area:aggregationbug
  #7220·LinSimon-901101 opened Oct 25, 2026Contributor·

  ·
  1

  3

• fix: convert timestamp_ntz values beyond chrono's year range to timestamp

  area:expressionsbug
  #7219·ninglinLiu opened Oct 25, 2026·

  ·
  1

  5

• fix: throw SparkArithmeticException for native NUMERIC_OUT_OF_SUPPORTED_RANGE errors

  bug
  #7218·ninglinLiu opened Oct 25, 2026·

  ·
  1

  5

• fix: saturate legacy float and double casts to timestamp on overflow

  area:expressionsbug
  #7217·bharadwaj-pendyala opened Oct 25, 2026·

  ·
  1

  3

• fix: resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast

  area:expressionsbug
  #7215·ninglinLiu opened Oct 25, 2026·

  ·
  1

  9