feat: add ProgressiveEval and rule to convert from SortPreservingMerge

[adamreeve]

Which issue does this PR close?

Related to #15191

Rationale for this change

Optimizes sorted reads by avoiding a merge when the inputs can simple be concatenated.

What changes are included in this PR?

• Add a new ProgressiveEvalExec plan, based on feat: Add ProgressiveEval operator #10490, but with some modifications.
• Add a new ReorderPartitionsExec plan, to allow reordering partitions when required to maintain a global order.
• Add a new SequenceSortedInputs optimizer rule, that does the conversion from SortPreservingMergeExec to ProgressiveEvalExec when possible.

Are these changes tested?

Yes, new tests have been added.

Are there any user-facing changes?

Yes. This adds to the public API and adds new configuration options, but the new optimization isn't enabled by default and there are no breaking changes.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-common v55.0.0 (current)
       Built [  37.220s] (current)
     Parsing datafusion-common v55.0.0 (current)
      Parsed [   0.065s] (current)
    Building datafusion-common v55.0.0 (baseline)
       Built [  32.557s] (baseline)
     Parsing datafusion-common v55.0.0 (baseline)
      Parsed [   0.066s] (baseline)
    Checking datafusion-common v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.988s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ExecutionOptions.progressive_eval_num_prefetch_input_streams in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:808
  field OptimizerOptions.sequence_sorted_inputs in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1475

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  72.277s] datafusion-common
    Building datafusion-physical-optimizer v55.0.0 (current)
       Built [  38.853s] (current)
     Parsing datafusion-physical-optimizer v55.0.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-physical-optimizer v55.0.0 (baseline)
       Built [  38.851s] (baseline)
     Parsing datafusion-physical-optimizer v55.0.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-physical-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.149s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  78.939s] datafusion-physical-optimizer
    Building datafusion-physical-plan v55.0.0 (current)
       Built [  36.733s] (current)
     Parsing datafusion-physical-plan v55.0.0 (current)
      Parsed [   0.157s] (current)
    Building datafusion-physical-plan v55.0.0 (baseline)
       Built [  36.676s] (baseline)
     Parsing datafusion-physical-plan v55.0.0 (baseline)
      Parsed [   0.154s] (baseline)
    Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.032s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  76.258s] datafusion-physical-plan
    Building datafusion-sqllogictest v55.0.0 (current)
       Built [  97.307s] (current)
     Parsing datafusion-sqllogictest v55.0.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v55.0.0 (baseline)
       Built [  96.850s] (baseline)
     Parsing datafusion-sqllogictest v55.0.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.115s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 197.345s] datafusion-sqllogictest