[SPARK-58966][SQL] Resolve SQL variables in UPDATE and MERGE INTO conditions

[joelrobin18]

What changes were proposed in this pull request?

SQL variables declared with DECLARE cannot be referenced in the conditions of an
UPDATE or MERGE INTO statement. This passes includeLastResort = true at the seven
condition-resolution sites that make up those clauses, matching what SPARK-57260 did for
OverwriteByExpression.deleteExpr.

Variable resolution only runs from resolveColsLastResort, which is reached when
resolveExpressionByPlanOutput / resolveExpressionByPlanChildren are called with
includeLastResort = true. Both default the flag to false. Plans with no dedicated
resolution rule fall through to the generic operator case in ResolveReferences, which
does pass the flag -- which is why DELETE ... WHERE already works. UPDATE and
MERGE INTO each have a dedicated rule that omitted it:

• UPDATE condition (ResolveReferencesInUpdate)
• MERGE ON condition
• MERGE WHEN MATCHED DELETE / UPDATE conditions
• MERGE UPDATE * condition
• MERGE WHEN NOT MATCHED BY SOURCE DELETE / UPDATE conditions

Assignment values (SET col = var, INSERT VALUES (var)) are affected by the same root
cause, but they resolve through resolveExprInAssignment, which sets
includeLastResort = false explicitly rather than by default. That is left unchanged
here pending a decision on whether the explicit false was deliberate, so this PR is
scoped to conditions only.

Why are the changes needed?

Variables resolve in SELECT, INSERT (including REPLACE WHERE) and DELETE, but in
no condition of UPDATE or MERGE INTO, which fails analysis with:

[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with
name `target_dep` cannot be resolved. SQLSTATE: 42703

The message itself offers "A column, variable, or function parameter", so the analyzer
reports that variable resolution was attempted. Nothing in error-conditions.json, the
tests, or the docs records a restriction here. Where Spark does intentionally block
variables (SPARK-57360, generated columns) it uses an explicit validation and a dedicated
error class, so the inconsistency looks like an oversight rather than a deliberate
limitation.

This affects both session variables and SQL scripting local variables.

Does this PR introduce any user-facing change?

Yes, a bug fix.

Before, against a table using the built-in DSv2 row-level operation framework:

DECLARE OR REPLACE VARIABLE target_dep STRING DEFAULT 'hr';
UPDATE cat.ns1.test_table SET salary = 0 WHERE dep = target_dep;
-- [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... `target_dep` cannot be resolved. SQLSTATE: 42703

After, the statement analyzes and executes, resolving target_dep to the declared
variable. Statements that previously succeeded are unaffected: the flag only enables a
last-resort resolution step that runs after normal column resolution fails, so column
references continue to take precedence over same-named variables.

How was this patch tested?

Four new tests, plus a shared withSessionVariable helper in
RowLevelOperationSuiteBase:

• UpdateTableSuiteBase -- a session variable and a SQL scripting local variable in an
  UPDATE condition.
• MergeIntoTableSuiteBase -- a variable in the MERGE ON condition, and in the
  WHEN MATCHED / WHEN NOT MATCHED BY SOURCE conditions.

All four were verified to fail on master without the fix (each raising
UNRESOLVED_COLUMN) and to pass with it.

Suites run: GroupBasedUpdateTableSuite + GroupBasedMergeIntoTableSuite (137 tests)
and DeltaBasedUpdateTableSuite + DeltaBasedMergeIntoTableSuite (148 tests), all
passing.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

[cloud-fan]

0 blocking, 1 non-blocking, 0 nits.
The shared fallback approach is sound, but the MERGE condition enumeration is incomplete.

Design / architecture (1)

• Non-blocking: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1899: Apply last-resort variable resolution to both WHEN NOT MATCHED insert-condition branches and cover the form with a focused test. -- see inline

Verification

I traced every MergeAction condition through ResolveReferences and the downstream merge rewrite. Matched and not-matched-by-source actions now opt into last-resort resolution, while InsertAction and InsertStarAction conditions still use the column-only resolver. Those insert conditions are consumed as normal MERGE predicates, and an unresolved variable reaches CheckAnalysis as UNRESOLVED_COLUMN.

PR metadata suggestions

• Correct the condition-coverage claim, preferably by including and testing WHEN NOT MATCHED AND <variable>; both insert-condition branches still omit last-resort variable resolution.

[cloud-fan]

Non-blocking:

WHEN NOT MATCHED conditions need the same last-resort resolution. The InsertAction and InsertStarAction branches at lines 1852 and 1860 still call resolveExpressionByPlanOutput without includeLastResort = true, so a declared variable remains unresolved even though the other MERGE condition forms now work. Please update both branches and add a focused WHEN NOT MATCHED AND <variable> test.