Expand correlated subquery testing for $lookup stage

[danielfrankcom]

This change adds additional testing for correlated $lookup, focusing on let variable value forms and the scoping of $$ROOT/$$CURRENT within the sub-pipeline.

This is related to #673, in particular this comment thread.

Changes based on @PatersonProjects'.

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: compatibility test, enhancement
Project fields suggested: Component test-coverage · Priority P3 · Effort M · Status Needs Review
Confidence: 0.82 (mixed)

Reasoning

component from path globs (test-coverage); effort from diff stats (334+0 LOC, 2 files); LLM: Adds additional test cases for correlated $lookup scenarios (let variables, $$ROOT/$$CURRENT scoping) with no production code changes.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

[yshanhu]

With the purpose of checking let passing through for all BSON types:

Can we please:
1.add more types:
e.g.

  # Add these 9 to the existing 7 types:
  "v_int64": Int64(100),
  "v_decimal128": Decimal128("123.45"),
  "v_objectid": ObjectId("507f1f77bcf86cd799439011"),
  "v_date": datetime(2024, 1, 1, tzinfo=timezone.utc),
  "v_binary": Binary(b"test_data", 0),
  "v_timestamp": Timestamp(1234567890, 1),
  "v_regex": Regex("^test", "i"),
  "v_minkey": MinKey(),
  "v_maxkey": MaxKey(),

2.modify this query shape to also check if type preserved, see example here

db.local.aggregate([
      {
          $lookup: {
              from: "foreign",
              let: {
                  vi: "$v_int",
                  ...
              },
              pipeline: [
                  {
                      $addFields: {
                          // Values
                          ri: "$$vi",
                         ...
                          // Types
                          type_i: { $type: "$$vi" },
                      }
                  }
              ],
              as: "joined"
          }
      }
  ]);

Note:
If we have not already, in the verbose match stage specific test, lets add a test with all BSON type again to test equality matching, not just passing through

[danielfrankcom]

I'll refer to the additional note at the bottom as "3." just for clarity:

1. Done in 3b9a2d5. The let passthrough test now covers all BSON types as variables, including the ones you listed.

2. The type is already enforced by the test framework, just at the assertion layer rather than in the query shape. The comparison runs through _strict_equal, which rejects cross-type numerics. I don't think we need an additional $type field here, as it would just be testing the type server side instead of client side.

3. I don't think this belongs in the $lookup test files. FOLDER_STRUCTURE.md treats $lookup as a container and says to keep sub-feature testing minimal:

   Container features ($expr, $match, $lookup): under the container's directory, only test that sub-features work inside it, one test case per sub-feature, no edge cases. Edge cases belong in each sub-feature's own directory.
   $lookup/ → 1-2 cases per pipeline sub-stage

   Equality matching across BSON types is edge-case coverage of $eq, and it already lives in comparisons/eq/. Re-running the full matrix here would duplicate that, and would also be inconsistent with the way we have handled this for other operators that combine with $eq.

[yshanhu]

Noticing all the let form tests were in let+pipeline syntax for correlated, correct me if anything I miss checking:

Can we add at least one test for:
1/concise correlated? (localField+foreignField + let + pipeline), we can do just one test but with all possible forms of let (constant/expressions/field reference/system variable) in this type of syntax;
2/add one test where let is not provided, since it is optional ->expect to function as uncorrelated.

[danielfrankcom]

Both of these are already covered, just in sibling files that aren't changed in this PR.

The concise correlated form (localField + foreignField + let + pipeline) is tested in test_lookup_concise_correlated_subquery.py, including a case that binds a let variable alongside the equality match and reads it in the sub-pipeline. I don't think we need to test the full form matrix (constant, expression, field reference, system variable) for every syntax, it lives once in test_lookup_correlated_subquery.py and seems like it falls under the repo guidance of not retesting behaviors in different contexts.

The "let is not provided" case is covered in test_lookup_uncorrelated_subquery.py, which has a test that covers "pipeline and no let runs the sub-pipeline independently." That file also shows let: null and let: {} behave the same as omitting it.

[yshanhu]

Can we also add error cases here, where invalid let is expected to be rejected?

Examples:

1.Starts with $     {"$var": "$a"}
2.Contains dot .    {"a.b": "$a"}
3.Starts with uppercase letter       {"Var": "$a"}
4.Contains special characters       {"a@b": "$a"},{"a-b": "$a"},{"a#b": "$a"}
5.Starts with number                     {"1abc": "$a"}
6.Empty string                                {"": "$a"}
7.Starts with underscore                {"_var": "$a"}
8.Contains space                            {"a b": "$a"}

Just use a simple let + pipeline (in the subpipeline, use a simple: $match with $expr with $eq) will be sufficient here. Thanks!

[danielfrankcom]

These are already covered in test_lookup_let_errors.py. The only one not covered is the a@b and a#b examples, but I think that's fine since those are just special characters which as a class is already covered by the existing cases like a$b and a-b. The aim of the existing tests is to cover each rule once rather than the full character set.

Worth noting this follows the pattern we already use for the other let consumers, with test_let_variable_names.py for the $let operator and test_aggregate_let_naming.py for the aggregate command.

[yshanhu]

regarding let scoping/shadowing, can we add these cases:

1.merge scope (in nested lookup case, let say 3 levels in total, the inner most child lookup should be able to access the outermost parent AND its parent's let variables):

e.g.
  db.level1.aggregate([
      {
          $lookup: {
              from: "level2",
              let: {outerVar: "$outerVal", level1Id: "$_id"},
              pipeline: [
                  {
                      $lookup: {
                          from: "level3",
                          let: {middleVar: "$middleVal", level2Id: "$_id"},
                          pipeline: [
                              {
                                  $project: {
                                      desc: 1,
                                      fromLevel1_outer: "$$outerVar",
                                      fromLevel1_id: "$$level1Id",
                                      fromLevel2_middle: "$$middleVar",
                                      fromLevel2_id: "$$level2Id"
                                  }
                              }
                          ],
                          as: "level3_results"
                      }
                  }
              ],
              as: "level2_results"
          }
      }
  ]);

2.Add a case: Middle level is UNCORRELATED (no let, no variable accessing), innermost still accesses outermost->to make sure even uncorrelated lookup pass let scope to its child

  db.level1.aggregate([
      {
          $lookup: {
              from: "level2",
              let: {outerVar: "$outerVal", level1Id: "$_id"},
              pipeline: [
                  {
                      $lookup: {
                          from: "level3",
                          // NO let - uncorrelated at this level
                          pipeline: [
                              {
                                  $lookup: {
                                      from: "level4",
                                      let: {thirdVar: "$thirdVal", level3Id: "$_id"},
                                      pipeline: [
                                          {
                                              $project: {
                                                  desc: 1,
                                                  // Access Level 1 (through uncorrelated Level 2)
                                                  fromLevel1: "$$outerVar",
                                                  fromLevel1Id: "$$level1Id",
                                                  // Access Level 3 (current parent)
                                                  fromLevel3: "$$thirdVar",
                                                  fromLevel3Id: "$$level3Id"
                                              }
                                          }
                                      ],
                                      as: "level4_results"
                                  }
                              }
                          ],
                          as: "level3_results"
                      }
                  }
              ],
              as: "level2_results"
          }
      }
  ]);

The query shapes look long, but I hope this helps explain the idea