[Test] Stabilize test_multiple_efs and test_queue_parameter_update

[gmarciani]

Description of changes

Stabilize tests:

• test_multiple_efs: Retry EFS mount assertion for IAM-authorized EFS in test_multiple_efs. The IAM auth can introduce not impactful delays in mounting on reboot.
• test_queue_parameter_update: fix flakiness by asserting the TERMINATE requeue deterministically.
  The TERMINATE branch snapshotted the transient CONFIGURING state via a single-shot after 10 seconds. If the comphute node boots quicker than the update, then the job goes running, not configuring. With this fix we assert on what matters, i.e. the job gets requeued then goes running on the replaced node.

Tests

PENDING

test-suites:
  storage:
    # Cluster: head + queue-0 (MinCount 1) + queue-1 (MinCount 1) = 3 static c5.xlarge nodes.
    test_efs.py::test_multiple_efs:
      dimensions:
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_4_INSTANCES_3_HOURS_NOPG_alinux2023 }}]
          instances: ["c5.xlarge"]
          oss: ["alinux2023", "ubuntu2404"]
          schedulers: ["slurm"]
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_4_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["c5.xlarge"]
          oss: ["rhel9"]
          schedulers: ["slurm"]
  update:
    # Cluster: head + queue1 (MinCount 1, MaxCount 2) + queue2 (MinCount 1, MaxCount 2). The resize
    # path can reach 5 c5.xlarge nodes, so the reservation is sized to 5.
    test_update.py::test_queue_parameters_update:
      dimensions:
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_alinux2023 }}]
          instances: ["c5.xlarge"]
          oss: ["alinux2023", "ubuntu2404"]
          schedulers: ["slurm"]
        - regions: [{{ c5_xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["c5.xlarge"]
          oss: ["rhel9"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.94%. Comparing base (abee849) to head (bb92d9d).
⚠️ Report is 3 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #7564   +/-   ##
========================================
  Coverage    89.94%   89.94%           
========================================
  Files          180      180           
  Lines        16255    16255           
========================================
  Hits         14621    14621           
  Misses        1634     1634           

Flag	Coverage Δ
unittests	89.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.