Bug fixes and address integration test flakes

[gagan16k]

/kind flake

What this PR does / why we need it:
This PR addresses some bugs in MCM surfaced by integration tests on kwok:

1. Machine stuck in Terminating phase
   • Files changed: machine.go
   • Root Cause: Some code paths (eg. delNodeVolAttachments()) do not return a error when returning. This, paired with changes in PR Trigger node deletion when machine is deleted #1102 cause it to not reconcile this machine again in the event there are no machine updates.
2. AutoPreserveFailedMachineMax reduction tests result in deletion of both machines
   • Instances: Run 1, Run 2, Run 3
   • Files changed: machineset.go, machineset_test.go
   • Observed Behaviour: When AutoPreserveFailedMachineMax was reduced (2 -> 1), the controller unpreserved both machines instead of one, and both got deleted.
   • Root Cause: Two reconciles run: the first un-preserved one machine, updated Status.AutoPreserveFailedMachineCount and wrote it to the API server but it hadn't yet been persisted in the lister cache . The second still read the stale count (2), thought it needed to remove one more, and unpreserved the second machine too.
   • Fix: Count the currently auto-preserved machines from their annotations instead of the stale Status.AutoPreserveFailedMachineCount. A second reconcile then sees the reduced count and does nothing. The oldest machine is un-preserved first. Added unit tests in machineset_test.go to cover these cases.

Other changes to address a set of flakes in the existing integration tests.

1. Reducing MCD replicas to 2 in Manual Preservation tests (from 3 in auto-preservation tests) terminated a machine that got picked up by a test spec.
   • Instances: Run
   • Files changed: framework.go
   • Fix: Keep the replica count consistent throughout the test body.
2. BeforeEach node readiness check stalls on a preserved node
   • Files changed: nodes.go, kwok-config.yaml
   • Root Cause: The BeforeEachwaits until every node is Ready (GetNumberOfReadyNodes == GetNumberOfNodes). Kwok sometimes does not update the leases until a node update event forces it to do so, and the kwok/fail-condition":"Recover" does this. The Get->Update way can sometimes not persist due to conflict errors. Nodes recovering from an earlier spec go through the kwok node-recover stage, which kept them NotReady for a long time. The next spec hangs waiting for nodes to be ready.
   • Fix: Add isNodePreserved (checks for the NodePreservedTaintKey taint) and exclude preserved nodes from both GetNumberOfReadyNodes and GetNumberOfNodes, in case kwok took a very long time to update (even if it fails to do so, this machine will be replaced). Tighten the kwok node-recover stage (immediateNextStage: true) so recovering nodes come back faster. Also switched addNodeRecoverAnnotation to a strategic-merge Patch to avoid update conflicts while the node is recovering.
3. Preservation specs capture a recovering or just-recovered machine as their target
   • Files changed: machine_resources.go
   • Root Cause: Sometimes, a machine used by a previous spec and still running DeferCleanup gets picked up. This could have a preserve=false annotation and can get deleted as a consequence.
   • Fix: Sort GetRunningMachineList by LastUpdateTime ascending, so specs pick the machine that has been running the longest instead of a recently churned one. The tightened kwok node-recover stage also shrinks this window.
4. Freeze/Unfreeze test captures more events than assertion requires when run against a real cluster multiple times.
   • Files changed: framework.go
   • Instances: Run
   • Root Cause: Across multiple runs, the machineset name is the same and the events can persist in the cluster across runs.
   • Fix: Track events by machineset UID instead of name

Other changes/improvements:
- Added more detailed logging in framework.go which now tells exactly which machine a test is being executed on.
- Make target to easily export kubeconfig of a simulated cluster
- Requeue preserved machines at their PreserveExpiryTime to avoid them potentially waiting a full reconcile (10mins)

Release note:

Fix bugs and integration test flakes

[gardener-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign takoverflow for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment