Repository navigation
EC2 fleet-request limit stops every EKS node group in the test account #38
Description
🤖 Generated by the Agentic Engineer
Evidence
- Three KSail end-to-end EKS runs in a row failed while creating the worker node group (2026-10-06 16:39 and 22:15 UTC, 2026-10-07 00:19 UTC). The control plane was created each time; the worker machines never joined.
- The third run (37551351240) printed the reason from the node-group stack:
AsgInstanceLaunchFailures— "You've reached your quota for maximum Fleet Requests for this account. Launching EC2 instance failed." Regionus-east-1. - Two runs of the same test from a branch succeeded on 2026-10-05, so the account could launch instances the day before. Something changed between those days: either the limit, or what is using it up.
- Each failed run removed its own cluster and stacks, and the last one read back that no cluster remained.
Audience and problem
Anyone relying on KSail's EKS support being tested. No EKS node group can be created in this account right now, so the end-to-end test cannot pass whatever KSail does, and each attempt costs about 50 minutes of control-plane time for no result. Anything else in the account that launches instances through an Auto Scaling group may be affected the same way.
What is not known yet
- Whether the account's limit is very low (new or restricted accounts start low) or whether leftover fleet or Spot requests are using it up.
- Whether the suite's own CI cluster in this account is affected.
Smallest useful change
Read the account's current EC2 fleet-request usage and limit in us-east-1, remove any leftover requests that belong to nothing, and if the limit itself is the cause, request an increase. Record what was found here.
Acceptance criteria
- The cause is recorded: limit too low, or usage left behind by something named.
- One KSail end-to-end EKS run creates its worker node group.
- If leftovers were the cause, what created them is named and tracked. (Not applicable: nothing was left behind.)
Resolution (2026-10-13)
- Cause: the account's applied limit for running On-Demand Standard instances was 0 vCPUs in every region that was read. Nothing left behind was using capacity: each report found no fleets, Spot requests, instances or scaling groups.
- Change: AWS Support raised the limit for
eu-central-1at the maintainer's request. The read-only report shows 16 vCPUs there (run 38400429101).us-east-1is still 0 (run 38401187144). - Proof: end-to-end run 38400764426 from KSail
maincreated the control plane and a worker node group with one Ready node ineu-central-1. Its cluster was removed afterwards and the account was read back empty (run 38409322992).
EKS tests must name eu-central-1 until the workflows' default region changes (devantler-tech/ksail#7719).
Activity
chore: agent-claim #38 nonce=11e8fee7678eb7917d20ee5c19407c5c
devantler commentedon Oct 7, 2026
First step is in review: devantler-tech/ksail#7592 adds a manual, read-only check that reports what is using launch capacity in this account (fleets, Spot requests, machines and scaling groups by state, plus the limits when the test role may read them), without creating a cluster. Once it merges, one run of it should settle the first acceptance point here: limit too low, or usage left behind. No EKS test run before that.
ci(eks): report what is using EC2 launch capacity without creating a …
devantler commentedon Oct 7, 2026
First reading of the account, 2026-10-07 05:43 UTC — the read-only report from devantler-tech/ksail#7592, run once from
main: run 37577740092 (success, regionus-east-1, no cluster created).What this settles. Nothing is holding launch capacity in this region: there are no leftover fleets, Spot requests, machines or scaling groups to remove. The "leftover requests are using it up" explanation is ruled out for the moment of this reading.
What it does not settle. The limits themselves could not be read, because the test role is not allowed to read service quotas. So this is consistent with the limit being very low or zero, but does not show the number. It is also one reading, about five hours after the last failed run, of one region.
Next steps, in order:
No further EKS smoke-test run is started until the limit is known, since each one costs about 50 minutes of control-plane time and cannot pass.
chore: agent-claim #38 nonce=8c326774eb2ac4c7db40cbc292a524a8
devantler commentedon Oct 7, 2026
Step 1 is in review: devantler-tech/platform#4623. The EKS test identity's permissions are owned by the platform repository, not this one, so the change is there (tracked as devantler-tech/platform#4622). It adds one read-only permission: listing the account's service limits. Nothing else about the identity changes.
Once it is merged and deployed, one run of the existing capacity report gives the fleet-request limit, and the value goes here. No EKS test run before that.
devantler commentedon Oct 7, 2026
Current test budget: no more than five Standard On-Demand vCPUs, with one EKS trial at a time. The current KSail smoke test uses one managed node group of
t3.mediumworkers and exercises 1 → 2 → 1 nodes. Each worker has 2 vCPUs, so the peak is 4 vCPUs, provided unrelated usage leaves that capacity available. The earlier statement here that this test required 8 or 16 vCPUs was incorrect. A larger quota is not a prerequisite for this fixture.Fresh observation, 2026-10-09 20:52 UTC: capacity report 37989724483 completed from KSail
mainat5d6163a2974e3ba31bd0cda94d6ed4a0579aecf8, without creating a cluster.us-east-1This is a successful quota read, not an unreadable value interpreted as zero. The European observations below extend the regional survey. An EKS cluster is not launched against an observed zero quota.
The separate upgrade trial in devantler-tech/ksail#7483 upgrades only the control plane and therefore does not request additional worker nodes. A future managed-node version upgrade needs its own peak-capacity assessment. Its existing dependency on devantler-tech/ksail#7273 and real-provider validation remain in place.
AWS measures this quota in vCPUs per account per Region, and the console's applied quota can differ from the published default of 5. References: AWS quota units and applied-value instructions, T3 worker vCPU counts.
European survey, 2026-10-09: the same read-only workflow ran sequentially in all eight European Regions at KSail revision
5d6163a2974e3ba31bd0cda94d6ed4a0579aecf8.eu-north-1eu-central-1eu-west-1eu-west-2eu-west-3eu-south-1eu-south-2eu-central-2All five successful reports also read zero Standard Spot quota and no EC2 fleets, Spot requests, non-terminated instances or Auto Scaling groups. The three failed reports never reached the quota read, so they are unknown, not zero. No AWS resources were created. A usable five-vCPU quota has not been found in the European Regions queried with KSail's test identity. The account and applied quota visible in the owner's console still need to be reconciled with these observations; AWS's published default of five is not an applied-account reading.
devantler commentedon Oct 13, 2026
Applied limit read live, 2026-10-13. The maintainer reported today that AWS Support approved the EC2 limit increase for Frankfurt. That is now confirmed by the read-only capacity report, run from KSail
mainat44cf4a43bf6ab09a42b3cb762714152c374fb332. Neither run created anything.eu-central-1us-east-1What this means. EKS tests can run in
eu-central-1. They still cannot run inus-east-1, which is the default region of both EKS workflows, so every EKS run has to nameeu-central-1until that default is changed.One trial is running now: the EKS smoke test from
mainineu-central-1, run 38400764426. It uses one control plane and one node group that goes 1 → 2 → 1 workers (4 vCPUs at its peak), is bounded by the workflow's own time limit, and removes its cluster when it ends. Please start no other EKS run until this one has finished and its teardown has been read back here.This issue stays open until that run shows a worker node group being created, which is its second acceptance point.
devantler commentedon Oct 13, 2026
Closing: all three acceptance points are met. The trial announced above has finished.
mainat44cf4a43bf6ab09a42b3cb762714152c374fb332,eu-central-1, 12:54 to 13:31 UTC.ksail cluster createcould not find the kubeconfig entry for it (ksail cluster create fails after the EKS cluster is ready: no kubeconfig context matches ksail#7716, fix in review as fix(eks): select the kubeconfig context eksctl actually writes ksail#7718). The scaling and GitOps steps did not run.Two things to know when starting the next test: only
eu-central-1has capacity (us-east-1, the workflows' default, is still at 0; devantler-tech/ksail#7719), and one more trial is running now, 38409548492. Please start no other EKS run until it has finished.