Repository navigation
fix(iac): purchase executor budget (4m + 30s + 30s) equals lambda_timeout (300s), leaving no headroom #1434
Description
Leftover from #1195 (one of its "smaller items"). #1245 fixed the main gap and closed #1195, but not this.
What happens
A purchase run on AWS Lambda can use exactly the whole Lambda timeout, with nothing to spare:
purchaseExecutionTimeout= 4 min andterminalSaveTimeout= 30 s (internal/purchase/manager.go:237-238, main at 6b586cc), which run back to back;- then, for a failed or partial row, the failure alert gets its own detached 30 s (
failureNotifyTimeout,internal/purchase/failure_notify.go:26, added by fix(ops): alert on failed purchases and scheduled tasks; wire monitoring per environment #1094/fix(purchase): alert on failed fan-out roots and every terminal-failure writer #1245); - total: 4 min + 30 s + 30 s = 300 s, and
lambda_timeoutis 300 in every shipped tfvars (terraform/environments/aws/github-{dev,staging,prod}.tfvars).
The 30 s margin that lambda_timeout >= 300 was meant to leave (the validation in terraform/environments/aws/variables.tf:291-300, and TestLambdaTimeoutCoversPurchaseBudget, which checks only purchaseExecutionTimeout + terminalSaveTimeout plus lambdatimeout.Margin) is now used up by the alert. Startup, the recipient lookup and the response are not counted at all.
Impact
When a run uses its full budget and then fails, Lambda can end the invocation in the middle of the failure email. The log marker that drives the alarm is written first, so the alarm still fires, but the email to the operators can be lost. A hung mail server makes this worse (#1433).
Owner decision needed
This is a deploy decision, so it is not claimed by an agent. Options:
- Raise
lambda_timeout(for example to 360) in the tfvars and in the variable default, and includefailureNotifyTimeoutinTestLambdaTimeoutCoversPurchaseBudgetand in the validation message. Applying this to real environments is a Terraform change a human has to make. - Shrink a budget (for example a 3m30s execution budget, or a shorter alert timeout) so the sum plus margin fits in 300 s. No deploy change is needed, but slow purchases get less time.
Acceptance criteria
- The budget test counts every detached phase that can follow a purchase: execution, terminal save and failure alert. It requires
lambda_timeoutto cover their sum plus the margin. - The shipped tfvars and the CloudFormation
Timeoutpass that test. - The comment at
manager.go:232-236and thelambda_timeoutvalidation message state the real sum.
Filed by IAmJSD's agent at IAmJSD's request (acting owner while the maintainers are away).
Activity