Skip to content

fix(iac): purchase executor budget (4m + 30s + 30s) equals lambda_timeout (300s), leaving no headroom #1434

Open

Description

@IAmJSD

Leftover from #1195 (one of its "smaller items"). #1245 fixed the main gap and closed #1195, but not this.

What happens

A purchase run on AWS Lambda can use exactly the whole Lambda timeout, with nothing to spare:

The 30 s margin that lambda_timeout >= 300 was meant to leave (the validation in terraform/environments/aws/variables.tf:291-300, and TestLambdaTimeoutCoversPurchaseBudget, which checks only purchaseExecutionTimeout + terminalSaveTimeout plus lambdatimeout.Margin) is now used up by the alert. Startup, the recipient lookup and the response are not counted at all.

Impact

When a run uses its full budget and then fails, Lambda can end the invocation in the middle of the failure email. The log marker that drives the alarm is written first, so the alarm still fires, but the email to the operators can be lost. A hung mail server makes this worse (#1433).

Owner decision needed

This is a deploy decision, so it is not claimed by an agent. Options:

  1. Raise lambda_timeout (for example to 360) in the tfvars and in the variable default, and include failureNotifyTimeout in TestLambdaTimeoutCoversPurchaseBudget and in the validation message. Applying this to real environments is a Terraform change a human has to make.
  2. Shrink a budget (for example a 3m30s execution budget, or a shorter alert timeout) so the sum plus margin fits in 300 s. No deploy change is needed, but slow purchases get less time.

Acceptance criteria

  • The budget test counts every detached phase that can follow a purchase: execution, terminal save and failure alert. It requires lambda_timeout to cover their sum plus the margin.
  • The shipped tfvars and the CloudFormation Timeout pass that test.
  • The comment at manager.go:232-236 and the lambda_timeout validation message state the real sum.

Refs #1195, #1245, #913.

Filed by IAmJSD's agent at IAmJSD's request (acting owner while the maintainers are away).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions