[Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT by seanrathier · Pull Request #7637 · elastic/cloudbeat

seanrathier · 2026-08-04T18:17:54Z

Summary

Adds all AWS integration permissions to the federated identity CloudFormation template, enabling zero-creds STS AssumeRole+ExternalID auth across the entire aws package.

Paired integrations PR: elastic/integrations#20527

Part of ingest-dev#8802

Policies added

Resource	Permissions	Integration
`ElasticAwsConfig`	`config:DescribeConfigRules`, `config:GetComplianceDetailsByConfigRule`	config
`ElasticAwsInspector`	`inspector2:ListFindings`	inspector
`ElasticAwsSecurityHub`	`securityhub:GetFindings`, `securityhub:GetInsights`	securityhub
`ElasticAwsCloudwatchLogs`	`logs:DescribeLogGroups`, `logs:FilterLogEvents`	cloudwatch/ec2/elb/lambda logs
`ElasticAwsMetrics`	`cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, `tag:GetResources`, `ec2:DescribeRegions`	all aws/metrics metricsets
`ElasticAwsHealth`	`health:DescribeEvents`, `health:DescribeEventDetails`, `health:DescribeAffectedEntities`	awshealth
`ElasticAwsBilling`	`ce:GetCostAndUsage`	billing
`ElasticAwsDynamoDB`	`dynamodb:ListTables`, `dynamodb:DescribeTable`	dynamodb
`ElasticAwsEBS`	`ec2:DescribeVolumes`	ebs
`ElasticAwsEC2`	`ec2:DescribeInstances`, `ec2:DescribeInstanceStatus`	ec2
`ElasticAwsECS`	`ecs:ListClusters`, `ecs:DescribeClusters`, `ecs:ListServices`, `ecs:DescribeServices`	ecs
`ElasticAwsELB`	`elasticloadbalancing:DescribeLoadBalancers`, `...DescribeTargetGroups`, `...DescribeTargetHealth`	elb
`ElasticAwsLambda`	`lambda:ListFunctions`, `lambda:GetFunction`	lambda
`ElasticAwsRDS`	`rds:DescribeDBInstances`, `rds:DescribeDBClusters`, `rds:ListTagsForResource`	rds
`ElasticAwsSNS`	`sns:ListTopics`, `sns:GetTopicAttributes`	sns
`ElasticAwsSQS`	`sqs:ListQueues`, `sqs:GetQueueAttributes`	sqs
`ElasticAwsTransitGateway`	`ec2:DescribeTransitGateways`, `ec2:DescribeTransitGatewayAttachments`	transitgateway

E2E test plan

Deploy this CFT (or update an existing federated-identity stack)
Validate per integration that no AccessDeniedException appears in agent logs for any listed action
Confirm Fleet shows no DEGRADED status for Config, Inspector, SecurityHub, CloudWatch, and a sampling of metrics data streams
Regression: existing GuardDuty policies using the same role must continue collecting data

Introduces federated-identity-aws.yml — a single CFT with one Enable* boolean parameter per AWS integration. CloudFormation Conditions attach only the IAM policies needed for the selected data streams, keeping the role minimal. Transport-layer policies (S3/SQS, CloudWatch Logs, Metrics) are auto-derived so users never need to think about them. Adds the template to publish_cft.sh so it is uploaded to S3 alongside the existing cloud-connectors templates. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…nally Drops the 24 Enable* parameters and all CloudFormation Conditions in favor of a single static read-only role. Every IAM policy needed by the agentless-enabled AWS policy templates is always attached, so Kibana only needs to pass ElasticResourceId in the quick-create URL and users never have to update the stack when they add integrations later. Policies are grouped into four inline policy resources (transport, metrics, security findings, service inventory) plus the GuardDuty managed policy. Trust policy, ExternalId derivation, and outputs are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Extends the federated identity role to cover CSPM, Cloud Asset Inventory, and KSPM-EKS: attaches the SecurityAudit managed policy and adds an ElasticSecurityPosture inline policy with the supplemental config, organizations, access-analyzer, account, cross-account sts:AssumeRole, and EKS read permissions declared in the per-package IaC patches of elastic/integrations#20240. CNVM is deliberately excluded: its scan operations (snapshot create/delete, RunInstances/TerminateInstances, iam:PassRole) require write access, and this role stays read-only. CNVM keeps its dedicated template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Cross-checked every inline action against the provider_permissions declarations in elastic/integrations#19405 (the AWS package's per-datastream permission manifest) and added the missing actions the agentless data streams require at runtime: - ec2:DescribeInstanceStatus (ec2_metrics) - ecs:DescribeClusters (ecs_metrics) - elasticloadbalancing:DescribeTargetHealth (elb_metrics) - lambda:GetFunction (lambda) - rds:DescribeDBClusters (rds) - health:DescribeAffectedEntities (awshealth) - securityhub:BatchGetSecurityControls, GetInsightResults (securityhub_findings_full_posture, securityhub_insights) securityhub:ListInsights, declared by #19405, is NOT added: cfn-lint confirms no such IAM action exists — listing insights is securityhub:GetInsights, which was already granted. Also corrects the Config grant: the aws.config data stream polls rule compliance (DescribeConfigRules, DescribeComplianceByConfigRule, GetComplianceDetailsByConfigRule), not resource inventory; the inventory-style Config reads moved to the SecurityPosturePolicy with the rest of the Asset Inventory permission set. Actions not declared by #19405 are retained when another primary source documents them (e.g. iam:ListAccountAliases in the package README); services whose policy templates are not agentless-enabled (apigateway, natgateway, vpn, emr, kafka, kinesis, redshift, s3_storage_lens) stay out of scope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Replace the all-at-once grant (five inline policies + SecurityAudit covering every agentless-enabled AWS integration upfront) with an incremental model: the template carries only the permissions of integrations that are actually federated, one block per integration, mirroring the provider_permissions declared in that integration's package manifest in elastic/integrations. Never grant ahead of a declaration. The baseline is GuardDuty only — the single integration federated in production today. Its AmazonGuardDutyReadOnlyAccess grant is carried over verbatim from the shipped cloud-connectors-guardduty template, pre-dating provider_permissions; it converts to a mirrored block when the aws package gains declarations. Per-integration additions land as separate PRs stacked on this one, each paired with the elastic/integrations PR that declares the permissions it mirrors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Adds 15 inline IAM policies to the federated identity template covering all remaining aws package integrations: inspector, securityhub, cloudwatch logs, shared metrics base (cloudwatch:ListMetrics, GetMetricData, tag:GetResources, ec2:DescribeRegions), awshealth, billing, dynamodb, ebs, ec2, ecs, elb, lambda, rds, sns, sqs, and transitgateway. Paired integrations PR: elastic/integrations#20526 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mergify · 2026-08-04T18:24:46Z

This pull request does not have a backport label. Could you fix it @seanrathier? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The Identity Federation quick-create link now targets cloudformation-federated-identity-aws-9.4.0.yml, published by cloudbeat's publish_cft.sh when elastic/cloudbeat#7637 merges. Until that publish runs, this URL 404s — cloudbeat#7637 must merge first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Consolidates the grants previously split across the standalone CSPM and Cloud Asset Inventory remote-role templates into the single combined CFT. SecurityAudit covers both cloud_security_posture and cloud_asset_inventory; the intent is for this template to replace the separate per-product CFTs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

jeniawhite

Similar to PR: elastic/integrations#19405.
I assume that there is some overlap between the roles and the permissions (the roles probably include some of the permissions), but this shouldn't be a problem. Cloudbeat wise it looks fine, but would be great if we could get a review from the integration stakeholders as well.

Co-authored-by: Oleg Sucharevich <oleg.sucharevich@elastic.co>

mergify · 2026-08-06T17:05:50Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

olegsu · 2026-08-06T17:06:04Z

+            Action: sts:AssumeRole
+            Condition:
+              StringEquals:
+                sts:ExternalId: !Join


Should not be part of this IaC

olegsu · 2026-08-06T17:06:35Z

+    Description: The ARN of the IAM Role. Paste this into Kibana.
+    Value: !GetAtt ElasticFederatedIdentityRole.Arn
+
+  ExternalId:


Same here, we plan to build the trust with WII

The agent composes the final external id as <resource id>-<output> (cloudConnectorsExternalID in beats), so the output must be only the stack UUID part, as in the shipped cloud-connectors templates. The previous full <resource id>-<uuid> output caused a double-prefixed external id and AssumeRole AccessDenied for every stream. Verified end-to-end against a live agentless GuardDuty policy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seanrathier · 2026-08-07T16:43:39Z

E2E testing tracker — combined Federated Identity CFT

Environment: ECH 9.5.0-SNAPSHOT (fi-e2e-test), agentless, package aws 7.2.0 from integrations#20527, account elastic-siem (144492464627), stack deployed manually via CLI (template not yet published to S3 — merge-order dependency on this PR).

Stream status

Stream	Region	Status	Notes
guardduty	us-east-1	✅ working E2E	Healthy via the combined CFT + shared FI connector; documents in `logs-aws.guardduty-default`
inspector	us-east-2	✅ working E2E	Healthy via the same shared connector; documents in `logs-aws.inspector-default` (2160h initial window picked up existing findings, no seeding needed)
aws/metrics auth path	us-east-2	✅ validated	Incidentally proven: a policy with awshealth/billing metrics streams completed the full cloud-connectors STS chain (failure was post-auth `UnauthorizedOperation` from a role without metrics permissions — caller was the assumed customer-account role)
config	us-east-2	🚫 blocked	Needs a customer-created Config rule (elastic/security-integrations#838) — the account's conformance-pack rules are service-linked and filtered out by the integration's `CreatedBy` check
aws-cloudwatch logs (4 streams)	us-east-2	🚫 blocked	Needs the heartbeat Lambda from security-integrations#838
aws/metrics data (13 streams)	us-east-2	🚫 blocked	Same — auth path already validated (above), data needs the Lambda

Observations / issues found while testing

ExternalId output contract bug (fixed in d5f4fe6, validated live). The agent composes the final external id as <resource id>-<pasted value> (cloudConnectorsExternalID in beats). The template output was the full joined string, producing a double-prefixed external id and AccessDenied on sts:AssumeRole for every stream. Output now emits the UUID part only, matching the shipped cloud-connectors templates. The final healthy state runs against a stack deployed with this fix.
RoleName length limit (open). ElasticFederatedIdentity-${AWS::StackName} exceeds IAM's 64-char role-name limit for stack names > 39 chars — role creation fails with a validation error users won't easily decode. The shipped GuardDuty template has the same latent issue (37-char budget). Possible fix: name the role from the stack UUID instead.
Failed-auth agents linger. Agentless agents whose connector has bad credentials stay enrolled and degraded indefinitely (Recoverable retry loop); each failed test attempt leaves one behind. Cleanup = delete the package policy.
Diagnosing agentless auth failures. The full STS error is on the agent's components[].units[].message via GET /api/fleet/agents/<id> — useful because the UI enrollment flyout only says "1 or more components/units in a degraded state". Note: denied cross-account AssumeRole attempts are logged in the caller's CloudTrail (Elastic's account), not the customer's — customer-side CloudTrail forensics is a dead end for FI auth failures.
"Collecting but unhealthy" (corrected). An agent whose policy mixes streams can be healthy on one input and failed on another — e.g. GuardDuty httpjson healthy while awshealth/billing metrics units fail Permanent on a role lacking metrics permissions. The agent rolls up as degraded even though data is flowing. Also: newly provisioned agentless agents took 30+ minutes to appear in Fleet → Agents (and are hidden by the list's default filters; the old "Show agentless agents" toggle is gone) — diagnosis required the Fleet API.
Connector External ID is fragile and unfixable once wrong (Kibana/Fleet). A wrong pasted External ID produces an opaque, retried-forever 403. The public API can't rotate it: PUT /api/fleet/cloud_connectors/{id} rejects a plaintext external_id ("must contain valid external_id secret reference"), so the only recovery is creating a new connector and re-pointing policies. Resolved here by POSTing a fresh connector with a programmatically-set value — which worked immediately, proving the stored value (not the CFT) was at fault.
API-created connectors need accountType. POST /api/fleet/cloud_connectors accepts a body without accountType, but the resulting connector never appears in the policy editor's connector selector. Include "accountType": "single-account".
Switching connectors on an existing agentless policy did not recover the agent. Updating an FI-enabled package policy to point at a different (valid) cloud connector took the agentless agent offline and it never re-enrolled or picked up the new credentials — the stale degraded state persisted until the policy was abandoned. Recovery required creating a fresh integration policy with the new connector, which provisioned a new agent that worked immediately.

Note on the ExternalId issues (1, 6): these are transitional. Once cloud connectors migrate to WII, the ExternalId is no longer part of the trust flow at all — the output contract, the paste fragility, and the no-rotation API gap all disappear with it.

seanrathier and others added 6 commits July 21, 2026 16:13

mergify Bot assigned seanrathier Aug 4, 2026

seanrathier mentioned this pull request Aug 4, 2026

[aws] Enable Identity Federation for all remaining policy templates elastic/integrations#20527

Open

9 tasks

Add aws Config permissions to the Federated Identity CFT

08dbf06

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

This was referenced Aug 4, 2026

Add CloudFormation template for Federated Identity (AWS) #7422

Closed

Add aws Config permissions to the Federated Identity template #7589

Closed

seanrathier added backport-skip Team:Security-Cloud Services Security Data Experience - Cloud Services team. labels Aug 4, 2026

seanrathier marked this pull request as ready for review August 4, 2026 18:36

seanrathier requested a review from a team as a code owner August 4, 2026 18:36

Remove ElasticAwsSecurityHub — moving to separate PR (ingest-dev#8812)

33cd03f

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seanrathier mentioned this pull request Aug 5, 2026

Create organization variant of federated-identity-aws CFT and update CSPM/CAI manifest URLs #7658

Open

6 tasks

olegsu reviewed Aug 5, 2026

View reviewed changes

Comment thread deploy/cloudformation/federated-identity-aws.yml Outdated

olegsu requested changes Aug 5, 2026

View reviewed changes

jeniawhite reviewed Aug 5, 2026

View reviewed changes

Update deploy/cloudformation/federated-identity-aws.yml

590a683

Co-authored-by: Oleg Sucharevich <oleg.sucharevich@elastic.co>

seanrathier requested a review from olegsu August 6, 2026 14:17

seanrathier changed the title ~~Add all remaining aws package permissions to the Federated Identity CFT~~ [Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT Aug 6, 2026

olegsu approved these changes Aug 6, 2026

View reviewed changes

olegsu reviewed Aug 6, 2026

View reviewed changes

olegsu requested changes Aug 6, 2026

View reviewed changes

olegsu reviewed Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

[Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT - #7637

[Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT#7637
seanrathier wants to merge 11 commits into
mainfrom
seanrathier/federated-identity-aws-remaining

seanrathier commented Aug 4, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Aug 4, 2026

Uh oh!

Uh oh!

jeniawhite left a comment

Uh oh!

mergify Bot commented Aug 6, 2026

Uh oh!

olegsu Aug 6, 2026

Uh oh!

olegsu Aug 6, 2026

Uh oh!

seanrathier commented Aug 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

seanrathier commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Policies added

E2E test plan

Uh oh!

mergify Bot commented Aug 4, 2026

Uh oh!

Uh oh!

jeniawhite left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 6, 2026

Uh oh!

olegsu Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

olegsu Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

seanrathier commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E testing tracker — combined Federated Identity CFT

Stream status

Observations / issues found while testing

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

seanrathier commented Aug 4, 2026 •

edited

Loading

seanrathier commented Aug 7, 2026 •

edited

Loading