Skip to content

Feat/aws ebs performance troubleshooting - #168

Open
Rania-Souid wants to merge 3 commits into
aws:mainfrom
Rania-Souid:feat/aws-ebs-performance-troubleshooting
Open

Rania-Souid wants to merge 3 commits into
aws:mainfrom
Rania-Souid:feat/aws-ebs-performance-troubleshooting

Conversation

@Rania-Souid

Copy link
Copy Markdown

Before you open this PR — read this. Every required box below must be checked, with the work actually done, before a maintainer will review. A PR with unchecked required items is placed on hold and is not reviewed until it is complete. Checking a box you haven't actually done will be caught at review and sends the PR back. Eval evidence is gated separately by the validate-skill-evals check. The per-entry-type sections apply by contribution type — complete every section that matches what this PR adds or changes.

Description

Adds aws-ebs-performance-troubleshooting, a read-only skill that finds out whether Amazon EBS is why disk I/O on an EC2 instance is slow or stuck, and why: a volume at its own IOPS, throughput or burst limit, the instance at its EBS bandwidth limit, a stalled or impaired volume, I/O disabled, snapshot initialization, or a change made in the account (including AWS FIS). Without the skill, the agent often misses AWS FIS actions in CloudTrail, blames an AWS-side fault or the workload, and suggests detach, stop/start or a support case.

Closes #146

Type of change

  • New skill
  • New custom agent
  • New MCP server
  • Update to an existing skill, agent, or MCP server
  • Documentation or infrastructure change

Required before review

  • Repository layout is correct — skills in /skills, MCP servers in /mcp, custom agents in /custom-agents; skills and MCP servers are never mixed in one directory.
  • A domain SME has reviewed and signed off. A named subject-matter expert for this domain — someone other than the author — has reviewed the tool for technical correctness and approved it in this PR. Required for any change to what the tool tells the agent; an edit that changes no technical claim can be marked n/a with the reason, as CONTRIBUTING.md describes.
  • Evaluation evidence is included (see Testing).
  • No internal-only or proprietary content — no internal hostnames, account IDs, customer names, ticket numbers, internal URLs, or secrets in the tool, its examples, or its tests.
  • README is complete — prerequisites, setup/deployment, and at least one sample prompt or usage example.
  • CHANGELOG updated and the version bumped.
  • Named owner for ongoing maintenance.

Testing

  • structure checks pass.
  • best-practices run across multiple iterations, with consistent results.
  • functional run with and without the change, so the difference in the agent's behavior is visible.
  • evals/evals.json includes positive cases (should trigger) and at least one negative case (should not trigger on unrelated input).

Skill evaluation tool results (judge us.anthropic.claude-sonnet-5):

  • structure: 12/12 passed
  • best-practices: 17/17 in 3 of 3 iterations, 100% consistency
  • functional, 3 iterations with and without the skill, live scenario in a test account:
Scenario Expected output (with / without) Assertions (with / without)
AWS FIS pause I/O on a volume (evals/functional/v1) 2/2 / 1/3 85% / 43%
AWS FIS I/O latency on a volume (v2) 3/3 / 0/3 67% / 14%
gp3 volume at its provisioned IOPS (v3) 2/3 / 1/3 90% / 29%

Notes:

  • The tool has no per-eval filter, and each scenario must be live while it runs, so each version folder holds one eval.
  • One with-skill iteration in v1 ended with "Response ended prematurely" (service side).
  • The other evals in evals.json (gp2 burst credits, instance EBS limit, snapshot initialization, one investigation, two negatives) are defined and validated; their results will follow in a later commit.
  • Known gap: the agent sometimes still ends with an offer to change the resource despite the skill's read-only rule (1 of 3 runs in v3).
  • Eval outputs are redacted: account IDs, IAM principal IDs and user names replaced with placeholders.

Manual testing: 6 scenarios (gp2 burst, instance EBS limit, FIS pause, FIS latency, gp3 IOPS limit, snapshot initialization) in two Agent Spaces, with and without the skill, chat and investigation.

Skills — complete if this PR adds or changes a skill

  • SKILL.md frontmatter is valid: name matches the directory, description present and within length limits, body at most 500 lines.
  • The description is written from the agent's point of view — scenarios, services, error types, symptoms.
  • Scoped to a specific agent type in metadata.aws-devops-agent-skills.agent-types (INCIDENT_RCA, INCIDENT_TRIAGE, INCIDENT_MITIGATION, …), not left GENERIC without reason.
  • Procedures are structured as decision trees on real conditions, not flat prose.
  • Reference material lives in references/ or assets/ and loads on demand.
  • Non-executable content only — no scripts or code in the skill directory. Deterministic compute belongs in an MCP server.

MCP servers — complete if this PR adds or changes an MCP server

  • Transport is Streamable HTTP; auth is documented (SigV4 / OAuth / API key).
  • Every tool is classified read-only vs mutative in the README, with mutative tools called out explicitly.
  • Tools are fixed and named — the server does not execute arbitrary caller-supplied commands, SQL, or queries. Any query surface is allowlisted and bounded (time range, result size, timeout).
  • Least-privilege IAM — the execution role grants only the actions the tools need; the scoping is stated in the README and the policy/template is included.
  • No standing secrets in code or config — credentials come from Secrets Manager or equivalent.
  • Target resources are validated against an allowlist before any call.
  • Tools return bounded summaries, not unbounded raw data.
  • Application security review is complete, in progress, or linked, with a threat model. An MCP server that can act in a customer account does not merge without it.
  • template.yaml (SAM) or cdk.* (CDK), src/, and tests/ are present and tests pass.

Custom agents — complete if this PR adds or changes a custom agent

  • SYSTEM_PROMPT.md is present and is what gets pasted into the agent configuration.
  • The composed tools and skills are listed in the README, and each is already in the catalog or included in this PR.
  • Any mutative tool the agent can call is identified, with its approval behavior described.
  • Run mode (on-demand vs scheduled) and the artifact it produces are documented.

Security and safety

  • Read-only by default; any action that changes customer state is explicit, documented, and approval-gated.
  • Customer data stays in the customer's account.
  • No destructive operations without an explicit, documented confirmation step.

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[request]: Skill to diagnose EBS performance problems on EC2

1 participant