Repository navigation
CI/CD: Add container memory, OOM kill and restart alarms and reduce noisy alarms - #3588
Conversation
|
After adding the docker metric service to the EC2 template it was discovered that sudo unlink /etc/systemd/system/cfn-hup.service
sudo tee /etc/systemd/system/cfn-hup.service > /dev/null <<'EOF'
[Unit]
Description=CloudFormation helper daemon
After=network.target
[Service]
Type=simple
ExecStart=/opt/aws/bin/cfn-hup --no-daemon --verbose
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
sudo chmod 0644 /etc/systemd/system/cfn-hup.service
sudo systemctl daemon-reload
sudo systemctl enable --now cfn-hup |
|
Version installed: Broken symlink Steps to fix: |
|
@denniskuijs good work finding the root cause, let's go with the remediation steps you describe as there's no need to manually write the service file as in my comment. These steps need to be manually performed on existing instances which would need coordinating. The adjustments in this PR are still relevant I believe and might need to rename the PR |
|
@richturner I've reverted the Please let me know if this PR is now ready for approval. |
…d restarts docker stats MemPerc includes active page cache and shared memory, so the PostgreSQL container, whose shared_buffers and cached data files fill its limit by design, kept the alarm above 85% while healthy. Read anonymous memory from each container's cgroup instead and publish it per container (ContainerMemoryPercent) as well as the instance maximum (MaxContainerMemoryPercent) used by the existing alarm, which now needs 5 minutes above the threshold. Add ContainerOomKills and ContainerRestarts, published as the number seen since the previous run, with alarms on any occurrence. An OOM kill of a container's main process (e.g. a JVM) restarts the container and resets its cgroup counters and OOMKilled flag, so it is only visible as a restart.
|
@denniskuijs Added more alarms and some refactoring of existing ones and a description for this PR. I have left the cfn-hup conf rename in place as it aligns with the AWS doc naming convention and makes more sense |
…tAlarm Several alarms fired on a single 5-minute datapoint for conditions that are normal when brief: - DeadTupleAlarm: dead tuples build up between autovacuum runs, so require 12 periods (1 hour) above 30% instead of one period above 20%, which matched the default autovacuum trigger and fired before every vacuum - LongRunningQueriesAlarm: 3 periods, so the nightly export and compression jobs don't trip it with a single slow query - PostgreSQLConnectionsAlarm: 3 periods, to ignore short connection bursts - AttributeEventAlarm: band width 3 and 2 of 3 periods, so ordinary traffic changes don't trip it; missing data still breaches, so a stopped manager is still reported TableBloatCountAlarm is removed: pg_table_bloat_count counts tables over 5 MB with dead tuples above 30% of live tuples, which is the same signal DeadTupleAlarm raises per table. The metric is still collected for the dashboard.
|
Verified this updated template on ****sten prod |
|
@richturner Looks good! I've indeed renamed the |
|



Description
Adds Docker container memory monitoring to
cloudformation-create-ec2.yml, so a container approaching its memory limit, or being killed for exceeding it, raises an alarm. It also tunes several existing alarms so they fire on sustained problems rather than brief, normal spikes.A
docker-metricssystemd timer runs/usr/local/bin/push-docker-metricsevery minute (only when metrics are enabled) and publishes to theCustom/Dockernamespace:ContainerMemoryPercentInstanceName,ContainerMaxContainerMemoryPercentInstanceNameContainerMemoryPercenton the instanceContainerOomKillsInstanceNameContainerRestartsInstanceNameAlarms:
ContainerMemoryUtilizationAlarm:MaxContainerMemoryPercentabove 85% for 5 minutes.ContainerMemoryPercentshows which container.ContainerOomKillAlarm: any OOM kill inside a container, e.g. a PostgreSQL backend.ContainerRestartAlarm: any container restart, e.g. a crash, or a JVM whose main process was OOM killed.Why not
docker statsdocker statsMemPerc includes active page cache and shared memory. PostgreSQL relies on both (shared_buffersplus the OS cache of its data files, whicheffective_cache_sizeassumes), so a healthy PostgreSQL container sits close to its limit and kept adocker statsbased alarm above 85%. That memory is reclaimable and doesn't lead to an OOM kill. The script readsanonfrom each container's cgroup v2memory.statinstead. For the JVM containers memory is almost entirely anonymous, so their values are unchanged.OOM kills and restarts are reported as counts since the previous run, using counters stored in
/run/docker-metrics/counters. An OOM kill of a container's main process restarts the container, which recreates its cgroup (resettingoom_kill) and clears Docker'sOOMKilledflag. That case is only visible as a restart, hence the separate restart metric. Containers without a memory limit are skipped for the memory percentages.Alarm tuning
Several existing alarms fired on a single 5-minute datapoint for conditions that are normal when brief. They now need the condition to last:
DeadTupleAlarmLongRunningQueriesAlarmPostgreSQLConnectionsAlarmAttributeEventAlarmTableBloatCountAlarmremoved.pg_table_bloat_countcounts tables over 5 MB whose dead tuples exceed 30% of live tuples. That is the same conditionDeadTupleAlarmchecks per table frompg_dead_tuple_percent, so the two fired together. The bloat count also flapped as dead tuples crossed 30% between vacuum runs.DeadTupleAlarmis the better alarm of the two because it names the table. The metric is still collected for the dashboard.DeadTupleAlarmis designed to work with #3599, which makespg_dead_tuple_percentreport 0 for tables with fewer than 10,000 dead tuples. Without it, small tables such asdashboardcan still sit above 30% for an hour.Other changes
daily-cleanupcron file: removed the#!/bin/bashline; cron files are not scripts, so it was only a stray commentdeployment_local_dir:mkdir -p, so re-runningcfn-initdoesn't fail when the directory exists/etc/cfn/hooks.d/cfn-auto-reloader.conf, as it reloads all ofAWS::CloudFormation::Init, not only the CloudWatch agentTesting
The template parses (alarm settings checked after the tuning changes), and the rendered script was run against real
docker ps/docker inspectoutput with a cgroup v2 tree laid out as on AL2023. It produces validput-metric-dataJSON with the expected percentages, skips containers without a limit, and reports OOM kill and restart deltas correctly, including a container that was really OOM killed and restarted, and anoom_killcounter reset. It hasn't yet been run on an EC2 instance.Checklist