diff --git a/bench/infra/README.md b/bench/infra/README.md index 04cc100..cbeca4b 100644 --- a/bench/infra/README.md +++ b/bench/infra/README.md @@ -15,7 +15,7 @@ Single one-shot script. Provisions, in order: 1. The default-VPC subnet group covering every default subnet in `us-east-1` (`vortex-bench-subnet-group`). 2. A security group `vortex-bench-sg` with inbound TCP 5432 from `0.0.0.0/0` — IAM auth is the gate, not network ACLs. -3. A DB parameter group `vortex-bench-pg16` (family `postgres16`) that sets `work_mem` to 32 MiB. +3. A DB parameter group `vortex-bench-pg16` (family `postgres16`) that sets `work_mem` to 32 MiB and turns on `log_connections` and `log_disconnections`, and the CloudWatch log group `/aws/rds/instance/vortex-bench-prod/postgresql` with 30-day retention. The instance exports its Postgres log there. See [Logs](#logs). 4. An RDS Postgres instance `vortex-bench-prod` on `db.t4g.micro`, Postgres 16, 20 GiB GP3 storage that autoscales to 100 GiB, IAM auth enabled, RDS-managed master password (auto-rotated, stored in Secrets Manager), publicly accessible, single-AZ, deletion protection, automatic minor upgrades, 35-day backup window. On an instance that already exists, the script sets deletion protection, the storage ceiling, the backup window and automatic minor upgrades; these apply without a restart. It does not move an existing instance to another parameter group or change an existing backup replication, because a parameter group needs a restart and a new replication starts a new off-region copy. It prints the commands instead. 5. Replication of the automated backups (daily snapshots and WAL) to `us-west-2`, with 14 days of point-in-time recovery there. See [Backups](#backups). 6. An SNS topic `vortex-bench-alarms`, three CloudWatch alarms and an RDS event subscription. See [Alarms](#alarms). @@ -225,6 +225,35 @@ recovers. Each alarm needs three 5-minute periods (15 minutes) in breach. The RDS event subscription `vortex-bench-prod-events` forwards `failure` and `low storage` events for `vortex-bench-prod`. +## Logs + +The instance is reachable from the internet, so Postgres logs every connection +and disconnection: who connected, from which address, and whether the login +worked. The log goes to the CloudWatch log group +`/aws/rds/instance/vortex-bench-prod/postgresql` and is kept for 30 days +(`DB_LOG_RETENTION_DAYS`). To see the failed logins of the last day: + +```sh +aws logs filter-log-events --log-group-name /aws/rds/instance/vortex-bench-prod/postgresql \ + --start-time $(( ($(date +%s) - 86400) * 1000 )) --filter-pattern '"authentication failed"' +``` + +## Account security + +These are set for the whole AWS account, not only for bench, so `provision.sh` +does not manage them and a bench tear-down must not remove them. All were added +on 2026-10-28. + +- **CloudTrail:** the trail `account-trail` records management events in every + region to the private bucket `aws-cloudtrail-logs-245040174862`, with log-file + validation. +- **GuardDuty** in us-east-1, with RDS login monitoring, so it reports unusual + logins to `vortex-bench-prod`. S3 data-event monitoring is off, because the + CI cache buckets are busy. Check the cost before the 30-day trial ends + (2026-11-27). +- **IAM Access Analyzer:** `account-external-access` in us-east-1 lists the + resources that are shared outside the account. + ## Upgrades Automatic minor version upgrades are on. RDS applies a new Postgres 16.x @@ -285,6 +314,7 @@ aws rds delete-db-instance --db-instance-identifier vortex-bench-prod \ --final-db-snapshot-identifier vortex-bench-prod-final --delete-automated-backups aws rds wait db-instance-deleted --db-instance-identifier vortex-bench-prod aws rds delete-db-parameter-group --db-parameter-group-name vortex-bench-pg16 +aws logs delete-log-group --log-group-name /aws/rds/instance/vortex-bench-prod/postgresql aws iam delete-role-policy --role-name GitHubBenchmarkSchemaRole --policy-name rds-db-connect-migrator aws iam delete-role --role-name GitHubBenchmarkSchemaRole aws iam delete-role-policy --role-name GitHubBenchmarkIngestRole --policy-name rds-db-connect-ingest @@ -312,6 +342,7 @@ was added in October 2026. NOTE: prod `vortex-bench-prod` was upsized to | Backup storage in us-east-1 beyond the free 20 GiB (35-day window, ~140 GB-month in total) | ~$11 | | Backup replica in us-west-2 (14-day window: ~66 GB-month at $0.095, plus ~120 GB a month of transfer at $0.02) | ~$9, estimate | | CloudWatch alarms (3) | $0.30 | +| CloudWatch Logs (Postgres log, 30 days) | <$1, estimate | | Secrets Manager (1 secret) | ~$0.40 | | Data transfer (CI ingests + reader fetches) | <$1 | | **Total** | **~$196/month** | diff --git a/bench/infra/provision.sh b/bench/infra/provision.sh index 6b71397..74ddc5e 100755 --- a/bench/infra/provision.sh +++ b/bench/infra/provision.sh @@ -32,7 +32,9 @@ # protection, storage autoscaling to 100 GiB, automatic minor upgrades. # 6. Waits for the instance to reach `available`, then applies the same # protection, storage, backup and upgrade settings to an instance that -# already existed. +# already existed. The Postgres log, with every connection, goes to the +# CloudWatch log group `/aws/rds/instance/vortex-bench-prod/postgresql` +# (30-day retention). # 7. Replicates the automated backups (snapshots and WAL) to `us-west-2`, # for 14 days of point-in-time recovery there. # 8. Creates the SNS topic `vortex-bench-alarms`, three CloudWatch alarms @@ -84,6 +86,8 @@ readonly BACKUP_REPLICA_REGION="${BACKUP_REPLICA_REGION:-us-west-2}" readonly BACKUP_REPLICA_RETENTION_DAYS="${BACKUP_REPLICA_RETENTION_DAYS:-14}" readonly ALARM_TOPIC_NAME="${ALARM_TOPIC_NAME:-vortex-bench-alarms}" readonly EVENT_SUBSCRIPTION_NAME="${EVENT_SUBSCRIPTION_NAME:-vortex-bench-prod-events}" +# The Postgres log (with every connection and disconnection) goes to CloudWatch Logs. +readonly DB_LOG_RETENTION_DAYS="${DB_LOG_RETENTION_DAYS:-30}" # The incident.io "Amazon CloudWatch" alert source URL. It carries a token, so # it is read from the environment only and never printed. Unset: no subscription. readonly ALARM_HTTPS_ENDPOINT="${ALARM_HTTPS_ENDPOINT:-}" @@ -270,12 +274,32 @@ ensure_parameter_group() { log " created." fi - # work_mem is in KiB. 32 MiB lets the load-all reads sort in memory. + # work_mem is in KiB. 32 MiB lets the load-all reads sort in memory. The + # instance is reachable from the internet, so log every connection. aws rds modify-db-parameter-group \ --db-parameter-group-name "$DB_PARAMETER_GROUP_NAME" \ --parameters "ParameterName=work_mem,ParameterValue=32768,ApplyMethod=immediate" \ + "ParameterName=log_connections,ParameterValue=1,ApplyMethod=immediate" \ + "ParameterName=log_disconnections,ParameterValue=1,ApplyMethod=immediate" \ >/dev/null - log " work_mem=32768 applied." + log " work_mem=32768, log_connections=1 and log_disconnections=1 applied." +} + +ensure_log_group() { + log "Step 4c: CloudWatch log group for the Postgres log." + + # Made before the instance exports to it, so the retention applies from + # the first event. RDS writes to this exact name. + local group="/aws/rds/instance/${DB_INSTANCE_IDENTIFIER}/postgresql" + if ! aws logs describe-log-groups --log-group-name-prefix "$group" \ + --query 'logGroups[].logGroupName' --output text | tr '\t' '\n' | grep -qxF "$group"; then + aws logs create-log-group --log-group-name "$group" \ + --tags "${TAG_KEY_PROJECT}=${TAG_VAL_PROJECT},${TAG_KEY_OWNER}=${TAG_VAL_OWNER}" + log " created ${group}." + fi + aws logs put-retention-policy --log-group-name "$group" \ + --retention-in-days "$DB_LOG_RETENTION_DAYS" + log " ${DB_LOG_RETENTION_DAYS}-day retention set." } ensure_rds_instance() { @@ -305,6 +329,8 @@ ensure_rds_instance() { --no-multi-az \ --auto-minor-version-upgrade \ --deletion-protection \ + --copy-tags-to-snapshot \ + --enable-cloudwatch-logs-exports postgresql \ --tags "Key=${TAG_KEY_PROJECT},Value=${TAG_VAL_PROJECT}" \ "Key=${TAG_KEY_OWNER},Value=${TAG_VAL_OWNER}" \ >/dev/null @@ -329,7 +355,7 @@ ensure_rds_instance() { fi # Re-assert the settings on an instance that existed before they were added - # here. All four take effect at once without a restart, and the call omits + # here. All of them take effect at once without a restart, and the call omits # --apply-immediately so it cannot pull a queued engine upgrade forward. aws rds modify-db-instance \ --db-instance-identifier "$DB_INSTANCE_IDENTIFIER" \ @@ -337,8 +363,19 @@ ensure_rds_instance() { ${max_storage_args[@]+"${max_storage_args[@]}"} \ --backup-retention-period "$DB_BACKUP_RETENTION_DAYS" \ --auto-minor-version-upgrade \ + --copy-tags-to-snapshot \ >/dev/null - log " deletion protection, storage autoscaling (${allocated} GiB allocated), ${DB_BACKUP_RETENTION_DAYS}-day backups and automatic minor upgrades set." + log " deletion protection, storage autoscaling (${allocated} GiB allocated), ${DB_BACKUP_RETENTION_DAYS}-day backups, automatic minor upgrades and snapshot tags set." + + if ! aws rds describe-db-instances --db-instance-identifier "$DB_INSTANCE_IDENTIFIER" \ + --query 'DBInstances[0].EnabledCloudwatchLogsExports' --output text \ + | tr '\t' '\n' | grep -qxF postgresql; then + aws rds modify-db-instance \ + --db-instance-identifier "$DB_INSTANCE_IDENTIFIER" \ + --cloudwatch-logs-export-configuration '{"EnableLogTypes":["postgresql"]}' \ + >/dev/null + log " Postgres log export to CloudWatch enabled." + fi # A new parameter group takes effect only after a reboot, so the script # reports a different group and leaves the move and the reboot to the operator. @@ -712,6 +749,7 @@ main() { ensure_db_subnet_group ensure_security_group ensure_parameter_group + ensure_log_group ensure_rds_instance ensure_backup_replication ensure_alarms