Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ All notable changes to this project are documented here. The format follows

## [Unreleased]

### Fixed
- Physical names that CloudFormation generates (a queue without `QueueName`, a table without `TableName`...) changed
after a restart. keepstack now journals `DescribeStackResources` once a stack deploy (`CreateStack`, `UpdateStack`,
`ExecuteChangeSet`) settles, and translates the replayed names back to the original ones
([#19](https://github.com/sanlega/keepstack/issues/19)).

## [0.3.1] - 2026-11-13

### Fixed
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,8 +421,8 @@ Run several isolated environments side by side with different `--port`, `--emula
Details in [how it works](docs/how-it-works.md#limitations).
- **The journal only grows** until you run `keepstack compact`, which drops resources that were created and
deleted.
- **CloudFormation / CDK:** stacks persist, but physical names that CloudFormation generates (resources without
`QueueName`, `BucketName`...) change after a restart. Name the resources you rely on.
- **CloudFormation / CDK:** stacks persist, including the physical names CloudFormation generates (resources without
`QueueName`, `BucketName`...), except for a stack that was still deploying when keepstack stopped.

</details>

Expand Down
13 changes: 7 additions & 6 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ the emulator container → `keepstack up` (no `--reset`) → `terraform plan` (n
| `ssm_parameter`, `secretsmanager_secret`, `secretsmanager_secret_version`, `ecr_repository` | ✅ | ✅ | ✅ | OK (secret ARN suffix translated) |
| `cloudwatch_event_rule`, `cloudwatch_event_target` | ✅ | ✅ | ✅ | OK |
| `route53_zone`, `route53_record` | ✅ | ✅ | ✅ | OK (zone ID translated) |
| `cloudformation_stack` (with named `AWS::SQS::Queue`, `AWS::DynamoDB::Table`) | ✅ | ✅ | ✅ | OK. Generated physical names change after a restart, see [CloudFormation and CDK](#cloudformation-and-cdk) |
| `cloudformation_stack` (with named `AWS::SQS::Queue`, `AWS::DynamoDB::Table`) | ✅ | ✅ | ✅ | OK, generated physical names included, see [CloudFormation and CDK](#cloudformation-and-cdk) |
| `security_group` (and `vpc_security_group_*_rule` that depend on it) | ❌ | n/a | n/a | **Emulator bug**, see below. ✅ with `keepstack up --shims` |
| `s3_bucket_policy` with `Principal = "*"` | ❌ | n/a | n/a | Emulator enforces `BlockPublicPolicy` (likely the same as AWS defaults) |
| `dynamodb_table` with `PAY_PER_REQUEST` | ✅ | ⚠️ | n/a | **Emulator bug**, see below. ✅ with `keepstack up --shims` |
Expand Down Expand Up @@ -48,11 +48,12 @@ or `cdk deploy` (CDK synthesizes CloudFormation) work, and keepstack persists th
every resource of the template. Verified with a stack containing an SQS queue and a DynamoDB table: after
`keepstack down` / `up -d`, the stack, the queue and the table exist and `terraform plan` is clean.

One limitation: **physical names generated by CloudFormation change after a restart.** A queue declared without
`QueueName` is called something like `stack-Q-C212895A2991`; after a restart it is `stack-Q-C44A15AE335D`. The name
only appears in read responses, which keepstack does not journal, so it cannot be translated yet
([#19](https://github.com/sanlega/keepstack/issues/19)). Give explicit names (`QueueName`, `TableName`, `BucketName`...)
to resources you reference from outside the stack. Note that CloudFormation in the emulator scores 16% in the official
**Physical names generated by CloudFormation survive restarts.** A queue declared without `QueueName` is called
something like `stack-Q-C212895A2991`, and the replayed stack creates `stack-Q-C44A15AE335D`. Those names only appear
in read responses, so after `CreateStack`, `UpdateStack` or `ExecuteChangeSet` keepstack waits for the stack to finish
and journals its `DescribeStackResources`; on replay it pairs the old and new names by logical ID and clients keep
seeing the original ones. A stack that was still deploying when keepstack stopped is not covered: give explicit names
(`QueueName`, `TableName`, `BucketName`...) to resources you reference from outside the stack if that matters. Note that CloudFormation in the emulator scores 16% in the official
fidelity tests (419/2612, v1.0.338), the lowest of the common services: run `keepstack validate` on your stacks.

## Running a group yourself
Expand Down
10 changes: 5 additions & 5 deletions docs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,8 @@ abandoned write is an end-to-end test (`tests/test_e2e_crash.py`).
again when the calls that caused them are replayed.
- **In-memory bodies.** Requests and responses are buffered (no streaming). Tested with 20 MB multipart S3 uploads.
- **The journal only grows** until `keepstack compact` drops the resources that were created and later deleted.
- **CloudFormation-generated names.** Replaying `CreateStack` rebuilds every resource of the template and the stack
ID is translated, but physical names that CloudFormation generates (a queue without `QueueName` becomes
`stack-Q-C212895A2991`) only appear in read responses (`DescribeStackResources`), which are not journaled, so they
get new values after a restart. Give explicit names to resources you reference from outside the stack
([#19](https://github.com/sanlega/keepstack/issues/19) tracks translating them).
- **CloudFormation-generated names.** Physical names that CloudFormation generates (a queue without `QueueName`
becomes `stack-Q-C212895A2991`) only appear in read responses. After a stack deploy (`CreateStack`, `UpdateStack`,
`ExecuteChangeSet`) keepstack polls `DescribeStackResources` until the stack settles and journals that answer; the
replay waits for the replayed stack the same way and pairs old and new names by logical ID. Names are lost for a
stack that was still deploying when keepstack stopped, and only the query (form) protocol is handled.
4 changes: 2 additions & 2 deletions docs/i18n/README.es.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,8 +423,8 @@ Puedes ejecutar varios entornos aislados en paralelo con distintos `--port`, `--
Detalles en [cómo funciona](../how-it-works.md#limitations).
- **El journal solo crece** hasta que ejecutas `keepstack compact`, que elimina los recursos creados y luego
borrados.
- **CloudFormation / CDK:** los stacks persisten, pero los nombres físicos que genera CloudFormation (recursos sin
`QueueName`, `BucketName`...) cambian tras un reinicio. Pon nombre a los recursos de los que dependas.
- **CloudFormation / CDK:** los stacks persisten, también los nombres físicos que genera CloudFormation (recursos sin
`QueueName`, `BucketName`...), salvo los de un stack que aún se estaba desplegando cuando keepstack se detuvo.

</details>

Expand Down
4 changes: 2 additions & 2 deletions docs/i18n/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -415,8 +415,8 @@ keepstack --version
- **ID 转换基于启发式规则**:只有看起来像生成标识符的值才会被学习。
详情参见[工作原理](../how-it-works.md#limitations)。
- **journal 只增不减**,直到你运行 `keepstack compact`,它会删除已创建又被删除的资源。
- **CloudFormation / CDK:** 堆栈会被持久化,但 CloudFormation 自动生成的物理名称(未设置 `QueueName`、`BucketName`
等的资源)在重启后会改变。请为你依赖的资源显式命名。
- **CloudFormation / CDK:** 堆栈会被持久化,CloudFormation 自动生成的物理名称(未设置 `QueueName`、`BucketName`
等的资源)也会保留,但 keepstack 停止时仍在部署中的堆栈除外。

</details>

Expand Down
79 changes: 76 additions & 3 deletions src/keepstack/proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import threading
import time
import urllib.parse
import xml.etree.ElementTree as ET
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

Expand All @@ -49,6 +50,11 @@
CLIENT_GONE = object() # App.handle() result: the client left, nothing to answer
REPLAY_RETRY_SECONDS = 30 # replay is faster than AWS: e.g. PutItem before the table is ACTIVE
MAX_ALIGN_TOKENS = 4000 # difflib is quadratic: above this, match by position only
# CloudFormation generates physical names (stack-Q-C212895A2991) that only reads return. After these writes keepstack
# journals a DescribeStackResources snapshot, so the replay can pair the old names with the new ones (#19).
STACK_DEPLOY_OPS = {"CreateStack", "UpdateStack", "ExecuteChangeSet"}
STACK_SNAPSHOT_OP = "DescribeStackResources"
STACK_SNAPSHOT_TIMEOUT = 600


@contextmanager
Expand Down Expand Up @@ -117,6 +123,25 @@ def same_shape_new_id(a, b):
return a != b and len(a) == len(b) >= 12 and a.isalnum() and b.isalnum() and a.isupper() and b.isupper()


def stack_physical_ids(xml):
"""DescribeStackResources XML -> {LogicalResourceId: PhysicalResourceId}."""
try:
root = ET.fromstring(xml or b"")
except ET.ParseError:
return {}
out = {}
for member in root.iter():
kids = {c.tag.rsplit("}", 1)[-1]: c.text or "" for c in member}
if kids.get("LogicalResourceId") and kids.get("PhysicalResourceId"):
out[kids["LogicalResourceId"]] = kids["PhysicalResourceId"].encode()
return out


def stack_settled(xml):
"""Every resource of the snapshot has finished (nothing *_IN_PROGRESS)."""
return bool(stack_physical_ids(xml)) and b"_IN_PROGRESS<" not in xml


def b64_blobs(data):
return [m for m in B64_RE.findall(data) if b"+" in m or m.endswith(b"=")]

Expand Down Expand Up @@ -165,6 +190,11 @@ def learn(self, orig_resp, new_resp):
learned += self._learn_pair(a[i], b[j])
return learned

def learn_stack(self, orig_xml, new_xml):
"""Pairs physical names by logical resource ID, so the order of the resources does not matter."""
new = stack_physical_ids(new_xml)
return sum(self.learn(phys, new[lid]) for lid, phys in stack_physical_ids(orig_xml).items() if lid in new)

def _learn_pair(self, x, y):
if x == y or x in self.to_new or y in self.to_orig:
return 0 # identical, or already known (e.g. another resource's VpcId appearing in this response)
Expand Down Expand Up @@ -240,6 +270,7 @@ def __init__(self, args):
self.lock = threading.RLock() # serializes writes and the replay
self.ready = threading.Event() # cleared while restarting/replaying
self.stopping = False
self.generation = 0 # bumped on every boot
self.container = f"keepstack-{args.port}" # one per instance: allows several in parallel
os.makedirs(args.data_dir, exist_ok=True)
self.db_path = os.path.join(args.data_dir, "journal.db")
Expand Down Expand Up @@ -294,6 +325,7 @@ def boot(self):
"""Starts a clean emulator and replays the journal. Blocks traffic meanwhile."""
self.ready.clear()
with self.lock:
self.generation += 1 # a stack snapshot taken across a restart would mix two emulators
shims.enabled(self) # decide (and log) before replaying, which may need them
self.start_emulator()
self.ids = IdMap()
Expand All @@ -306,19 +338,28 @@ def boot(self):
if not 200 <= status < 300:
log(f"warning: #{seq} {name} was never confirmed (keepstack was interrupted mid-call); skipping")
continue
deadline, delay = time.time() + REPLAY_RETRY_SECONDS, 0.25
snapshot = op == STACK_SNAPSHOT_OP
start, delay = time.time(), 0.25
while True:
try:
st, data = self.send(method, path, json.loads(headers), body, op)
except Exception as e: # noqa: BLE001 - one broken row must not prevent restoring the rest
st, data = 0, repr(e).encode()
if 200 <= st < 300 or time.time() > deadline:
ok_st = 200 <= st < 300
# the replayed stack creates its resources in the background: wait until they all exist
if ok_st and (not snapshot or stack_settled(data)):
break
limit = STACK_SNAPSHOT_TIMEOUT if snapshot and ok_st else REPLAY_RETRY_SECONDS
if time.time() - start > limit:
break
time.sleep(delay)
delay = min(delay * 2, 2)
if 200 <= st < 300:
ok += 1
self.ids.learn(resp, data)
if snapshot:
self.ids.learn_stack(resp, data)
else:
self.ids.learn(resp, data)
else:
fail += 1
log(f"REPLAY FAILED #{seq} {name} {path}: HTTP {st} {data[:200]!r}")
Expand Down Expand Up @@ -371,6 +412,35 @@ def send(self, method, path, headers, body, op=None):
status, _, _, data = self.forward(*self.prepare(op, method, path, headers, body))
return status, shims.on_response(self, op, status, data)

def snapshot_stack(self, path, headers, body):
"""After a stack deploy, wait until its resources exist and journal their physical names (#19)."""
# shortcut: query (form) protocol only, which is what the SDKs and the CLI use for CloudFormation
stack = urllib.parse.parse_qs((body or b"").decode("utf-8", "replace")).get("StackName", [None])[0]
if not stack:
return
gen = self.generation
req = urllib.parse.urlencode({"Action": STACK_SNAPSHOT_OP, "StackName": stack,
"Version": "2010-05-15"}).encode()
headers = {k: v for k, v in headers.items() if k.lower() != "x-amz-target"}
headers = fix_body_headers(headers, req)
deadline = time.time() + STACK_SNAPSHOT_TIMEOUT
while not self.stopping and time.time() < deadline:
time.sleep(1)
with self.lock:
if gen != self.generation:
return # the emulator restarted: the replay of the deploy takes a new snapshot
status, _, _, data = self.forward(*self.prepare(STACK_SNAPSHOT_OP, "POST", path, headers, req))
if not 200 <= status < 300:
return # stack gone, or the emulator does not answer: nothing to translate
if stack_settled(data):
with self.db() as db:
db.execute("insert into journal(ts, op, method, path, headers, body, status, resp) "
"values(?,?,?,?,?,?,?,?)", (time.time(), STACK_SNAPSHOT_OP, "POST", path,
json.dumps(headers), req, status,
self.ids.to_client(data)))
return
log(f"stack {stack} did not finish within {STACK_SNAPSHOT_TIMEOUT} s; its generated names are not saved")

def handle(self, method, path, headers, body, client_gone=lambda: False):
if path.startswith(dashboard.PREFIX): # served by keepstack itself, never forwarded or journaled
return dashboard.handle(self, method, path, headers)
Expand Down Expand Up @@ -409,6 +479,9 @@ def handle(self, method, path, headers, body, client_gone=lambda: False):
with self.db() as db:
if seq and ok:
db.execute("update journal set status=?, resp=? where seq=?", (status, data, seq))
if op in STACK_DEPLOY_OPS:
threading.Thread(target=self.snapshot_stack, args=(path, headers, body),
daemon=True).start()
elif seq:
db.execute("delete from journal where seq=?", (seq,))
elif ok and b"ReceiptHandle" in data: # ReceiveMessage that returned messages
Expand Down
118 changes: 118 additions & 0 deletions tests/test_e2e_cfn.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
"""E2E (#19): names that CloudFormation generates survive a restart. Real emulator; skipped without docker.

Env vars: KEEPSTACK_E2E_PORT (default 9340; emulator = +1).
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.parse
import urllib.request

from test_e2e import CHILD_ENV, ROOT, sh

PORT = int(os.environ.get("KEEPSTACK_E2E_PORT", "9340"))
CONTAINER = f"keepstack-{PORT}"
ENDPOINT = f"http://127.0.0.1:{PORT}/"
TEMPLATE = {"Resources": {"Q": {"Type": "AWS::SQS::Queue"}}, "Outputs": {"Url": {"Value": {"Ref": "Q"}}}}


def call(service, body, headers):
auth = f"AWS4-HMAC-SHA256 Credential=test/20261113/us-east-1/{service}/aws4_request, SignedHeaders=host, Signature=x"
req = urllib.request.Request(ENDPOINT, data=body, headers={"Authorization": auth, **headers})
try:
with urllib.request.urlopen(req, timeout=60) as r:
return r.status, r.read().decode()
except urllib.error.HTTPError as e:
return e.code, e.read().decode()


def cfn(**params):
return call("cloudformation", urllib.parse.urlencode({**params, "Version": "2010-05-15"}).encode(),
{"Content-Type": "application/x-www-form-urlencoded"})


def sqs(op, **params):
return call("sqs", json.dumps(params).encode(),
{"X-Amz-Target": f"AmazonSQS.{op}", "Content-Type": "application/x-amz-json-1.0"})


class E2ECloudFormation(unittest.TestCase):
proc = None

@classmethod
def setUpClass(cls):
cls.tmp = tempfile.mkdtemp(prefix="keepstack-e2e-cfn-")
if sh("docker", "version", "--format", "{{.Server.Version}}").returncode != 0:
shutil.rmtree(cls.tmp, ignore_errors=True)
raise unittest.SkipTest("docker is not accessible")
sh("docker", "rm", "-f", CONTAINER)

@classmethod
def tearDownClass(cls):
cls.stop()
sh("docker", "rm", "-f", CONTAINER)
shutil.rmtree(cls.tmp, ignore_errors=True)

@classmethod
def stop(cls):
if cls.proc:
cls.proc.kill()
cls.proc.wait()
cls.proc = None

@classmethod
def start(cls):
log = os.path.join(cls.tmp, f"proxy-{time.time()}.log")
with open(log, "w", encoding="utf-8") as f:
cls.proc = subprocess.Popen(
[sys.executable, "-m", "keepstack", "up", "--port", str(PORT), "--emulator-port", str(PORT + 1),
"--data-dir", os.path.join(cls.tmp, "data")], stdout=f, stderr=subprocess.STDOUT, cwd=ROOT,
env=CHILD_ENV)
end = time.time() + 300 # the first run may build the image
while time.time() < end:
with open(log, encoding="utf-8", errors="replace") as f:
text = f.read()
if "ready on" in text:
return text
if cls.proc.poll() is not None:
break
time.sleep(0.5)
raise AssertionError(f"keepstack did not start:\n{text}")

def queue_url(self):
end = time.time() + 120
while time.time() < end:
status, body = cfn(Action="DescribeStackResources", StackName="names")
if status == 200 and "_IN_PROGRESS<" not in body and "<PhysicalResourceId>" in body:
return body.split("<PhysicalResourceId>")[1].split("<")[0]
time.sleep(1)
self.fail(f"the stack did not finish: {status} {body}")

def test_generated_queue_name_survives_restart(self):
self.start()
status, body = cfn(Action="CreateStack", StackName="names", TemplateBody=json.dumps(TEMPLATE))
self.assertEqual(status, 200, body)
url = self.queue_url()
self.assertIn("names-Q-", url)
time.sleep(3) # keepstack polls every second before journaling the snapshot

self.stop()
sh("docker", "rm", "-f", CONTAINER)
log = self.start()
self.assertIn("0 failed", log)

self.assertEqual(self.queue_url(), url) # translated back to the name clients know
status, body = sqs("SendMessage", QueueUrl=url, MessageBody="hi")
self.assertEqual(status, 200, body)
status, body = sqs("GetQueueAttributes", QueueUrl=url, AttributeNames=["ApproximateNumberOfMessages"])
self.assertEqual((status, json.loads(body)["Attributes"]["ApproximateNumberOfMessages"]), (200, "1"), body)


if __name__ == "__main__":
unittest.main()
Loading
Loading