diff --git a/CHANGELOG.md b/CHANGELOG.md index 68c94c6..e4685f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,12 @@ All notable changes to this project are documented here. The format follows ## [Unreleased] +### Fixed +- Physical names that CloudFormation generates (a queue without `QueueName`, a table without `TableName`...) changed + after a restart. keepstack now journals `DescribeStackResources` once a stack deploy (`CreateStack`, `UpdateStack`, + `ExecuteChangeSet`) settles, and translates the replayed names back to the original ones + ([#19](https://github.com/sanlega/keepstack/issues/19)). + ## [0.3.1] - 2026-11-13 ### Fixed diff --git a/README.md b/README.md index 5c67ae2..e061235 100644 --- a/README.md +++ b/README.md @@ -421,8 +421,8 @@ Run several isolated environments side by side with different `--port`, `--emula Details in [how it works](docs/how-it-works.md#limitations). - **The journal only grows** until you run `keepstack compact`, which drops resources that were created and deleted. -- **CloudFormation / CDK:** stacks persist, but physical names that CloudFormation generates (resources without - `QueueName`, `BucketName`...) change after a restart. Name the resources you rely on. +- **CloudFormation / CDK:** stacks persist, including the physical names CloudFormation generates (resources without + `QueueName`, `BucketName`...), except for a stack that was still deploying when keepstack stopped. diff --git a/docs/compatibility.md b/docs/compatibility.md index 611dc49..93c65d4 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -19,7 +19,7 @@ the emulator container → `keepstack up` (no `--reset`) → `terraform plan` (n | `ssm_parameter`, `secretsmanager_secret`, `secretsmanager_secret_version`, `ecr_repository` | ✅ | ✅ | ✅ | OK (secret ARN suffix translated) | | `cloudwatch_event_rule`, `cloudwatch_event_target` | ✅ | ✅ | ✅ | OK | | `route53_zone`, `route53_record` | ✅ | ✅ | ✅ | OK (zone ID translated) | -| `cloudformation_stack` (with named `AWS::SQS::Queue`, `AWS::DynamoDB::Table`) | ✅ | ✅ | ✅ | OK. Generated physical names change after a restart, see [CloudFormation and CDK](#cloudformation-and-cdk) | +| `cloudformation_stack` (with named `AWS::SQS::Queue`, `AWS::DynamoDB::Table`) | ✅ | ✅ | ✅ | OK, generated physical names included, see [CloudFormation and CDK](#cloudformation-and-cdk) | | `security_group` (and `vpc_security_group_*_rule` that depend on it) | ❌ | n/a | n/a | **Emulator bug**, see below. ✅ with `keepstack up --shims` | | `s3_bucket_policy` with `Principal = "*"` | ❌ | n/a | n/a | Emulator enforces `BlockPublicPolicy` (likely the same as AWS defaults) | | `dynamodb_table` with `PAY_PER_REQUEST` | ✅ | ⚠️ | n/a | **Emulator bug**, see below. ✅ with `keepstack up --shims` | @@ -48,11 +48,12 @@ or `cdk deploy` (CDK synthesizes CloudFormation) work, and keepstack persists th every resource of the template. Verified with a stack containing an SQS queue and a DynamoDB table: after `keepstack down` / `up -d`, the stack, the queue and the table exist and `terraform plan` is clean. -One limitation: **physical names generated by CloudFormation change after a restart.** A queue declared without -`QueueName` is called something like `stack-Q-C212895A2991`; after a restart it is `stack-Q-C44A15AE335D`. The name -only appears in read responses, which keepstack does not journal, so it cannot be translated yet -([#19](https://github.com/sanlega/keepstack/issues/19)). Give explicit names (`QueueName`, `TableName`, `BucketName`...) -to resources you reference from outside the stack. Note that CloudFormation in the emulator scores 16% in the official +**Physical names generated by CloudFormation survive restarts.** A queue declared without `QueueName` is called +something like `stack-Q-C212895A2991`, and the replayed stack creates `stack-Q-C44A15AE335D`. Those names only appear +in read responses, so after `CreateStack`, `UpdateStack` or `ExecuteChangeSet` keepstack waits for the stack to finish +and journals its `DescribeStackResources`; on replay it pairs the old and new names by logical ID and clients keep +seeing the original ones. A stack that was still deploying when keepstack stopped is not covered: give explicit names +(`QueueName`, `TableName`, `BucketName`...) to resources you reference from outside the stack if that matters. Note that CloudFormation in the emulator scores 16% in the official fidelity tests (419/2612, v1.0.338), the lowest of the common services: run `keepstack validate` on your stacks. ## Running a group yourself diff --git a/docs/how-it-works.md b/docs/how-it-works.md index b737fa2..f5820ec 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -126,8 +126,8 @@ abandoned write is an end-to-end test (`tests/test_e2e_crash.py`). again when the calls that caused them are replayed. - **In-memory bodies.** Requests and responses are buffered (no streaming). Tested with 20 MB multipart S3 uploads. - **The journal only grows** until `keepstack compact` drops the resources that were created and later deleted. -- **CloudFormation-generated names.** Replaying `CreateStack` rebuilds every resource of the template and the stack - ID is translated, but physical names that CloudFormation generates (a queue without `QueueName` becomes - `stack-Q-C212895A2991`) only appear in read responses (`DescribeStackResources`), which are not journaled, so they - get new values after a restart. Give explicit names to resources you reference from outside the stack - ([#19](https://github.com/sanlega/keepstack/issues/19) tracks translating them). +- **CloudFormation-generated names.** Physical names that CloudFormation generates (a queue without `QueueName` + becomes `stack-Q-C212895A2991`) only appear in read responses. After a stack deploy (`CreateStack`, `UpdateStack`, + `ExecuteChangeSet`) keepstack polls `DescribeStackResources` until the stack settles and journals that answer; the + replay waits for the replayed stack the same way and pairs old and new names by logical ID. Names are lost for a + stack that was still deploying when keepstack stopped, and only the query (form) protocol is handled. diff --git a/docs/i18n/README.es.md b/docs/i18n/README.es.md index 17a8f0a..8060ba2 100644 --- a/docs/i18n/README.es.md +++ b/docs/i18n/README.es.md @@ -423,8 +423,8 @@ Puedes ejecutar varios entornos aislados en paralelo con distintos `--port`, `-- Detalles en [cómo funciona](../how-it-works.md#limitations). - **El journal solo crece** hasta que ejecutas `keepstack compact`, que elimina los recursos creados y luego borrados. -- **CloudFormation / CDK:** los stacks persisten, pero los nombres físicos que genera CloudFormation (recursos sin - `QueueName`, `BucketName`...) cambian tras un reinicio. Pon nombre a los recursos de los que dependas. +- **CloudFormation / CDK:** los stacks persisten, también los nombres físicos que genera CloudFormation (recursos sin + `QueueName`, `BucketName`...), salvo los de un stack que aún se estaba desplegando cuando keepstack se detuvo. diff --git a/docs/i18n/README.zh-CN.md b/docs/i18n/README.zh-CN.md index 16b76f5..246e8d2 100644 --- a/docs/i18n/README.zh-CN.md +++ b/docs/i18n/README.zh-CN.md @@ -415,8 +415,8 @@ keepstack --version - **ID 转换基于启发式规则**:只有看起来像生成标识符的值才会被学习。 详情参见[工作原理](../how-it-works.md#limitations)。 - **journal 只增不减**,直到你运行 `keepstack compact`,它会删除已创建又被删除的资源。 -- **CloudFormation / CDK:** 堆栈会被持久化,但 CloudFormation 自动生成的物理名称(未设置 `QueueName`、`BucketName` - 等的资源)在重启后会改变。请为你依赖的资源显式命名。 +- **CloudFormation / CDK:** 堆栈会被持久化,CloudFormation 自动生成的物理名称(未设置 `QueueName`、`BucketName` + 等的资源)也会保留,但 keepstack 停止时仍在部署中的堆栈除外。 diff --git a/src/keepstack/proxy.py b/src/keepstack/proxy.py index 3ea40bf..980bc4a 100644 --- a/src/keepstack/proxy.py +++ b/src/keepstack/proxy.py @@ -27,6 +27,7 @@ import threading import time import urllib.parse +import xml.etree.ElementTree as ET from contextlib import contextmanager from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer @@ -49,6 +50,11 @@ CLIENT_GONE = object() # App.handle() result: the client left, nothing to answer REPLAY_RETRY_SECONDS = 30 # replay is faster than AWS: e.g. PutItem before the table is ACTIVE MAX_ALIGN_TOKENS = 4000 # difflib is quadratic: above this, match by position only +# CloudFormation generates physical names (stack-Q-C212895A2991) that only reads return. After these writes keepstack +# journals a DescribeStackResources snapshot, so the replay can pair the old names with the new ones (#19). +STACK_DEPLOY_OPS = {"CreateStack", "UpdateStack", "ExecuteChangeSet"} +STACK_SNAPSHOT_OP = "DescribeStackResources" +STACK_SNAPSHOT_TIMEOUT = 600 @contextmanager @@ -117,6 +123,25 @@ def same_shape_new_id(a, b): return a != b and len(a) == len(b) >= 12 and a.isalnum() and b.isalnum() and a.isupper() and b.isupper() +def stack_physical_ids(xml): + """DescribeStackResources XML -> {LogicalResourceId: PhysicalResourceId}.""" + try: + root = ET.fromstring(xml or b"") + except ET.ParseError: + return {} + out = {} + for member in root.iter(): + kids = {c.tag.rsplit("}", 1)[-1]: c.text or "" for c in member} + if kids.get("LogicalResourceId") and kids.get("PhysicalResourceId"): + out[kids["LogicalResourceId"]] = kids["PhysicalResourceId"].encode() + return out + + +def stack_settled(xml): + """Every resource of the snapshot has finished (nothing *_IN_PROGRESS).""" + return bool(stack_physical_ids(xml)) and b"_IN_PROGRESS<" not in xml + + def b64_blobs(data): return [m for m in B64_RE.findall(data) if b"+" in m or m.endswith(b"=")] @@ -165,6 +190,11 @@ def learn(self, orig_resp, new_resp): learned += self._learn_pair(a[i], b[j]) return learned + def learn_stack(self, orig_xml, new_xml): + """Pairs physical names by logical resource ID, so the order of the resources does not matter.""" + new = stack_physical_ids(new_xml) + return sum(self.learn(phys, new[lid]) for lid, phys in stack_physical_ids(orig_xml).items() if lid in new) + def _learn_pair(self, x, y): if x == y or x in self.to_new or y in self.to_orig: return 0 # identical, or already known (e.g. another resource's VpcId appearing in this response) @@ -240,6 +270,7 @@ def __init__(self, args): self.lock = threading.RLock() # serializes writes and the replay self.ready = threading.Event() # cleared while restarting/replaying self.stopping = False + self.generation = 0 # bumped on every boot self.container = f"keepstack-{args.port}" # one per instance: allows several in parallel os.makedirs(args.data_dir, exist_ok=True) self.db_path = os.path.join(args.data_dir, "journal.db") @@ -294,6 +325,7 @@ def boot(self): """Starts a clean emulator and replays the journal. Blocks traffic meanwhile.""" self.ready.clear() with self.lock: + self.generation += 1 # a stack snapshot taken across a restart would mix two emulators shims.enabled(self) # decide (and log) before replaying, which may need them self.start_emulator() self.ids = IdMap() @@ -306,19 +338,28 @@ def boot(self): if not 200 <= status < 300: log(f"warning: #{seq} {name} was never confirmed (keepstack was interrupted mid-call); skipping") continue - deadline, delay = time.time() + REPLAY_RETRY_SECONDS, 0.25 + snapshot = op == STACK_SNAPSHOT_OP + start, delay = time.time(), 0.25 while True: try: st, data = self.send(method, path, json.loads(headers), body, op) except Exception as e: # noqa: BLE001 - one broken row must not prevent restoring the rest st, data = 0, repr(e).encode() - if 200 <= st < 300 or time.time() > deadline: + ok_st = 200 <= st < 300 + # the replayed stack creates its resources in the background: wait until they all exist + if ok_st and (not snapshot or stack_settled(data)): + break + limit = STACK_SNAPSHOT_TIMEOUT if snapshot and ok_st else REPLAY_RETRY_SECONDS + if time.time() - start > limit: break time.sleep(delay) delay = min(delay * 2, 2) if 200 <= st < 300: ok += 1 - self.ids.learn(resp, data) + if snapshot: + self.ids.learn_stack(resp, data) + else: + self.ids.learn(resp, data) else: fail += 1 log(f"REPLAY FAILED #{seq} {name} {path}: HTTP {st} {data[:200]!r}") @@ -371,6 +412,35 @@ def send(self, method, path, headers, body, op=None): status, _, _, data = self.forward(*self.prepare(op, method, path, headers, body)) return status, shims.on_response(self, op, status, data) + def snapshot_stack(self, path, headers, body): + """After a stack deploy, wait until its resources exist and journal their physical names (#19).""" + # shortcut: query (form) protocol only, which is what the SDKs and the CLI use for CloudFormation + stack = urllib.parse.parse_qs((body or b"").decode("utf-8", "replace")).get("StackName", [None])[0] + if not stack: + return + gen = self.generation + req = urllib.parse.urlencode({"Action": STACK_SNAPSHOT_OP, "StackName": stack, + "Version": "2010-05-15"}).encode() + headers = {k: v for k, v in headers.items() if k.lower() != "x-amz-target"} + headers = fix_body_headers(headers, req) + deadline = time.time() + STACK_SNAPSHOT_TIMEOUT + while not self.stopping and time.time() < deadline: + time.sleep(1) + with self.lock: + if gen != self.generation: + return # the emulator restarted: the replay of the deploy takes a new snapshot + status, _, _, data = self.forward(*self.prepare(STACK_SNAPSHOT_OP, "POST", path, headers, req)) + if not 200 <= status < 300: + return # stack gone, or the emulator does not answer: nothing to translate + if stack_settled(data): + with self.db() as db: + db.execute("insert into journal(ts, op, method, path, headers, body, status, resp) " + "values(?,?,?,?,?,?,?,?)", (time.time(), STACK_SNAPSHOT_OP, "POST", path, + json.dumps(headers), req, status, + self.ids.to_client(data))) + return + log(f"stack {stack} did not finish within {STACK_SNAPSHOT_TIMEOUT} s; its generated names are not saved") + def handle(self, method, path, headers, body, client_gone=lambda: False): if path.startswith(dashboard.PREFIX): # served by keepstack itself, never forwarded or journaled return dashboard.handle(self, method, path, headers) @@ -409,6 +479,9 @@ def handle(self, method, path, headers, body, client_gone=lambda: False): with self.db() as db: if seq and ok: db.execute("update journal set status=?, resp=? where seq=?", (status, data, seq)) + if op in STACK_DEPLOY_OPS: + threading.Thread(target=self.snapshot_stack, args=(path, headers, body), + daemon=True).start() elif seq: db.execute("delete from journal where seq=?", (seq,)) elif ok and b"ReceiptHandle" in data: # ReceiveMessage that returned messages diff --git a/tests/test_e2e_cfn.py b/tests/test_e2e_cfn.py new file mode 100644 index 0000000..b124441 --- /dev/null +++ b/tests/test_e2e_cfn.py @@ -0,0 +1,118 @@ +"""E2E (#19): names that CloudFormation generates survive a restart. Real emulator; skipped without docker. + +Env vars: KEEPSTACK_E2E_PORT (default 9340; emulator = +1). +""" +import json +import os +import shutil +import subprocess +import sys +import tempfile +import time +import unittest +import urllib.error +import urllib.parse +import urllib.request + +from test_e2e import CHILD_ENV, ROOT, sh + +PORT = int(os.environ.get("KEEPSTACK_E2E_PORT", "9340")) +CONTAINER = f"keepstack-{PORT}" +ENDPOINT = f"http://127.0.0.1:{PORT}/" +TEMPLATE = {"Resources": {"Q": {"Type": "AWS::SQS::Queue"}}, "Outputs": {"Url": {"Value": {"Ref": "Q"}}}} + + +def call(service, body, headers): + auth = f"AWS4-HMAC-SHA256 Credential=test/20261113/us-east-1/{service}/aws4_request, SignedHeaders=host, Signature=x" + req = urllib.request.Request(ENDPOINT, data=body, headers={"Authorization": auth, **headers}) + try: + with urllib.request.urlopen(req, timeout=60) as r: + return r.status, r.read().decode() + except urllib.error.HTTPError as e: + return e.code, e.read().decode() + + +def cfn(**params): + return call("cloudformation", urllib.parse.urlencode({**params, "Version": "2010-05-15"}).encode(), + {"Content-Type": "application/x-www-form-urlencoded"}) + + +def sqs(op, **params): + return call("sqs", json.dumps(params).encode(), + {"X-Amz-Target": f"AmazonSQS.{op}", "Content-Type": "application/x-amz-json-1.0"}) + + +class E2ECloudFormation(unittest.TestCase): + proc = None + + @classmethod + def setUpClass(cls): + cls.tmp = tempfile.mkdtemp(prefix="keepstack-e2e-cfn-") + if sh("docker", "version", "--format", "{{.Server.Version}}").returncode != 0: + shutil.rmtree(cls.tmp, ignore_errors=True) + raise unittest.SkipTest("docker is not accessible") + sh("docker", "rm", "-f", CONTAINER) + + @classmethod + def tearDownClass(cls): + cls.stop() + sh("docker", "rm", "-f", CONTAINER) + shutil.rmtree(cls.tmp, ignore_errors=True) + + @classmethod + def stop(cls): + if cls.proc: + cls.proc.kill() + cls.proc.wait() + cls.proc = None + + @classmethod + def start(cls): + log = os.path.join(cls.tmp, f"proxy-{time.time()}.log") + with open(log, "w", encoding="utf-8") as f: + cls.proc = subprocess.Popen( + [sys.executable, "-m", "keepstack", "up", "--port", str(PORT), "--emulator-port", str(PORT + 1), + "--data-dir", os.path.join(cls.tmp, "data")], stdout=f, stderr=subprocess.STDOUT, cwd=ROOT, + env=CHILD_ENV) + end = time.time() + 300 # the first run may build the image + while time.time() < end: + with open(log, encoding="utf-8", errors="replace") as f: + text = f.read() + if "ready on" in text: + return text + if cls.proc.poll() is not None: + break + time.sleep(0.5) + raise AssertionError(f"keepstack did not start:\n{text}") + + def queue_url(self): + end = time.time() + 120 + while time.time() < end: + status, body = cfn(Action="DescribeStackResources", StackName="names") + if status == 200 and "_IN_PROGRESS<" not in body and "" in body: + return body.split("")[1].split("<")[0] + time.sleep(1) + self.fail(f"the stack did not finish: {status} {body}") + + def test_generated_queue_name_survives_restart(self): + self.start() + status, body = cfn(Action="CreateStack", StackName="names", TemplateBody=json.dumps(TEMPLATE)) + self.assertEqual(status, 200, body) + url = self.queue_url() + self.assertIn("names-Q-", url) + time.sleep(3) # keepstack polls every second before journaling the snapshot + + self.stop() + sh("docker", "rm", "-f", CONTAINER) + log = self.start() + self.assertIn("0 failed", log) + + self.assertEqual(self.queue_url(), url) # translated back to the name clients know + status, body = sqs("SendMessage", QueueUrl=url, MessageBody="hi") + self.assertEqual(status, 200, body) + status, body = sqs("GetQueueAttributes", QueueUrl=url, AttributeNames=["ApproximateNumberOfMessages"]) + self.assertEqual((status, json.loads(body)["Attributes"]["ApproximateNumberOfMessages"]), (200, "1"), body) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_unit.py b/tests/test_unit.py index 4b094cf..ab7dd45 100644 --- a/tests/test_unit.py +++ b/tests/test_unit.py @@ -174,6 +174,82 @@ def test_reads_are_answered_anyway(self): forward.assert_called_once() +def stack_xml(*resources, status="CREATE_COMPLETE"): + members = "".join(f"{lid}{phys}" + f"{status}" + for lid, phys in resources) + return (f'' + f"{members}" + f"").encode() + + +QURL = "https://sqs.us-east-1.amazonaws.com/000000000000/" + + +class TestStackPhysicalNames(unittest.TestCase): # #19 + def app(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + app = proxy.App(Namespace(port=0, data_dir=tmp.name)) + app.ready.set() + return app + + def test_names_pair_by_logical_id_in_any_order(self): + ids = IdMap() + orig = stack_xml(("Q", QURL + "s-Q-C212895A2991"), ("T", "s-T-41AE15097CA5")) + new = stack_xml(("T", "s-T-9F00AA11BB22"), ("Q", QURL + "s-Q-C44A15AE335D")) + self.assertEqual(ids.learn_stack(orig, new), 2) + self.assertEqual(ids.to_client(b'{"QueueUrl":"' + QURL.encode() + b's-Q-C44A15AE335D"}'), + b'{"QueueUrl":"' + QURL.encode() + b's-Q-C212895A2991"}') + self.assertEqual(ids.to_emulator(b"TableName=s-T-41AE15097CA5"), b"TableName=s-T-9F00AA11BB22") + + def test_deploy_journals_a_snapshot_once_the_stack_settles(self): + app = self.app() + busy, done = stack_xml(("Q", "s-Q-C212895A2991"), status="CREATE_IN_PROGRESS"), \ + stack_xml(("Q", "s-Q-C212895A2991")) + replies = [(200, "OK", [], b""), (200, "OK", [], busy), (200, "OK", [], done)] + with mock.patch.object(app, "forward", side_effect=replies), mock.patch.object(proxy.time, "sleep"), \ + mock.patch.object(proxy.threading, "Thread") as thread: + app.handle("POST", "/", {"Content-Type": "application/x-www-form-urlencoded"}, + b"Action=CreateStack&StackName=s&TemplateBody=%7B%7D&Version=2010-05-15") + thread.assert_called_once() + app.snapshot_stack(*thread.call_args.kwargs["args"]) # run it here instead of in a thread + with proxy.connect(app.db_path) as db: + ops = db.execute("select op, body, resp from journal order by seq").fetchall() + self.assertEqual([o for o, _, _ in ops], ["CreateStack", "DescribeStackResources"]) + self.assertIn(b"StackName=s", ops[1][1]) + self.assertEqual(ops[1][2], done) + + def test_replay_waits_for_the_stack_and_translates_its_names(self): + app = self.app() + orig = stack_xml(("Q", QURL + "s-Q-C212895A2991")) + new = stack_xml(("Q", QURL + "s-Q-C44A15AE335D")) + with proxy.connect(app.db_path) as db: + for op, body, resp in [("CreateStack", b"Action=CreateStack&StackName=s", b""), + ("DescribeStackResources", b"Action=DescribeStackResources&StackName=s", orig)]: + db.execute("insert into journal(ts, op, method, path, headers, body, status, resp) " + "values(0,?,'POST','/','{}',?,200,?)", (op, body, resp)) + busy = stack_xml(("Q", ""), status="CREATE_IN_PROGRESS") + with mock.patch.object(app, "start_emulator"), mock.patch.object(proxy.shims, "enabled"), \ + mock.patch.object(proxy.time, "sleep"), \ + mock.patch.object(app, "send", side_effect=[(200, b""), (200, busy), (200, new)]) as send: + app.boot() + self.assertEqual(send.call_count, 3) + self.assertEqual(app.ids.to_client(new), orig) + + def test_snapshot_of_a_deploy_interrupted_by_a_restart_is_dropped(self): + app = self.app() + busy = stack_xml(("Q", "s-Q-C212895A2991"), status="CREATE_IN_PROGRESS") + + def restart_meanwhile(*_): + app.generation += 1 + return 200, "OK", [], busy + with mock.patch.object(app, "forward", side_effect=restart_meanwhile), mock.patch.object(proxy.time, "sleep"): + app.snapshot_stack("/", {}, b"Action=CreateStack&StackName=s") + with proxy.connect(app.db_path) as db: + self.assertEqual(db.execute("select count(*) from journal").fetchone()[0], 0) + + class TestActionMetadata(unittest.TestCase): # regression #41 def test_marketplace_description_limit(self): root = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")