Skip to content

feat(run-store): make the run-ops router correct at N shards - #4771

Merged
d-cs merged 25 commits into
mainfrom
feat/routing-semantics-n-tri-13427
Aug 25, 2026
Merged

feat(run-store): make the run-ops router correct at N shards#4771
d-cs merged 25 commits into
mainfrom
feat/routing-semantics-n-tri-13427

Conversation

@d-cs

@d-cs d-cs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

What

Makes RoutingRunStore correct when the run-ops layer routes across more than two Postgres stores. Today it routes between a gen-1 new dedicated database and a legacy control-plane database; this generalizes every routing policy to N shards while keeping the two-store behaviour byte-identical.

The change sets the four routing decisions that were implicit in code order, and fixes one hazard that failed silently:

  • Id → shard key. The router resolves a shard key with resolveShard instead of the binary residency classifier, so a gen-2 id reaches its own shard through the keyed map.
  • Membership vs routing. #distinctStores (one entry per physical database, aliases excluded by a declared aliasOf) drives every sum, probe, and merge; #shards drives routing. An aliased shard can no longer make a sum count one database twice.
  • Probe order. A keyless lookup stays a sequential short-circuit at two stores; above two it fans out in parallel, picks by precedence, tolerates a single down leg, and keeps the canonical not-found throw on the legacy leg.
  • Precedence and duplicates. One merge helper across all four merge sites. A duplicate id confined to {new, legacy} stays silent (the known drain-mirror case); any other cross-shard duplicate increments runops_shard_duplicate_id_total and logs at error level.
  • Disjoint sum (the silent hazard). countPendingWaitpoints and the waitpoint collector now partition absent ids by shard and union by id rather than summing counts. A drain-mirrored waitpoint on both gen-1 stores is counted once, so a blocked run can no longer hang forever on a double-counted pending waitpoint.
  • Waitpoint completion. A gen-2 waitpoint completes on its own shard, overriding the legacy pins; a cuid waitpoint keeps its two-member gen-1-pair probe unchanged.
  • Fail-loud creates. A create with no shard key throws instead of silently defaulting to new. An id resolving to an unconfigured shard throws instead of being dropped.

Two new counters are exported: runops_shard_duplicate_id_total and runops_waitpoint_probe_fallback_total.

Why it is safe to merge

With only {new, legacy} configured every generalized rule reduces to today's behaviour. resolveShard returns exactly what the old classifier returned for every id shape that exists today, and no gen-2 id is minted yet. The only intentional behaviour change is the fail-loud create throw; an enumeration of production call sites confirmed no caller trips it.

Testing

  • New container-free algebra suite (50 cases) over probe order, precedence, the duplicate alarm, the disjoint-sum partition, the waitpoint probes, and the fail-loud paths.
  • New runOpsStore.nShardMatrix.test.ts runs a four-store matrix (legacy + new + two gen-2 shards) against real Postgres containers: the disjoint-sum union, the alias topology, cross-tree completion, pagination merges, and mixed-id hydration.
  • New makeNShardRunOpsPostgresTest(k) fixture in @internal/testcontainers.
  • Full run-store corpus green: 71 files, 480 tests. Typecheck, lint, format, and knip all clean.

Notes

  • Draft: opened for review; not marking ready yet.
  • No changeset or .server-changes file: internal routing infrastructure, no user-visible behaviour change.
  • TRI-13427.

d-cs added 17 commits August 25, 2026 09:23
@changeset-bot

changeset-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d6ec477

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@d-cs d-cs self-assigned this Aug 25, 2026
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c4d628eb-fc6b-40cf-b608-3557fbb1a528

📥 Commits

Reviewing files that changed from the base of the PR and between 623c61c and d6ec477.

📒 Files selected for processing (4)
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts
  • internal-packages/testcontainers/src/index.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📜 Recent review details
🧰 Additional context used
📓 Path-based instructions (7)
We use vitest exclusively. **Never mock anything** - use testcontainers instead.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/testcontainers/src/index.ts
  • internal-packages/run-store/src/runOpsStore.ts
  • internal-packages/run-store/src/index.ts
Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/testcontainers/src/index.ts
  • internal-packages/run-store/src/runOpsStore.ts
  • internal-packages/run-store/src/index.ts
Use vitest for all tests in the Trigger.dev repository

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/testcontainers/src/index.ts
  • internal-packages/run-store/src/runOpsStore.ts
  • internal-packages/run-store/src/index.ts
Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/testcontainers/src/index.ts
  • internal-packages/run-store/src/runOpsStore.ts
  • internal-packages/run-store/src/index.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/testcontainers/src/index.ts
  • internal-packages/run-store/src/runOpsStore.ts
  • internal-packages/run-store/src/index.ts
🧠 Learnings (1)
📚 Learning: 2026-08-21T14:26:14.909Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 4752
File: internal-packages/run-store/src/runOpsStore.shardMap.test.ts:5-11
Timestamp: 2026-08-21T14:26:14.909Z
Learning: For these RoutingRunStore unit tests, use an instrumented fakeStore() with a shared ordered call log when verifying routing algebra such as sequential probe order and merge precedence. Use testcontainer-backed tests separately for database behavior, including mixed residency and replica-lag scenarios.

Applied to files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
🔇 Additional comments (4)
internal-packages/run-store/src/index.ts (1)

7-7: LGTM!

internal-packages/run-store/src/runOpsStore.ts (1)

99-108: LGTM!

Also applies to: 122-156, 305-306, 339-344, 393-397, 412-416, 492-512, 696-700, 766-804, 1411-1475, 1532-1544, 1656-1716, 1847-1876, 1915-1922, 2037-2043, 2057-2062, 2092-2095, 2107-2116, 2134-2143, 2255-2268

internal-packages/run-store/src/runOpsStore.shardMap.test.ts (1)

22-29: LGTM!

Also applies to: 88-123, 347-372, 657-661, 816-848

internal-packages/testcontainers/src/index.ts (1)

434-489: LGTM!

Also applies to: 569-653, 737-751, 1099-1100


Walkthrough

The run store now supports configurable gen-2 shard routing with shard aliases and distinct-database fan-out. It deduplicates cross-shard results, applies deterministic precedence, partitions waitpoint queries, and records duplicate and fallback metrics. Gen-2 waitpoints route directly to their named shard, while invalid co-location and missing routing identifiers now raise errors. Idempotency matches include run IDs and creation times. New PostgreSQL fixtures and tests validate four-store routing, aggregation, completion, aliases, and pagination.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 46.15% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 11 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: generalizing the run-ops router for N shards.
Description check ✅ Passed The description explains the objectives, key implementation changes, safety considerations, testing, and issue reference. It omits the template checklist, explicit Closes line, and screenshots section…
Full details: Description check

Explanation

The description explains the objectives, key implementation changes, safety considerations, testing, and issue reference. It omits the template checklist, explicit Closes line, and screenshots section, but it provides sufficient detail for review.

✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/routing-semantics-n-tri-13427

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
internal-packages/testcontainers/src/nShardFixture.test.ts (1)

6-7: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add crumb markers to the new test blocks.

The added test code has no // @Crumbs marker or `// `#region` `@crumbs block. Add markers while developing these tests and strip them before merge.

Proposed change
+// `@crumbs`
 // Booting the PG14 + PG17 containers and cloning four databases on a cold runner far exceeds
 // vitest's 5s default (this package sets no global testTimeout), so pass a generous per-test one.
 nShardTest(
   "builds 4 distinct databases (legacy + new + 2 gen-2 shards)",
   async ({ legacyUri, newUri, shardUris }) => {
     expect(shardUris).toHaveLength(2);
     const all = [legacyUri, newUri, ...shardUris];
     expect(new Set(all).size).toBe(4);
   },
   120_000
 );
 
+// `@crumbs`
 nShardTest(
   "each gen-2 clone carries the run-ops subset schema",
   async ({ shardPrismas }) => {

As per coding guidelines: **/*: Add crumbs as you write code with // @Crumbs or `// `#region` `@crumbs, then strip them with agentcrumbs strip before merge.

Also applies to: 18-18

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 52a9ae21-41a5-4c86-a63b-b3e7040c24a5

📥 Commits

Reviewing files that changed from the base of the PR and between 433dbac and 2869f6a.

📒 Files selected for processing (1)
  • internal-packages/testcontainers/src/nShardFixture.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📜 Review details
⏰ Context from checks skipped due to timeout. (33)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
  • GitHub Check: internal / 🧪 Unit Tests: Internal
  • GitHub Check: obsmap / 🧪 Unit Tests: Observability Map
  • GitHub Check: runops-guard / runops-guard
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: fk-cascade-guard / fk-cascade-guard
  • GitHub Check: code-quality / code-quality
  • GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (7)
We use vitest exclusively. **Never mock anything** - use testcontainers instead.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
Use vitest for all tests in the Trigger.dev repository

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

  • internal-packages/testcontainers/src/nShardFixture.test.ts
🔇 Additional comments (1)
internal-packages/testcontainers/src/nShardFixture.test.ts (1)

8-16: LGTM!

Also applies to: 19-27

@d-cs
d-cs marked this pull request as ready for review August 25, 2026 11:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (4)
internal-packages/run-store/src/runOpsStore.ts (3)

100-105: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject alias cycles.

A self-alias or cycle passes this validation because each target exists in configured. #distinctStores then removes every cycle member. Unrouted reads and fan-out writes omit that database.

Reject self-aliases and cycles, or resolve every alias to a non-aliased root before building #distinctStores.


759-764: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Classify idempotency candidates by row.id.

If shard "a" aliases "new", fan-out returns an "a" run through the "new" leg. This condition then treats that gen-2 candidate as gen-1 and selects the last database result instead of the earliest createdAt. The idempotency lookup can select the wrong existing run.

Determine gen-2 participation from this.#shardKeyOfSafe(row.id). Keep the leg key only for gen-1 mirror precedence.


98-136: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add an approved crumb to the new shard-routing flow.

Add a // @Crumbs marker or `// `#region` `@crumbs block for the new shard configuration and alias resolution path. AGENTS.md does not list internal-packages/run-store, so obtain an approved namespace before adding the marker. Do not invent a namespace.

Source: Coding guidelines

internal-packages/run-store/src/runOpsStore.shardMap.test.ts (1)

86-116: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Replace the fake store with a testcontainers-backed fixture.

This fake reports presentIds only for pending waitpoints. It cannot represent a completed but present waitpoint. The routing tests can therefore miss an incorrect fallback probe.

Use the N-shard testcontainers fixture for these cases instead of extending FakeStore.

As per coding guidelines, “We use vitest exclusively. Never mock anything - use testcontainers instead.”

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: cea49659-9f09-47d9-9b5d-2caeed4fe352

📥 Commits

Reviewing files that changed from the base of the PR and between 2869f6a and 2160486.

📒 Files selected for processing (2)
  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review.

📜 Review details
⏰ Context from checks skipped due to timeout. (27)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
  • GitHub Check: internal / 🧪 Unit Tests: Internal
  • GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (7)
We use vitest exclusively. **Never mock anything** - use testcontainers instead.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts
Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts
Use vitest for all tests in the Trigger.dev repository

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts
Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

  • internal-packages/run-store/src/runOpsStore.shardMap.test.ts
  • internal-packages/run-store/src/runOpsStore.ts

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@d-cs

d-cs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — worked through the three merge-risk points. One was real and is fixed; the other two I believe are false positives, detail below.

1. Alias cycles/chains omitting a database — fixed (623c61c). Correct. The constructor validated that aliasOf names a configured key but not that the target is a real (non-aliased) root store. A self-alias, a chain (a→b→new), or a cycle (a→b, b→a) would leave keys in the aliased set and drop them from #distinctStores, so that database would vanish from every read/write fan-out. The constructor now rejects a self-alias, an alias whose target is itself aliased, and cycles. Added tests for all three.

2. Idempotency selecting the wrong run under an alias — I don't think this can occur. findRunsByIdempotencyKeys fans out via #fanOut, which iterates #distinctStores — aliased keys are excluded, so an alias is never queried as a separate leg and the tiebreak only ever sees distinct-store keys. An alias is one physical database, and the per-DB unique index on (runtimeEnvironmentId, taskIdentifier, idempotencyKey) prevents two same-key runs from coexisting there, so there is no in-alias duplicate to mis-rank. The createdAt-vs-precedence tiebreak only engages for genuine cross-database duplicates. If you see a concrete path where an aliased key reaches the idempotency legs, I'll dig in — but I can't construct one.

3. Creates failing when routing identifiers are omitted — intended, and verified safe. These are the fail-loud throws the ticket licenses: createTaskRunCheckpoint (no ownerRunId), createBatchTaskRun (no data.id), and a waitpoint co-located onto a gen-2 shard without a shard-stamped id. An enumeration of production call sites confirmed every caller passes an explicit id, so none of these fire in practice. They replace a silent wrong-shard write with a loud failure — the safe direction, and inert at two stores.

All changes are inert with two configured stores (isGen2 / alias validation never trigger there). Corpus green.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@d-cs

d-cs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Manual testing

Booted the webapp on this branch against all three run-ops database topologies and exercised the run hot path — create → block on a waitpoint → complete → resume — in each, via the SDK test-suite canary (the master orchestrator task, which fans out the full primitive suite: waitpoint tokens, waitpoint race, wait idempotency, consecutive waits, long waits (for/until/batch), nested dependencies, and trigger/batchTrigger-and-wait).

Topology Store path Boot Canary hot path
Single-DB bare PostgresRunStore (no router) ✅ HTTP 200
2-DB (control-plane + legacy coresident, run-ops on its own DB) RoutingRunStore, 2 stores ✅ HTTP 200 · coresident:true, expectSplit:false
3-DB (control-plane, legacy, run-ops all distinct DBs) RoutingRunStore, 3 stores ✅ HTTP 200 · coresident:false, expectSplit:true

In every mode: the app started cleanly (no FATAL, no sentinel-assertion failures), the boot interlocks resolved correctly (the coresidency sentinel flips as expected between 2-DB and 3-DB), a routed run-detail read rendered, and the canary was driven to completion end to end.

Key result: the canary produced identical results across all three topologies. The router path (2-DB and 3-DB) behaves the same as the no-router path (single-DB), and every waitpoint primitive passed in every mode — confirming the rewritten completion/count hot path (countPendingWaitpoints, forWaitpointCompletion, the disjoint-sum union) is behaviour-preserving on the live gen-1 path.

Gen-2 N-way sharding stays dormant here (no id minting, no shard config wired), so it was not exercised by this manual pass — that path is covered by the four-store container matrix in this PR and will be validated live via the alias soak in a follow-up.

…s-n-tri-13427

# Conflicts:
#	internal-packages/run-store/src/index.ts
devin-ai-integration[bot]

This comment was marked as resolved.

@d-cs
d-cs merged commit 97d70b8 into main Aug 25, 2026
51 checks passed
@d-cs
d-cs deleted the feat/routing-semantics-n-tri-13427 branch August 25, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants