Skip to content

Include memo in test server list workflow executions - #3182

Closed
shinseongsu wants to merge 1 commit into
temporalio:mainfrom
shinseongsu:fix-test-server-memo-2480
Closed

shinseongsu wants to merge 1 commit into
temporalio:mainfrom
shinseongsu:fix-test-server-memo-2480

Conversation

@shinseongsu

Copy link
Copy Markdown

The test server returned an empty memo for every entry from ListOpenWorkflowExecutions and ListClosedWorkflowExecutions. For the same workflow, DescribeWorkflowExecution returned the memo, and so does the real server for both list calls:

DescribeWorkflowExecution  -> memo = {orderId: "A-123"}
ListOpenWorkflowExecutions -> memo = {}

The list entries are built in TestWorkflowStoreImpl.constructWorkflowExecutionInfo from history alone, and that method never set the memo. Describe goes through the mutable state instead, which is why it was not affected.

What was changed

  • TestWorkflowStoreImpl.constructWorkflowExecutionInfo now sets the memo, computed by a new private memoFromHistory:
    1. take the memo from the WorkflowExecutionStarted event
    2. collect the upserted memo of every WorkflowPropertiesModified event, keeping the last value per key
    3. merge the two once with StateUtils.mergeMemo, the same merge the mutable state uses, so a null upsert removes the key in both places
  • Added ListWorkflowExecutionsMemoTest.
  • Added a CHANGELOG.md entry under Fixed.

No public API changes; everything is inside the internal test server.

Why?

Code that lists workflows and reads their memo works against temporal server start-dev but sees an empty memo under TestWorkflowEnvironment, so it cannot be tested without a real server. Reported in #2480.

Checklist

  1. Closes Memo-s not present in listOpenWorkflowExecutions when running with temporal-testing #2480

  2. How was this tested:

Correctness

ListWorkflowExecutionsMemoTest starts a workflow with memo {startKey, removedKey}. The workflow upserts upsertedKey and removes removedKey with a null upsert. The test asserts that the open listing returns {startKey, upsertedKey}, then completes the workflow and asserts that the closed listing returns the same memo.

  • without the fix: fails with expected:<{startKey=..., upsertedKey=...}> but was:<{}>
  • with the fix: passes

Performance

The memo is now rebuilt from history on every list call, so I measured ListOpenWorkflowExecutions before and after the change. Setup: 200 open workflows, each started with one memo key and then upserting 0, 50 or 200 more keys. After 10 warm-up calls, 50 calls were timed with System.nanoTime() through the blocking stub against the in-process test server (Apple M4 Pro, macOS 15.6, OpenJDK 21.0.10, Gradle 8.10.2).

upserts per workflow memo keys per entry (before → after) response size (before → after) p50 (before → after) p95 (before → after)
0 0 → 1 19 KB → 28 KB 0.14 ms → 0.18 ms 0.26 ms → 0.29 ms
50 0 → 51 19 KB → 415 KB 0.20 ms → 2.62 ms 0.29 ms → 3.35 ms
200 0 → 201 19 KB → 1.6 MB 0.14 ms → 11.53 ms 0.24 ms → 12.97 ms

With a few memo keys there is no noticeable difference. With many keys the call gets slower roughly in proportion to the response size, which is the memo that was previously missing. The memo work itself is linear in history length. My first version called mergeMemo once per upsert event, which copies the map every time, and took 72 ms (p50) for the 200-upsert case; collecting upserts per key and merging once brought that to 11.5 ms.

Benchmark harness (not part of this PR)
// Placed in temporal-test-server/src/test/java/io/temporal/testserver/functional/ and run with
// ./gradlew :temporal-test-server:test --tests "*ListMemoBenchmark" -i
public class ListMemoBenchmark {
  static final int WORKFLOWS = 200;
  static final int UPSERTS = 200; // 0, 50 and 200 were measured
  static final int CALLS = 50;

  @Rule
  public SDKTestWorkflowRule rule =
      SDKTestWorkflowRule.newBuilder().setWorkflowTypes(Impl.class).build();

  @Test(timeout = 600_000)
  public void bench() throws InterruptedException {
    List<WorkflowStub> stubs = new ArrayList<>();
    for (int i = 0; i < WORKFLOWS; i++) {
      Wf wf =
          rule.getWorkflowClient()
              .newWorkflowStub(
                  Wf.class,
                  WorkflowOptions.newBuilder()
                      .setTaskQueue(rule.getTaskQueue())
                      .setMemo(Collections.singletonMap("startKey", "v"))
                      .build());
      WorkflowClient.start(wf::execute, UPSERTS);
      stubs.add(WorkflowStub.fromTyped(wf));
    }
    ListOpenWorkflowExecutionsRequest req =
        ListOpenWorkflowExecutionsRequest.newBuilder()
            .setNamespace(rule.getWorkflowClient().getOptions().getNamespace())
            .build();
    // Wait until every workflow has completed its first workflow task.
    long deadline = System.currentTimeMillis() + 120_000;
    while (System.currentTimeMillis() < deadline) {
      List<WorkflowExecutionInfo> l = list(req);
      if (l.size() == WORKFLOWS && l.stream().allMatch(e -> e.getHistoryLength() >= 4)) {
        Thread.sleep(2000);
        break;
      }
    }
    for (int i = 0; i < 10; i++) {
      list(req); // warm-up
    }
    long[] ns = new long[CALLS];
    List<WorkflowExecutionInfo> last = null;
    for (int i = 0; i < CALLS; i++) {
      long t = System.nanoTime();
      last = list(req);
      ns[i] = System.nanoTime() - t;
    }
    Arrays.sort(ns);
    double avgMemo =
        last.stream().mapToInt(e -> e.getMemo().getFieldsCount()).average().orElse(0);
    long bytes = last.stream().mapToLong(WorkflowExecutionInfo::getSerializedSize).sum();
    System.err.printf(
        "workflows=%d memoKeys=%.0f respKB=%d p50=%.2fms p95=%.2fms%n",
        last.size(), avgMemo, bytes / 1024, ns[CALLS / 2] / 1e6, ns[(int) (CALLS * 0.95)] / 1e6);
    stubs.forEach(s -> s.signal("finish"));
  }

  private List<WorkflowExecutionInfo> list(ListOpenWorkflowExecutionsRequest req) {
    return rule.getWorkflowClient()
        .getWorkflowServiceStubs()
        .blockingStub()
        .listOpenWorkflowExecutions(req)
        .getExecutionsList();
  }

  @WorkflowInterface
  public interface Wf {
    @WorkflowMethod
    void execute(int upserts);

    @SignalMethod
    void finish();
  }

  public static class Impl implements Wf {
    private boolean done;

    @Override
    public void execute(int upserts) {
      for (int i = 0; i < upserts; i++) {
        Workflow.upsertMemo(Collections.singletonMap("k" + i, "v" + i));
      }
      Workflow.await(() -> done);
    }

    @Override
    public void finish() {
      done = true;
    }
  }
}

Regression

:temporal-test-server:test (99 tests) and :temporal-sdk:test --tests "io.temporal.internal.testing.*" (35 tests) pass. In one of my full runs WorkflowIdConflictPolicyTest.conflictPolicyUseExisting failed on the buffered flag of a request id info. That test does not call the list APIs, so it never reaches the changed code, and rerunning it on its own passed 20/20 on main and 20/20 with this change. It looks timing dependent: the flag depends on whether a workflow task is in flight when the second start request arrives.

  1. Any docs updates needed?
    No.

The test server built WorkflowExecutionInfo for ListOpenWorkflowExecutions
and ListClosedWorkflowExecutions from history without the memo, so callers
always got an empty memo even though DescribeWorkflowExecution and the real
server return it.

Rebuild the memo from the started event and every WorkflowPropertiesModified
upsert, using the same merge rules as the mutable state.

Fixes temporalio#2480
@shinseongsu
shinseongsu requested a review from a team as a code owner October 30, 2026 04:29
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Quinn-With-Two-Ns

Copy link
Copy Markdown
Contributor

We're not accepting fixes for this sort of thing, since we are planning to begin deprecation for this server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memo-s not present in listOpenWorkflowExecutions when running with temporal-testing

3 participants