Rare test failure in CLI: "Current branch token and request branch token doesn't match."

[josh-berry]

The Temporal CLI tests hit a couple rare failures in CI after updating to Server 1.25 that we (Dan and I) believe to be a Server issue—we see spurious errors of the form: Current branch token and request branch token doesn't match.

These seem very rare because neither Dan nor I were able to repro them on our local machines (despite ~100 attempts each), and when we re-ran the failing CI job, the tests succeeded.

Expected Behavior

Successful CLI test run (no such errors reported)

Actual Behavior

The following two CLI tests failed — full log output here: https://github.com/temporalio/cli/actions/runs/10885937402/job/30204717954

In TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude:

=== FAIL: temporalcli TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude (0.02s)
    commands.workflow_reset_test.go:287: 
        	Error Trace:	/home/runner/actions-runner/_work/cli/cli/temporalcli/commands.workflow_reset_test.go:287
        	Error:      	Received unexpected error:
        	            	Current branch token and request branch token doesn't match.
        	Test:       	TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude

In TestSharedServerSuite/TestActivity_Complete:

=== FAIL: temporalcli TestSharedServerSuite/TestActivity_Complete (0.12s)
    commands_test.go:182: Calling: activity complete --activity-id dev-activity-id --workflow-id bb9450c4-9cf0-4ee6-b0f5-c5e1c1390db5 --result "complete-activity-result" --identity MyIdentity --address 127.0.0.1:37747
    commands.activity_test.go:27: 
        	Error Trace:	/home/runner/actions-runner/_work/cli/cli/temporalcli/commands.activity_test.go:27
        	Error:      	Received unexpected error:
        	            	Current branch token and request branch token doesn't match.
        	Test:       	TestSharedServerSuite/TestActivity_Complete

TestActivity_Complete is particularly interesting because there are no workflow resets performed during this test. This test is quite simple; here's what it does:

1. Launch a workflow
2. Wait for the workflow to launch an activity that never completes
3. Try to force-complete the activity using the CLI command: temporal activity complete […] --result […] --identity […]

I think this is a server-side issue because nothing in the test actually does a workflow reset, and so there should be no reason why the workflow history has diverging branches. (We are also not using clustering or MRN or anything like that; this is just a single-process dev-server, so replication doesn't come into play at all.)

Steps to Reproduce the Problem

1. Check out temporalio/cli
2. go test -v -count 1 -run '^TestSharedServerSuite/(TestWorkflow_Reset_ReapplyExclude|TestActivity_Complete)$' github.com/temporalio/cli/temporalcli

Specifications

• Version: 1.25
• Platform: ubuntu-arm