Cassandra Schema Migration Error

[harisanker]

Version of Cadence server, and client(which language)
This is very important to root cause bugs.

• Server version: 0.22.0
• Client version:
• Client langauge:

Describe the bug
This issue is happening in cadence helm chart https://artifacthub.io/packages/helm/banzaicloud-stable/cadence/0.22.0
Cadence schema migration is failing with below error in cadence-schema-setup pod

2023/05/25 13:32:59 Starting schema setup, config=&{SchemaFilePath: InitialVersion:0.0 Overwrite:false DisableVersioning:false} 2023/05/25 13:32:59 Setting up version tables 2023/05/25 13:32:59 Cannot add already existing table "schema_version" to keyspace "cadence"

Because of the schema migration issues , all other cadence pods are in CrashLoopBackOff status with error
cassandra schema version compatibility check failed: version mismatch for keyspace/database: "cadence". Expected version: 0.31 cannot be greater than Actual version: 0.0

To Reproduce
Is the issue reproducible?

• [Yes]

Steps to reproduce the behavior:
Issue is always happening when we included cadence as a dependent chart in our main helm chart.
dependencies:

• name: cadence
  version: 0.22.0
  repository: https://kubernetes-charts.banzaicloud.com

[ibarrajo]

Hi, recently the Cadence team worked on the helm chart and this bug came up.

From my understanding, there was a fix which involved creating a bootstraping instance that would perform the migrations, then once done startup would continue. I believe the root cause was multiple instances attempting to migrate at the same time on first startup.