Skip to content

Commit

Permalink
Create migration for only max schema key of the same schema model
Browse files Browse the repository at this point in the history
After starting to use lookupSchemasUntil in fetchSchemasWithSameModel, we are only getting
schemas until the given schema key for every schema key. Previously, we were getting all the
schemas for the same schema model.

This change caused change of behavior when a message contains multiple schema keys for same schema model.
When this happens, RDB Loader tries to create same table multiple times. In order to solve this problem,
this commit contains the change for creating the migration for only max schema key of the same schema model.
  • Loading branch information
spenes committed Oct 2, 2024
1 parent b2dbafb commit 9e87b47
Showing 1 changed file with 4 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,11 @@ object Migration {
target: Target[I],
disableRecovery: List[SchemaCriterion]
): F[Migration[C]] = {
val nonAtomicTypes = discovery.shreddedTypes.filterNot(_.isAtomic)
val maxSchemaKeysPerTableName = DataDiscovery.getMaxSchemaKeyPerTableName(nonAtomicTypes).values.toList
val filteredTypes = nonAtomicTypes.filter(s => maxSchemaKeysPerTableName.contains(s.info.getSchemaKey))
val descriptions: LoaderAction[F, List[Description]] =
discovery.shreddedTypes.filterNot(_.isAtomic).traverse {
filteredTypes.traverse {
case s: ShreddedType.Tabular =>
if (!disableRecovery.contains(s.info.toCriterion))
EitherT.rightT[F, LoaderError](Description.Table(discovery.shredModels(s.info.getSchemaKey).mergeRedshiftSchemasResult))
Expand Down

0 comments on commit 9e87b47

Please sign in to comment.