0.19.3

What's Changed

• Allow type narrowing from VARCHAR to other types in redshift (#184)
• Remove redundant Product type from redshift (#181)

Full Changelog: 0.19.2...0.19.3

0.19.0

This release lays the solid foundation to improve schema evolution for Redshift.

Full Changelog: 0.18.2...0.19.0

Version 0.18.2

Fixes a bug involving parquet migrations. Before this fix, it was possible for loading to Databricks to break under certain edge-case schema evolutions.

Changelog

• Drop not null constraint in parquet migrations, when required field is added from new version (#170)

Version 0.18.1

Fixes for two bugs involving parquet migrations. Before these fixes, it was possible for loading to Databricks to break under certain edge-case schema evolutions.

Changelog

• Avoid field name collisions after snakeCase and other normalizations in parquet (#164)
• Drop not null constraint in parquet migrations, when required field is removed from new version (#168)

Version 0.18.0

This release adds $supersededBy and $supersedes properties to meta schemas defined in SelfSyntaxChecker object.

Changelog

• Allow supersededBy and supersedes properties in a schema (#155)

Version 0.17.0

This release amends the SelfSyntaxChecker linter to disallow schemas if the name and vendor properties contain invalid characters like spaces. See #139

Changelog

• Update license to 2023 (#147)
• Bump jackson-databind to 2.14.1 (#146)
• Bump scala-collection-compat to 2.9.0 (#145)
• Bump cats-parse to 0.3.9 (#144)
• Bump json-schema-validator to 1.0.76 (#143)
• Bump circe to 0.14.3 (#142)
• Bump iglu-scala-core to 1.1.2 (#141)
• Update links in Readme (#131)
• SelfSyntaxChecker validate schema names and vendors (#139)

Version 0.16.0

We are pleased to announce version 0.16.0 of the Schema DDL library.

https://github.com/snowplow/schema-ddl/releases/tag/0.16.0

This release is a preparation for the automatic recovery for breaking schema evolution in Databricks Loader/Transformer.
We introduce functionality to merge parquet schemas and detect incompatible schemas. We also introduce categories for
the schema evolution:

• NonBreaking - Miscellaneous changes that would be handled by the merge.schema:
  • NullableRequired - nullable to required change.
  • KeyAddition - addition of the new Struct key.
  • KeyRemoval - removing field in Struct.
  • RequiredNullable - Nullable to required field or array migration.
• Breaking - Breaking change that would either lead to oen of these outcomes:
  • IncompatibleType Type casting is not allowed with an exception for Struct fields.

Databricks loading does not support nullable fields, so they were categorised as NonBreaking.

Upgrading to 0.16.0

Add following to the build.sbt:

libraryDependencies += "com.snowplowanalytics" %% "schema-ddl" % "0.16.0"

0.15.1

Changelog

Bump scalacheck to 1.16.0 (#127)
Bump specs2 to 4.15.0 (#127)
Bump scala-collection-compat to 2.8.1 (#127)
Bump cats-parse to 0.3.8 (#127)
Bump json-schema-validator to 1.0.72 (#127)
Bump circe-generic to 0.14.2 (#127)
Bump iglu-core-json4s to 1.1.0 (#127)
Bump jackson-databind to 2.13.2.1 (#127)

0.15.0

Feature

• Add Parquet schema AST (#119)
• Utils for oneOf/anyOf properties to become fields of storage ASTs (#120)

Bug fix

• Redshift AddColumn migration should never be NOT NULL (#123)
• Fix Redshift column type for enum with a max length (#121)

0.14.5

This patch release fixes a problem that we made the linter far too strict in 0.14.4. The new release finds a better balance:

• Error messages are issued if a schema is completely un-usable by the Iglu system
• Warning messages are issued if a schema is technically valid, but might contain unknown additional properties.

Changelog

• SelfSyntaxChecker should validate against a more permissive version of jsonschema v4 (#117)