Great Expectations regression when upgrading databricks runtime

[ivankpenev]

Hi folks,

Me and my team are working on upgrading our databricks runtime environment from 7.3 to 9.1 This also means that we are going from running spark 3.0.1 to spark 3.1.2. One of the validation jobs we run is on a 89.9 GB parquet dataset that we load up into a dataframe and then run our great expectations on.

Our great expectations runs on spark and it seems quite strange to encounter a regression like this particularly from a version upgrade with spark. Are there any known issues or possible areas to look into?

Thanks

[kyleaton]

Hi @ivankpenev,

Can you specify what is regressing into what version?

Thanks,
Kyle