update for cluster config and template

[thesqlpro]

Type of PR

Template change for Databricks Cluster Configuration

Purpose

Update cluster configuration and template file. Newer spark version and change to auto termination (reduced to 10 minutes).

Does this introduce a breaking change? If yes, details on what can break

Configurations and notebooks that reference the use of DBFS (internal databricks file system) - Investigation notes in comments below

Author pre-publish checklist

Issues Closed or Referenced

• Closes Upgrade Spark version to 15.4 LTS #896

[ydaponte]

This change alone will not work - you need to deploy the sample e2e and test:

1. The deployment itself (running deploy.sh) - because when the deployment tries to install the libraries in the cluster they will fail on this version because DBFS will no longer be available on the cluster.
2. The ADF pipelines will fail because the libraries will not be properly installed. My suggestion is deploy the solution with the Spark 15.4 version and start fixing in the interface the problems - when you know what to change then integrate the automation and run the deployment again e2e and make sure is working before submitting the PR.

Also, please update the metadata in the PR properly and describe the Type of PR - you left all the bullets, include the validation steps and which issues will be closed or reference when the PR closes. The body of the PR looks just like the template. Thanks!

[thesqlpro]

Documenting the issues @ydaponte listed
configure_Databricks.sh
line 88-102
echo "Uploading libs TO dbfs..."
databricks fs cp --recursive --overwrite "./databricks/libs/ddo_transform-localdev-py2.py3-none-any.whl" "dbfs:/ddo_transform-localdev-py2.py3-none-any.whl"

Create JSON file for library installation

json_file="./databricks/config/libs.config.json"
cat < $json_file
{
"cluster_id": "$cluster_id",
"libraries": [
{
"whl": "dbfs:/ddo_transform-localdev-py2.py3-none-any.whl"
}
]
}
EOF

Databricks recommends using the Unity Catalog or workspace instead.
https://learn.microsoft.com/en-us/azure/databricks/init-scripts/cluster-scoped

All of the notebooks reference a dbfs location (still testing if this will work)
"Backwards compatibility path" is /FileStore/tables/ no longer the standard dbfs:/mnt/
recommendation is to use volumes as well
example: /Volumes////files.csv

That way files and init scripts are in Unity Catalog.

[ydaponte]

@thesqlpro - yes, precisely, that was a temporary solution until we have the Unity Catalog + new Spark version in place. I've mention that on the very first sprint why I've mentioned that the dbfs file system will not be available anymore with the 15.4 version.

[thesqlpro]

This is not part of a feature #765
Has pre-requisites that need to be completed first. However the task itself is complete and ready to push after the other tasks before it are completed.