[Unity Catalog] - Change the import library process

[thesqlpro]

Libraries are being imported into DBFS - as of now with Unity Catalog enabled these files go into the legacy Hive metastore. We would like to upload them into Unity Catalog. Additionally, need to back port the work to ADF as it will fail with the new way of storing the libraries. #896 depends on this task as the upgrade to the cluster will cause problems for ADF. Task #896 is ready, but this needs to be completed first.

DoD

• Library files are stored in the Unity Catalog metastore and no longer in the legacy Hive Metastore
• Back port of ADF pipelines to use new file paths from metastore

Task for #765

[ydaponte]

@thesqlpro, please do add a DoD and fill in the metadata.

[thesqlpro]

DoD and meta updated