[Fabric E2E Sample] Added updates on notebook and pipeline

[camaderal]

Type of PR

• Code changes

Purpose

• This PR addresses the following issues:
  • temporarily updated setup notebook to download data from static repo in place of data generator
  • added logic for the data standardization and transformation steps in the Fabric notebook.
  • added implementation for great expectations in Fabric
  • added open telemetry reporting to the notebooks
  • updated pipeline to facilitate data ingestion, standardization, and transformation
  • fixed schema for specific columns: updated interim.rd_seg_id and malformed.rd_seg_id from STRING to INT

Does this introduce a breaking change? If yes, details on what can break

Yes, added some configuration that need to be set so that the pipeline could work.

Author pre-publish checklist

• Added test to prove my fix is effective or new feature works
• No PII in logs
• Made corresponding changes to the documentation

Validation steps

• Deployed the sample as per the README.md instructions.

Issues Closed or Referenced

• Closes Create Fabric notebook for Parking sensor data standardization step #835
• Closes Create Fabric notebook for data transform step #837
• Closes Spike of Transform notebook on Fabric #924
• Closes Spike of Standardize notebook on Fabric #926
• Closes Spike on Great Expectations on Fabric #948
• Closes Spike on configuring Data pipeline #958

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[promisinganuj]

Hi @camaderal, FYI I have changed the base of this PR to "feat/e2e-fabric-dataops-sample-v0-2" instead of "main".