Fabric E2E Sample (Milestone 2)

[devlace]

Github Milestone: https://github.com/Azure-Samples/modern-data-warehouse-dataops/milestone/9
Milestone 2 target date: TBD

Automated Testing

• Integration Testing / Functional E2E testing (test/pre-prod environment - this is a permanent environment. Release is serial.)
• Unit testing of data pipelines and notebooks
• PR validation + Deployment to ephemeral pipeline (middle ground?)
  • This step takes substantial amount of time.
  • Requires a spike (can we go pure ephemeral or some are permanent; )

User Stories:

• Build unit testing of Data Pipelines #807
• Build integration testing package #808
• Include integration testing as part of release pipeline #810
• Spike: Ephemeral workspace creation as part of PR validation pipeline #806

Observability

• Open Telemetry standard

Python Package modernization

For Package development, update to latest / best practices. Align with Databricks + ADF update team on python package modernization.

• Spike: Investigate what are potential options to be in scope to update python package to "modernize" and simplify python package. #811
• Decide on whether we move Python package to a shared folder between the Parking Sensor samples (Working with @ydaponte ) #812

Consideration of Fabric functionality to showcases

• TBD: Functions, Python notebook

VNext Milestone ideas:

• Add streaming scenario
• Data Governance (Purview)