Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fabric E2E Sample] Added updates on notebook and pipeline #959

Open
wants to merge 14 commits into
base: feat/e2e-fabric-dataops-sample-v0-2
Choose a base branch
from

Conversation

camaderal
Copy link

@camaderal camaderal commented Dec 16, 2024

Type of PR

  • Code changes

Purpose

  • This PR addresses the following issues:
    • temporarily updated setup notebook to download data from static repo in place of data generator
    • added logic for the data standardization and transformation steps in the Fabric notebook.
    • added implementation for great expectations in Fabric
    • added open telemetry reporting to the notebooks
    • updated pipeline to facilitate data ingestion, standardization, and transformation
    • fixed schema for specific columns: updated interim.rd_seg_id and malformed.rd_seg_id from STRING to INT

Does this introduce a breaking change? If yes, details on what can break

Yes, added some configuration that need to be set so that the pipeline could work.

Author pre-publish checklist

  • Added test to prove my fix is effective or new feature works
  • No PII in logs
  • Made corresponding changes to the documentation

Validation steps

  • Deployed the sample as per the README.md instructions.

Issues Closed or Referenced

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@ydaponte ydaponte added the e2e: fabric Related with E2E Fabric Sample label Dec 16, 2024
@promisinganuj promisinganuj self-requested a review December 16, 2024 23:15
@promisinganuj promisinganuj changed the base branch from main to feat/e2e-fabric-dataops-sample-v0-2 December 17, 2024 02:40
@promisinganuj
Copy link
Contributor

Hi @camaderal, FYI I have changed the base of this PR to "feat/e2e-fabric-dataops-sample-v0-2" instead of "main".

@maye-msft maye-msft linked an issue Dec 19, 2024 that may be closed by this pull request
13 tasks
@yuna-s yuna-s mentioned this pull request Dec 19, 2024
9 tasks
@yuna-s yuna-s changed the title Added updates on notebook and pipeline [Fabric E2E Sample] Added updates on notebook and pipeline Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment