Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(snowflake-driver): support azure exports buckets #8730

Merged
merged 45 commits into from
Sep 30, 2024
Merged

feat(snowflake-driver): support azure exports buckets #8730

merged 45 commits into from
Sep 30, 2024

Conversation

KSDaemon
Copy link
Member

@KSDaemon KSDaemon commented Sep 19, 2024 •
edited
Loading

This PR brings support for Azure buckets in Snowflake driver and also STORAGE_INTEGRATION option for AWS S3 buckets.

Other changes included:

  • Refactored extraction of S3 signed urls of exported files in BaseDriver
  • All drivers that could export to S3 now use the same implementation from BaseDriver
  • Refactored extraction of Azure signed urls of exported files in BaseDriver
  • All drivers that could export to Azure now use the same implementation from BaseDriver
  • Refactored extraction of GCP signed urls of exported files in BaseDriver
  • All drivers that could export to GCP now use the same implementation from BaseDriver (besides BigQuery)

Check List

  • Tests has been run in packages where changes made if available
  • Linter has been run for changed code
  • Tests for the changes have been added if not covered yet
  • Docs have been added / updated if required

@vercel Vercel
Copy link

vercel bot commented Sep 19, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

8 Skipped Deployments
Name Status Preview Comments Updated (UTC)
examples-angular-dashboard ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-react-d3 ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-react-dashboard ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-react-data-table ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-react-highcharts ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-react-material-ui ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-react-pivot-table ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am
examples-vue-query-builder ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 8:23am

@KSDaemon KSDaemon force-pushed the feat/snowflake-azure-bucket branch 5 times, most recently from 66e9f3a to f09ae8e Compare September 25, 2024 11:44
@KSDaemon
wip: add azure export bucket options to Snowflake driver
78acec5
@KSDaemon
add env var for Azure SAS Token
02af37b
@KSDaemon
add STORAGE_INTEGRATION variant to export bucket flows
d3f0593
@KSDaemon
Simplify isUnloadSupported()
3f28ea0
@KSDaemon
add extractFilesFromGCS() to BaseDriver Class
d34d448
@KSDaemon
implement extractFilesFromAzure() in SnowFlake Driver
3a630b9
@KSDaemon
some refactoring in extractFilesFromAzure()
2a7007d
@KSDaemon
debug and fix extractFilesFromAzure()
748d507
@KSDaemon
remove error if no csv files were exported (that's ok)
187161b
@KSDaemon
move extractFilesFromAzure to BaseDriver from Snowflake
77e3ed5
@KSDaemon
switch Athena Driver to use extractUnloadedFilesFromS3() from BaseDvi…
4d46a81 
…er impl
@KSDaemon
switch RedShift Driver to use extractUnloadedFilesFromS3() from BaseD…
53e31c4 
…vier impl
@KSDaemon
switch Databricks Driver to use extractUnloadedFilesFromS3() and extr…
4c79e3c 
…actFilesFromAzure() from BaseDvier impl
@KSDaemon
Remove unused/cleanup from databricks driver
5ed6ca5
@KSDaemon
update extractFilesFromGCS() in BaseDriver to receive credentials ins…
6ca19b3 
…tead of Storage instance
@KSDaemon
remove dep on @google-cloud/storage in Databricks for export bucket flow
3ff0844
@KSDaemon
fix Databricks driver export bucket flows after refactoring
2073213
@KSDaemon
set up CI to test clickhouse with s3 export bucket
b801306
@KSDaemon
Rename databricks export bucket test to reflect the s3 type
9f92483
@KSDaemon
fix clickhouse-export-bucket-s3-full script in package.json
94fb026
@KSDaemon
set up CI to test Snowflake with s3 export bucket
01ea8cd
@KSDaemon
rename databricks tests snapshot after renaming test job
99d2577
@KSDaemon
improve extractUnloadedFilesFromS3 flow to support different bucket n…
29d1d32 
…aming schemes
@KSDaemon
add snowflake test snapshots for export bucket test
95493ce
@KSDaemon KSDaemon marked this pull request as ready for review September 26, 2024 14:22
@KSDaemon KSDaemon requested review from a team as code owners September 26, 2024 14:22
@KSDaemon KSDaemon merged commit 37c6ccc into master Sep 30, 2024
56 checks passed
@KSDaemon KSDaemon deleted the feat/snowflake-azure-bucket branch September 30, 2024 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srh srh srh approved these changes

@paveltiunov paveltiunov paveltiunov approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@KSDaemon @srh @paveltiunov