Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SNOW-90] Introduce CI job to test schemachange updates against a clone #78

Open
wants to merge 87 commits into
base: dev
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
ed485ba
Initial commit. A simple clone of the DB
jaymedina Oct 23, 2024
b388915
Install snowsql
jaymedina Oct 23, 2024
e3c575a
add pw
jaymedina Oct 23, 2024
ae8d75d
add debug step
jaymedina Oct 23, 2024
4371e0b
.
jaymedina Oct 23, 2024
c9989e5
debugging
jaymedina Oct 23, 2024
1b136f0
syntax
jaymedina Oct 23, 2024
680ecaa
directly pass vars
jaymedina Oct 23, 2024
7d946ed
new .sql file to run the command isntead
jaymedina Oct 23, 2024
a342f85
global vars
jaymedina Oct 23, 2024
244216b
dynamically change the name of the clone DB
jaymedina Oct 23, 2024
4f35760
syntax
jaymedina Oct 23, 2024
ab2a048
syntax
jaymedina Oct 23, 2024
be2e82f
using IDENTIFIER for db object
jaymedina Oct 23, 2024
fb4a208
.
jaymedina Oct 23, 2024
913ef4c
.
jaymedina Oct 23, 2024
ded7408
set var sub true
jaymedina Oct 23, 2024
b25b10d
add identifier
jaymedina Oct 23, 2024
2bdc6ef
.
jaymedina Oct 23, 2024
2090fad
.
jaymedina Oct 23, 2024
b0ebf0c
typo
jaymedina Oct 23, 2024
d91d5ab
.
jaymedina Oct 23, 2024
6a7b631
.
jaymedina Oct 23, 2024
d8c8927
?
jaymedina Oct 23, 2024
38da909
.
jaymedina Oct 23, 2024
c75abba
go back to running command in ci step itself
jaymedina Oct 23, 2024
1a5160f
role is data_engineer
jaymedina Oct 23, 2024
1edbb83
branch sha
jaymedina Oct 23, 2024
1bc7f04
Run schemachagne on the clone
jaymedina Oct 24, 2024
a94ea31
.
jaymedina Oct 24, 2024
cee2c5e
.
jaymedina Oct 24, 2024
df3a91f
remove clone_db.sql
jaymedina Oct 24, 2024
46c8390
label name changed
jaymedina Oct 25, 2024
ca73fb6
change job name. change clone name convention. fix typos. new delete_…
jaymedina Oct 28, 2024
6fe39ca
address syntax issues in clone name
jaymedina Oct 28, 2024
88c7b00
add a status message
jaymedina Oct 28, 2024
939c9f7
missing env var
jaymedina Oct 28, 2024
4116019
vars
jaymedina Oct 28, 2024
bac91b8
sysadmin to run schemachange
jaymedina Oct 28, 2024
1ffafa9
syntax fix
jaymedina Oct 28, 2024
c44ae9e
refactor and introduce drop_clone job
jaymedina Oct 28, 2024
a3eade5
no refactoring
jaymedina Oct 28, 2024
d76999f
drop_clone will only run after PR merging
jaymedina Oct 28, 2024
89f06b4
new contributing.md
jaymedina Oct 28, 2024
93253a2
updating contributing
jaymedina Oct 29, 2024
6e0d537
move tip
jaymedina Oct 29, 2024
9b187c1
.
jaymedina Oct 29, 2024
3bbd05c
.
jaymedina Oct 29, 2024
4f042c0
.
jaymedina Oct 29, 2024
d2a2452
.
jaymedina Oct 29, 2024
ba646fa
remove redundancies from drop_clone job
jaymedina Oct 29, 2024
6e24035
testing
jaymedina Oct 29, 2024
fc3a85d
testing done
jaymedina Oct 29, 2024
cb218a5
final comment on time travel
jaymedina Oct 29, 2024
06996ab
Update CONTRIBUTING.md
jaymedina Oct 29, 2024
ea07518
Update CONTRIBUTING.md
jaymedina Oct 29, 2024
9d4aaaa
fix typo. small formatting update.
jaymedina Oct 29, 2024
03dfc5a
only run create_clone on open prs. run drop_clone on closed and merge…
jaymedina Oct 30, 2024
ab9297a
improve clarity in contributing.md
jaymedina Oct 30, 2024
8498feb
rephrasing
jaymedina Oct 31, 2024
ed645f3
use snowflake cli
jaymedina Nov 4, 2024
ced4d2c
snowflake connection
jaymedina Nov 4, 2024
3a90e92
.
jaymedina Nov 4, 2024
a85a5ca
lol
jaymedina Nov 4, 2024
8e78bd8
.
jaymedina Nov 4, 2024
294244f
.
jaymedina Nov 4, 2024
f75fb49
we no longer need to auth via the query command
jaymedina Nov 4, 2024
65cfebb
run for every commit when the PR is labeled
jaymedina Nov 5, 2024
9d8fde0
V script to make dummy table
jaymedina Nov 5, 2024
6f2052a
quick change
jaymedina Nov 5, 2024
55cbfd8
rename V to see if that fixes the problem
jaymedina Nov 5, 2024
9a622ca
change to R script
jaymedina Nov 5, 2024
2c9bab2
rename table. update drop_clone job.
jaymedina Nov 5, 2024
912b974
.
jaymedina Nov 5, 2024
d0880d8
try running schemachange as data_engineer
jaymedina Nov 5, 2024
c6ba8d5
.
jaymedina Nov 5, 2024
8b50065
grant privileges to dpe_engineer
jaymedina Nov 6, 2024
00c2a3c
attempt at using dpe_engineer for schematic again
jaymedina Nov 6, 2024
5babc72
put both configs in the same file
jaymedina Nov 6, 2024
97d0b7b
typo
jaymedina Nov 7, 2024
517525f
grant ownership to CLONE database
jaymedina Nov 7, 2024
8e3dc71
dynamic tables are separate object types
jaymedina Nov 7, 2024
80c00ff
typo
jaymedina Nov 7, 2024
cd89f20
explicit grants for each asset level
jaymedina Nov 7, 2024
95be97e
apply the scripts only. ignore tasks.
jaymedina Nov 8, 2024
30be842
remove flag
jaymedina Nov 8, 2024
a053393
no R scripts before V scripts
jaymedina Nov 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Install snowsql
  • Loading branch information
jaymedina committed Oct 23, 2024
commit b3889154cc4f4112a345dde7d2ccafdcc5ee8f34
5 changes: 5 additions & 0 deletions .github/workflows/test_with_clone.yaml
Original file line number Diff line number Diff line change
@@ -42,6 +42,11 @@ jobs:
pip install numpy==1.26.4
pip install pandas==1.5.3

- name: Install SnowSQL
run: |
curl -O https://sfc-repo.snowflakecomputing.com/snowsql/bootstrap/1.2/linux_x86_64/snowsql-1.2.9-linux_x86_64.bash
SNOWSQL_DEST=~/bin SNOWSQL_LOGIN_SHELL=~/.profile bash snowsql-1.2.9-linux_x86_64.bash
jaymedina marked this conversation as resolved.
Show resolved Hide resolved

# We clone the DB using the syntax seen here:
# https://docs.snowflake.com/en/sql-reference/sql/create-clone#databases-schemas
- name: zero-copy clone the database