Upgrade SQLAlchemy to 2.0.22 and Pandas to 1.5.3 #2572

IsaacMilarky · 2023-10-30T17:42:47Z

Description

Update SQLAlchemy to version 2.0.22
Update Pandas to version 1.5.3
Removed redundant ORM Relationship definitions
Fix MacOS not finding location of go
Fix Alembic env.py not working
Fix broken util endpoints

This PR fixes #2557 #2556

Signed commits

Yes, I signed my commits.

- Switch to SQLAlchemy ORM syntax for repo filtering - Will replace the large SQL string in repo_load_controller.py - Current implementation located in augur/api/view/utils.py - Add clearer error message when page number is incorrect on repo view - Upgrade SQLAlchemy to v2.0.17 - Required for new syntax that the old version does not support - Specifically `ilike` and `regexp_replace` Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]>

…v-dependcy-core-patch

…to use db.config.json

Signed-off-by: Isaac Milarsky <[email protected]>

…cute() Signed-off-by: Isaac Milarsky <[email protected]>

… with connection object. This allows a commit to be implicit within the resulting context manager and emulates the functionality of the now removed autocommit Signed-off-by: Isaac Milarsky <[email protected]>

…nsert_data method. fix missing .begin() call in block to deal with return columns. fix dict conversion for list of rows in sqlalchemy 2.x in insert_data and contributor breadth worker Signed-off-by: Isaac Milarsky <[email protected]>

Signed-off-by: Isaac Milarsky <[email protected]>

…nsactions Signed-off-by: Isaac Milarsky <[email protected]>

…x method sqlalchemy Signed-off-by: Isaac Milarsky <[email protected]>

…some reason Signed-off-by: Isaac Milarsky <[email protected]>

Signed-off-by: Isaac Milarsky <[email protected]>

Updating SQLAlchemy_upgrade branch from dev

Signed-off-by: Isaac Milarsky <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]>

Signed-off-by: Isaac Milarsky <[email protected]>

Ulincsys · 2023-10-30T19:29:16Z

augur/api/view/utils.py

+    cache_files_requested.append(request)
+    return render_template('index.j2', body="loading", title="Loading", d=dest, query_key=query, api_url=getSetting('serving'))
+
+with DatabaseEngine() as engine:


I think it's fine to merge for testing, but before this makes it into main, we need to ensure that the repo load controller gets updated with the below syntax (or something similar). This is just here for testing, and is not intended to be final

Signed-off-by: Isaac Milarsky <[email protected]>

ABrain7710

I think all the engine.connect() should be using a context manager to ensure the connection is being disposed of. I also don't think we should be removing valid relationships from the orm as this removes functionality without any benefit that I can see

augur/api/metrics/commit.py

augur/application/db/models/augur_operations.py

augur/tasks/data_analysis/insight_worker/tasks.py

Signed-off-by: Isaac Milarsky <[email protected]>

IsaacMilarky · 2023-11-01T15:52:09Z

I think all the engine.connect() should be using a context manager to ensure the connection is being disposed of. I also don't think we should be removing valid relationships from the orm as this removes functionality without any benefit that I can see

I made the suggested changes. I think the orm relationship thing should be resolved in a separate pr so I just reverted all of my changes to the orm relationships for now.

ABrain7710

There are still a few context managers missing for the engine.connect(). Also the process_ossf_dependency_metrics needs to stay in the secondary collection

augur/api/metrics/pull_request.py

augur/application/db/models/augur_operations.py

augur/tasks/data_analysis/insight_worker/tasks.py

augur/tasks/start_tasks.py

Signed-off-by: Isaac Milarsky <[email protected]>

ABrain7710

Looks good

Ulincsys and others added 30 commits July 3, 2023 18:47

fixing order

f2620ce

email test

005d345

Signed-off-by: Sean P. Goggins <[email protected]>

test

8e2876f

Signed-off-by: Sean P. Goggins <[email protected]>

update version

0954a83

Signed-off-by: Sean P. Goggins <[email protected]>

updating materialized view refresh code.

b8e0892

Merge remote-tracking branch 'origin/dev-dependcy-core-patch' into de…

1635141

…v-dependcy-core-patch

materialized view refresh script: Started. Needs to be parameterized …

b81a07e

…to use db.config.json

Merge branch 'dev-dependcy-core-patch' into SQLAlchemy_upgrade

9b6697d

fix fetchall_data_from_sql_text for sqlalchemy 2.x

7df2d08

Signed-off-by: Isaac Milarsky <[email protected]>

new method of converting row to dict in sqlalchemy 2.x

6d01398

Signed-off-by: Isaac Milarsky <[email protected]>

Merge branch 'dev' into SQLAlchemy_upgrade

5a35a27

deal with redundant relationship definition

fb5091e

Signed-off-by: Isaac Milarsky <[email protected]>

patch contributor breadth worker to use 2.x version of sqlalchemy exe…

20e2f26

…cute() Signed-off-by: Isaac Milarsky <[email protected]>

fix insert_data to work with sqlalchemy 2.x by using new begin method…

fd2b4c5

… with connection object. This allows a commit to be implicit within the resulting context manager and emulates the functionality of the now removed autocommit Signed-off-by: Isaac Milarsky <[email protected]>

update mappings syntax

a39eb0b

Signed-off-by: Isaac Milarsky <[email protected]>

update alembic scripts to work with sqlalchemy 2.x

c390c93

Signed-off-by: Isaac Milarsky <[email protected]>

get rid of redundancies in relationship definitions in orm

98c0a51

Signed-off-by: Isaac Milarsky <[email protected]>

use begin() syntax to implicitly commit results of sqlalchemy 2.x tra…

e92023b

…nsactions Signed-off-by: Isaac Milarsky <[email protected]>

missed a dictionary conversion that used the 1.4 method instead of 2.…

82ba6f8

…x method sqlalchemy Signed-off-by: Isaac Milarsky <[email protected]>

remove call to undefined method in gunicorn start that was there for …

b31bd78

…some reason Signed-off-by: Isaac Milarsky <[email protected]>

merge incoming changes

c51a73c

Signed-off-by: Isaac Milarsky <[email protected]>

Merge branch 'SQLAlchemy_upgrade' into dev

897f380

Merge pull request #2559 from chaoss/dev

0f36424

Updating SQLAlchemy_upgrade branch from dev

fix legacy alembic upgrade

ec5d543

Signed-off-by: Isaac Milarsky <[email protected]>

upgrade sqlalchemy version and give context for schemas for alembic

7408477

Signed-off-by: Isaac Milarsky <[email protected]>

fixing go install on OSX where brew is used

740e418

Signed-off-by: Sean P. Goggins <[email protected]>

library version updates and fixed OSSF Scorecard Cloning (I think)

5d2b515

Signed-off-by: Sean P. Goggins <[email protected]>

update sqlalchemy syntax

2b6ad79

Signed-off-by: Isaac Milarsky <[email protected]>

IsaacMilarky added 3 commits October 30, 2023 10:20

Merge branch 'dev' into SQLAlchemy_upgrade

e392457

Fix syntax and get rid of SQLAlchemy warning

ad74442

Signed-off-by: Isaac Milarsky <[email protected]>

get API working with sqlalchemt 2.x

4f3f415

Signed-off-by: Isaac Milarsky <[email protected]>

IsaacMilarky requested a review from sgoggins as a code owner October 30, 2023 17:42

Ulincsys reviewed Oct 30, 2023

View reviewed changes

linter fixes

97030d1

Signed-off-by: Isaac Milarsky <[email protected]>

sgoggins requested review from Ulincsys and ABrain7710 and removed request for sgoggins October 31, 2023 13:05

ABrain7710 requested changes Oct 31, 2023

View reviewed changes

augur/api/metrics/commit.py Outdated Show resolved Hide resolved

augur/application/db/models/augur_operations.py Outdated Show resolved Hide resolved

augur/tasks/data_analysis/insight_worker/tasks.py Outdated Show resolved Hide resolved

IsaacMilarky added 2 commits November 1, 2023 10:49

Andrew suggested changes

b50bf0d

Signed-off-by: Isaac Milarsky <[email protected]>

Andrew suggested changes

4489f1a

Signed-off-by: Isaac Milarsky <[email protected]>

IsaacMilarky requested a review from ABrain7710 November 1, 2023 15:52

ABrain7710 requested changes Nov 1, 2023

View reviewed changes

missed fixes for andrew

f5b3014

Signed-off-by: Isaac Milarsky <[email protected]>

IsaacMilarky requested a review from ABrain7710 November 1, 2023 17:45

ABrain7710 approved these changes Nov 1, 2023

View reviewed changes

sgoggins merged commit 6c979f2 into dev Nov 1, 2023
4 checks passed

IsaacMilarky deleted the SQLAlchemy_upgrade branch November 1, 2023 18:07

This was referenced Nov 1, 2023

Broken API Endpoint to get REPO_ID given repo name and organization #2556

Closed

Fix endpoint to get repos in a repo group #2557

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade SQLAlchemy to 2.0.22 and Pandas to 1.5.3 #2572

Upgrade SQLAlchemy to 2.0.22 and Pandas to 1.5.3 #2572

IsaacMilarky commented Oct 30, 2023 •

edited

Loading

Ulincsys Oct 30, 2023

ABrain7710 left a comment

IsaacMilarky commented Nov 1, 2023

ABrain7710 left a comment

ABrain7710 left a comment

Upgrade SQLAlchemy to 2.0.22 and Pandas to 1.5.3 #2572

Upgrade SQLAlchemy to 2.0.22 and Pandas to 1.5.3 #2572

Conversation

IsaacMilarky commented Oct 30, 2023 • edited Loading

Ulincsys Oct 30, 2023

Choose a reason for hiding this comment

ABrain7710 left a comment

Choose a reason for hiding this comment

IsaacMilarky commented Nov 1, 2023

ABrain7710 left a comment

Choose a reason for hiding this comment

ABrain7710 left a comment

Choose a reason for hiding this comment

IsaacMilarky commented Oct 30, 2023 •

edited

Loading