Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade SQLAlchemy to 2.0.22 and Pandas to 1.5.3 #2572

Merged
merged 37 commits into from
Nov 1, 2023
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
529905d
Upgrade SQLAlchemy version
Ulincsys Jul 3, 2023
f2620ce
fixing order
sgoggins Jul 8, 2023
005d345
email test
sgoggins Jul 9, 2023
8e2876f
test
sgoggins Jul 9, 2023
0954a83
update version
sgoggins Jul 9, 2023
b8e0892
updating materialized view refresh code.
sgoggins Jul 13, 2023
1635141
Merge remote-tracking branch 'origin/dev-dependcy-core-patch' into de…
sgoggins Jul 13, 2023
b81a07e
materialized view refresh script: Started. Needs to be parameterized …
sgoggins Jul 13, 2023
9b6697d
Merge branch 'dev-dependcy-core-patch' into SQLAlchemy_upgrade
sgoggins Jul 18, 2023
7df2d08
fix fetchall_data_from_sql_text for sqlalchemy 2.x
IsaacMilarky Jul 25, 2023
6d01398
new method of converting row to dict in sqlalchemy 2.x
IsaacMilarky Jul 25, 2023
5a35a27
Merge branch 'dev' into SQLAlchemy_upgrade
IsaacMilarky Aug 7, 2023
fb5091e
deal with redundant relationship definition
IsaacMilarky Aug 7, 2023
20e2f26
patch contributor breadth worker to use 2.x version of sqlalchemy exe…
IsaacMilarky Aug 7, 2023
fd2b4c5
fix insert_data to work with sqlalchemy 2.x by using new begin method…
IsaacMilarky Aug 7, 2023
9e4445c
fix missing argument in splitting insert_data calls in recursion in i…
IsaacMilarky Aug 7, 2023
a39eb0b
update mappings syntax
IsaacMilarky Aug 7, 2023
c390c93
update alembic scripts to work with sqlalchemy 2.x
IsaacMilarky Aug 7, 2023
98c0a51
get rid of redundancies in relationship definitions in orm
IsaacMilarky Aug 8, 2023
e92023b
use begin() syntax to implicitly commit results of sqlalchemy 2.x tra…
IsaacMilarky Aug 8, 2023
82ba6f8
missed a dictionary conversion that used the 1.4 method instead of 2.…
IsaacMilarky Aug 11, 2023
b31bd78
remove call to undefined method in gunicorn start that was there for …
IsaacMilarky Aug 14, 2023
c51a73c
merge incoming changes
IsaacMilarky Oct 5, 2023
897f380
Merge branch 'SQLAlchemy_upgrade' into dev
sgoggins Oct 25, 2023
0f36424
Merge pull request #2559 from chaoss/dev
sgoggins Oct 25, 2023
ec5d543
fix legacy alembic upgrade
IsaacMilarky Oct 26, 2023
7408477
upgrade sqlalchemy version and give context for schemas for alembic
IsaacMilarky Oct 26, 2023
740e418
fixing go install on OSX where brew is used
sgoggins Oct 26, 2023
5d2b515
library version updates and fixed OSSF Scorecard Cloning (I think)
sgoggins Oct 26, 2023
2b6ad79
update sqlalchemy syntax
IsaacMilarky Oct 30, 2023
e392457
Merge branch 'dev' into SQLAlchemy_upgrade
IsaacMilarky Oct 30, 2023
ad74442
Fix syntax and get rid of SQLAlchemy warning
IsaacMilarky Oct 30, 2023
4f3f415
get API working with sqlalchemt 2.x
IsaacMilarky Oct 30, 2023
97030d1
linter fixes
IsaacMilarky Oct 31, 2023
b50bf0d
Andrew suggested changes
IsaacMilarky Nov 1, 2023
4489f1a
Andrew suggested changes
IsaacMilarky Nov 1, 2023
f5b3014
missed fixes for andrew
IsaacMilarky Nov 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion augur/api/metrics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ from augur.application.db.engine import engine
4. Define any queries with the structure show below
```py
repo_sql = s.sql.text(""" SELECT repo.repo_name FROM repo WHERE repo.repo_id = :repo_id """)
results = pd.read_sql(repo_sql, engine, params={'repo_id': repo_id})
results = pd.read_sql(repo_sql, engine.connect(), params={'repo_id': repo_id})
```
5. Return either a pandas dataframe, dict, or json.
- Note: If you return a pandas dataframe or dict it will be automatically converted into json
Expand Down
14 changes: 7 additions & 7 deletions augur/api/metrics/commit.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ def committers(repo_group_id, repo_id=None, begin_date=None, end_date=None, peri
"""
)

results = pd.read_sql(committersSQL, engine, params={'repo_id': repo_id,
results = pd.read_sql(committersSQL, engine.connect(), params={'repo_id': repo_id,
IsaacMilarky marked this conversation as resolved.
Show resolved Hide resolved
'repo_group_id': repo_group_id,'begin_date': begin_date, 'end_date': end_date, 'period':period})

return results
Expand Down Expand Up @@ -167,7 +167,7 @@ def annual_commit_count_ranked_by_new_repo_in_repo_group(repo_group_id, repo_id=
ORDER BY YEAR ASC
""".format(table, period))

results = pd.read_sql(cdRgNewrepRankedCommitsSQL, engine, params={'repo_id': repo_id,
results = pd.read_sql(cdRgNewrepRankedCommitsSQL, engine.connect(), params={'repo_id': repo_id,
'repo_group_id': repo_group_id,'begin_date': begin_date, 'end_date': end_date})
return results

Expand Down Expand Up @@ -265,7 +265,7 @@ def annual_commit_count_ranked_by_repo_in_repo_group(repo_group_id, repo_id=None
LIMIT 10
""")

results = pd.read_sql(cdRgTpRankedCommitsSQL, engine, params={ "repo_group_id": repo_group_id,
results = pd.read_sql(cdRgTpRankedCommitsSQL, engine.connect(), params={ "repo_group_id": repo_group_id,
"repo_id": repo_id})
return results

Expand Down Expand Up @@ -296,7 +296,7 @@ def top_committers(repo_group_id, repo_id=None, year=None, threshold=0.8):
ORDER BY patches DESC) a
""")

results = pd.read_sql(total_commits_SQL, engine,
results = pd.read_sql(total_commits_SQL, engine.connect(),
params={'year': year, 'repo_group_id': repo_group_id})
else:
total_commits_SQL = s.sql.text("""
Expand All @@ -308,7 +308,7 @@ def top_committers(repo_group_id, repo_id=None, year=None, threshold=0.8):
ORDER BY patches DESC) a
""")

results = pd.read_sql(total_commits_SQL, engine,
results = pd.read_sql(total_commits_SQL, engine.connect(),
params={'year': year, 'repo_id': repo_id})

if not results.iloc[0]['sum']:
Expand All @@ -334,7 +334,7 @@ def top_committers(repo_group_id, repo_id=None, year=None, threshold=0.8):
ORDER BY commits DESC
""")

results = pd.read_sql(committers_SQL, engine,
results = pd.read_sql(committers_SQL, engine.connect(),
params={'year': year, 'repo_group_id': repo_group_id})
else:
committers_SQL = s.sql.text("""
Expand All @@ -353,7 +353,7 @@ def top_committers(repo_group_id, repo_id=None, year=None, threshold=0.8):
ORDER BY commits DESC
""")

results = pd.read_sql(committers_SQL, engine,
results = pd.read_sql(committers_SQL, engine.connect(),
params={'year': year, 'repo_id': repo_id})

cumsum = 0
Expand Down
16 changes: 8 additions & 8 deletions augur/api/metrics/contributor.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ def contributors(repo_group_id, repo_id=None, period='day', begin_date=None, end
ORDER BY total DESC
""")

results = pd.read_sql(contributorsSQL, engine, params={'repo_id': repo_id, 'period': period,
results = pd.read_sql(contributorsSQL, engine.connect(), params={'repo_id': repo_id, 'period': period,
'begin_date': begin_date, 'end_date': end_date})
else:
contributorsSQL = s.sql.text("""
Expand Down Expand Up @@ -211,7 +211,7 @@ def contributors(repo_group_id, repo_id=None, period='day', begin_date=None, end
ORDER BY total DESC
""")

results = pd.read_sql(contributorsSQL, engine, params={'repo_group_id': repo_group_id, 'period': period,
results = pd.read_sql(contributorsSQL, engine.connect(), params={'repo_group_id': repo_group_id, 'period': period,
'begin_date': begin_date, 'end_date': end_date})
return results

Expand Down Expand Up @@ -281,7 +281,7 @@ def contributors_new(repo_group_id, repo_id=None, period='day', begin_date=None,
GROUP BY date, repo.repo_id, repo_name
""")

results = pd.read_sql(contributorsNewSQL, engine, params={'repo_id': repo_id, 'period': period,
results = pd.read_sql(contributorsNewSQL, engine.connect(), params={'repo_id': repo_id, 'period': period,
'begin_date': begin_date, 'end_date': end_date})
else:
contributorsNewSQL = s.sql.text("""
Expand Down Expand Up @@ -330,7 +330,7 @@ def contributors_new(repo_group_id, repo_id=None, period='day', begin_date=None,
GROUP BY date, repo.repo_id, repo_name
""")

results = pd.read_sql(contributorsNewSQL, engine, params={'repo_group_id': repo_group_id, 'period': period,
results = pd.read_sql(contributorsNewSQL, engine.connect(), params={'repo_group_id': repo_group_id, 'period': period,
'begin_date': begin_date, 'end_date': end_date})
return results

Expand All @@ -351,7 +351,7 @@ def lines_changed_by_author(repo_group_id, repo_id=None):
GROUP BY commits.repo_id, date_trunc('week', cmt_author_date::date), cmt_author_affiliation, cmt_author_email, repo_name
ORDER BY date_trunc('week', cmt_author_date::date) ASC;
""")
results = pd.read_sql(linesChangedByAuthorSQL, engine, params={"repo_id": repo_id})
results = pd.read_sql(linesChangedByAuthorSQL, engine.connect(), params={"repo_id": repo_id})
return results
else:
linesChangedByAuthorSQL = s.sql.text("""
Expand All @@ -362,7 +362,7 @@ def lines_changed_by_author(repo_group_id, repo_id=None):
GROUP BY repo_id, date_trunc('week', cmt_author_date::date), cmt_author_affiliation, cmt_author_email
ORDER BY date_trunc('week', cmt_author_date::date) ASC;
""")
results = pd.read_sql(linesChangedByAuthorSQL, engine, params={"repo_group_id": repo_group_id})
results = pd.read_sql(linesChangedByAuthorSQL, engine.connect(), params={"repo_group_id": repo_group_id})
return results

@register_metric()
Expand Down Expand Up @@ -420,7 +420,7 @@ def contributors_code_development(repo_group_id, repo_id=None, period='all', beg
GROUP BY a.email, a.repo_id, repo_name
""")

results = pd.read_sql(contributorsSQL, engine, params={'repo_id': repo_id, 'period': period,
results = pd.read_sql(contributorsSQL, engine.connect(), params={'repo_id': repo_id, 'period': period,
'begin_date': begin_date, 'end_date': end_date})
else:
contributorsSQL = s.sql.text("""
Expand Down Expand Up @@ -455,6 +455,6 @@ def contributors_code_development(repo_group_id, repo_id=None, period='all', beg
ORDER BY commits desc, email
""")

results = pd.read_sql(contributorsSQL, engine, params={'repo_group_id': repo_group_id, 'period': period,
results = pd.read_sql(contributorsSQL, engine.connect(), params={'repo_group_id': repo_group_id, 'period': period,
'begin_date': begin_date, 'end_date': end_date})
return results
5 changes: 3 additions & 2 deletions augur/api/metrics/deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import sqlalchemy as s
import pandas as pd
from augur.api.util import register_metric
import datetime

from ..server import engine

Expand Down Expand Up @@ -45,7 +46,7 @@ def deps(repo_group_id, repo_id=None, period='day', begin_date=None, end_date=No
AND repo_dependencies.repo_id = :repo_id
""")

results = pd.read_sql(depsSQL, engine)
results = pd.read_sql(depsSQL, engine.connect())

else:

Expand All @@ -69,7 +70,7 @@ def deps(repo_group_id, repo_id=None, period='day', begin_date=None, end_date=No
AND repo.repo_group_id = :repo_group_id
""")

results = pd.read_sql(depsSQL, engine)
results = pd.read_sql(depsSQL, engine.connect())
return results


Expand Down
2 changes: 1 addition & 1 deletion augur/api/metrics/insight.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,5 @@ def top_insights(repo_group_id, num_repos=6):
LIMIT :num_repos
)
""")
results = pd.read_sql(topInsightsSQL, engine, params={'repo_group_id': repo_group_id, 'num_repos': num_repos})
results = pd.read_sql(topInsightsSQL, engine.connect(), params={'repo_group_id': repo_group_id, 'num_repos': num_repos})
return results
Loading
Loading