Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New microservices image with working RunServices #81

Closed
wants to merge 146 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
146 commits
Select commit Hold shift + click to select a range
bed8560
Database creation for daq-kube Microservices image
TiagoTAlves Oct 12, 2023
81ce781
added db queries framework but commented out for now
TiagoTAlves Nov 10, 2023
624d44b
corrected return out of function
TiagoTAlves Nov 15, 2023
b49ac2e
making some changes to dockerfile so image can build, will change bac…
TiagoTAlves Nov 15, 2023
0364811
change from ssh back to https
TiagoTAlves Nov 15, 2023
954c72f
continued changes to runserv microservices image
TiagoTAlves Nov 15, 2023
8c0e76d
getting errors with oracle backend, testing with just postgres for now
TiagoTAlves Nov 15, 2023
3e3c939
removing oracle mentions
TiagoTAlves Nov 16, 2023
a0481c9
added sqlalchemy to requirements
TiagoTAlves Nov 16, 2023
d62dd8a
added a backend fpr runnumber-rest
TiagoTAlves Nov 16, 2023
a689db4
typo fixes
TiagoTAlves Nov 16, 2023
586dc56
changes to rest.py and credentials.py
TiagoTAlves Nov 16, 2023
dab3ca0
added sqlalchemy_utils as another required package
TiagoTAlves Nov 16, 2023
6728749
changing the postgresql port
TiagoTAlves Nov 20, 2023
8b74b4e
indentation error
TiagoTAlves Nov 20, 2023
39ca665
fixing microservices image
TiagoTAlves Nov 20, 2023
5f79856
added MetaData to import list
TiagoTAlves Nov 20, 2023
bf5df7f
more connection tests
TiagoTAlves Nov 21, 2023
976e780
more testing
TiagoTAlves Nov 21, 2023
b8a37e0
add as a service
TiagoTAlves Nov 21, 2023
778d685
try a different service
TiagoTAlves Nov 21, 2023
d4abc4c
remove engine from inside metadata
TiagoTAlves Nov 21, 2023
13f5937
timezone errors
TiagoTAlves Nov 21, 2023
0d011b0
more timezone errors
TiagoTAlves Nov 21, 2023
6132346
psycopg2 added to engine
TiagoTAlves Nov 22, 2023
3e833f9
couldn't resolve host
TiagoTAlves Nov 22, 2023
a5fa76a
debugging
TiagoTAlves Nov 22, 2023
cd7671c
Attempt a code cleanup org for runnumber
jcpunk Nov 28, 2023
c5fe5a5
Try to facter out into parts
jcpunk Nov 28, 2023
6fec266
cleaned up runregistry in the same way as runnumber
TiagoTAlves Dec 4, 2023
df0d51c
changed scalar_one() to scalar() and removed credentials.py
TiagoTAlves Dec 4, 2023
4b17ddc
chanegd run_number to rn so that it is more inline with RunNumber table
TiagoTAlves Dec 4, 2023
dbe83f8
adding Column to import
TiagoTAlves Dec 4, 2023
b9ddd05
made changes to requirements so all modules are compatible when creat…
TiagoTAlves Dec 4, 2023
ab676e9
still unfinished chanegs to runregistry/api
TiagoTAlves Dec 4, 2023
7f09dae
Column is still complaining so I added the sqlalchemy module too
TiagoTAlves Dec 4, 2023
b0db812
solving database errors
TiagoTAlves Dec 5, 2023
51d9b9a
replaced sqlalchemy with flask_sqlalchemy modules
TiagoTAlves Dec 5, 2023
b578b92
changed sqlite to postgresql
TiagoTAlves Dec 5, 2023
7b13f5b
continued image problem solving
TiagoTAlves Dec 5, 2023
65f5204
taking it back to what it was before
TiagoTAlves Dec 5, 2023
40d4b1a
Try to register db object with application
jcpunk Dec 5, 2023
7d2d875
Fix path error
jcpunk Dec 5, 2023
6ccfeae
reorganised db sqlalchemy
TiagoTAlves Dec 5, 2023
721a608
fix path error for runregistry
TiagoTAlves Dec 5, 2023
b911945
added SQLAlchemy as app
TiagoTAlves Dec 5, 2023
afafed9
changing path to database
TiagoTAlves Dec 5, 2023
2120129
changing connection string and fixed circular db
TiagoTAlves Dec 5, 2023
322d909
change entrypoint for runregistry
TiagoTAlves Dec 5, 2023
9869de8
Drop Oracle LD libpath
jcpunk Dec 5, 2023
496f454
Drop obsolete comment
jcpunk Dec 5, 2023
f21ffa5
Drop old interface files
jcpunk Dec 5, 2023
de9e39c
Drop old comments
jcpunk Dec 5, 2023
3771bea
Minor nits in entrypoint.sh
jcpunk Dec 5, 2023
395bd2a
Don't enclose, just run unconditionally
jcpunk Dec 5, 2023
1583ebf
Reset name back to test mode
jcpunk Dec 5, 2023
f7e2487
Fix duplicate declare
jcpunk Dec 5, 2023
b439e1e
changed deployments to match ones in daq-kube
TiagoTAlves Dec 6, 2023
cb3d07f
Minify and restore image spec
jcpunk Dec 6, 2023
5c65334
Note required bits
jcpunk Dec 6, 2023
0710493
added run number as a foreign key connected to RunNumber table
TiagoTAlves Dec 7, 2023
60c96f6
added a row for the inital run so that all subsequent runs can platfo…
TiagoTAlves Dec 8, 2023
0dac8d3
changed some queries so its more similar to runnumber queries#
TiagoTAlves Dec 8, 2023
cddc8ae
bringing dockerfile back to original in prep for PR
TiagoTAlves Dec 8, 2023
ca0d986
Try to cleanup queries
jcpunk Dec 8, 2023
cf55c04
Merge remote-tracking branch 'origin' into riehecky/runnumber_reorg
TiagoTAlves Dec 12, 2023
16cb930
revert to working point
TiagoTAlves Dec 13, 2023
149a7fb
if no data, add a value starting at run 1
TiagoTAlves Dec 13, 2023
9497fa2
circular arguments makeing image not work
TiagoTAlves Dec 13, 2023
c3c830f
foreign key problem
TiagoTAlves Dec 13, 2023
7674575
changes in response to PR comments
TiagoTAlves Dec 14, 2023
6554a02
better comments for api query in run number
TiagoTAlves Dec 14, 2023
0520ed9
final comments and commits changing the curl commands and adding more…
TiagoTAlves Dec 14, 2023
d7c9538
set initial runnumber as environment variable
TiagoTAlves Dec 14, 2023
26ca9a9
black format
jcpunk Dec 14, 2023
69f4250
added missing curl, env variable description and changed scalar_one t…
TiagoTAlves Dec 14, 2023
f7115f6
Simplify imports
jcpunk Dec 14, 2023
a3bd7e3
added RUN_START to the right place
TiagoTAlves Dec 14, 2023
a27c746
added a foreignkey between RunReg tables
TiagoTAlves Dec 14, 2023
855a0da
asthetic changes
TiagoTAlves Dec 14, 2023
2690450
testing for self-doc
TiagoTAlves Jan 29, 2024
84bb17d
database changes
TiagoTAlves Jan 29, 2024
7b1930b
changes to api and rest to make it work
TiagoTAlves Jan 29, 2024
733bc65
asthetic changes
TiagoTAlves Jan 29, 2024
44f4d03
removed irrelevant stuff from merging
TiagoTAlves Jan 29, 2024
1fdc490
more aesthetic changes
TiagoTAlves Jan 29, 2024
a39bb75
bad changes made reverted
TiagoTAlves Feb 1, 2024
8cdff49
merging with develop
TiagoTAlves Feb 2, 2024
80c74af
Merge failed, fixing it
TiagoTAlves Feb 2, 2024
bdc6c04
Merge branch 'develop' into riehecky/runnumber_reorg
TiagoTAlves Feb 5, 2024
88c00c5
added two env_variables, one to establish what database (Oracle or Po…
TiagoTAlves Feb 13, 2024
07137bc
added more env_variable conditions for oracle/postgresql and dev/prod
TiagoTAlves Feb 13, 2024
0d165be
Merge branch 'develop' into riehecky/runnumber_reorg
TiagoTAlves Feb 13, 2024
5183b6b
made some changes to api's that I had forgotten to change when conver…
TiagoTAlves Feb 13, 2024
d4c768e
small corrections to previous commits
TiagoTAlves Feb 13, 2024
a48c4dd
small changes for oracle test
TiagoTAlves Feb 13, 2024
d472486
changes in a hurry for oracle testing
TiagoTAlves Feb 13, 2024
66fec83
cleanup leaks from daq-kube
jcpunk Feb 14, 2024
e730d1f
python black/isort
jcpunk Feb 14, 2024
18d6ba3
database type is derrived from database uri
jcpunk Feb 14, 2024
bf72cba
Switch to open source oracle client
jcpunk Feb 16, 2024
27f5c24
oracles doesn't like quotation marks so changed to inverted commas
TiagoTAlves Feb 26, 2024
d705fd5
aesthetic changes to database.py in both runservices
TiagoTAlves Feb 26, 2024
9564074
Fix exception catch
jcpunk Feb 26, 2024
4ad2180
Switch to fstrings
jcpunk Feb 26, 2024
671220c
rn isn't used in run registry so changed it to run_number
TiagoTAlves Feb 26, 2024
bbb9d7e
changes to runregistry and runnumber to fix api queries
TiagoTAlves Feb 28, 2024
4e1fd4c
further changes to get runregistry working
TiagoTAlves Feb 28, 2024
5caf5d7
foreign key and formatting
TiagoTAlves Feb 28, 2024
fcdcad2
Ignore accidentially compiled python objects
jcpunk Feb 28, 2024
641231b
changes to where function
TiagoTAlves Feb 28, 2024
c1fb4b0
making previous queries from oracle output the same as with new image
TiagoTAlves Feb 28, 2024
7122f5e
Try and unify/simplify service/sqlalchemy
jcpunk Feb 28, 2024
2bf67d0
Also include path
jcpunk Feb 28, 2024
f0e23b9
Add indexes and maybe compat url paths?
jcpunk Feb 28, 2024
f99a9e9
Setup compat paths for old and new api
jcpunk Feb 29, 2024
a4cc340
Run through black
jcpunk Feb 29, 2024
3fdffb0
Fix typo assignment
jcpunk Feb 29, 2024
5e2d621
Run isort
jcpunk Feb 29, 2024
bd39044
Minor spaces
jcpunk Feb 29, 2024
25138b3
Asked Chat GPT to simplify the codebase
jcpunk Feb 29, 2024
db94587
Need to import after `db` is init
jcpunk Feb 29, 2024
ae3c3bd
Simplify a bit more
jcpunk Feb 29, 2024
b611a3b
Since we're using the default names, just let the names default
jcpunk Feb 29, 2024
fa83032
Try to make database start at the right run number
jcpunk Feb 29, 2024
0fdce9f
Fix typo in debug messages
jcpunk Feb 29, 2024
ded3fc2
template out the help page. Not sure if this is cleaner or not...
jcpunk Feb 29, 2024
b5e6d5e
Not nested enough
jcpunk Feb 29, 2024
0f4de16
Update doc
jcpunk Feb 29, 2024
3c6b1d3
Use positive logic
jcpunk Feb 29, 2024
bf483cd
Drop default lines
jcpunk Feb 29, 2024
4e6f116
Oracle thick client not required
jcpunk Feb 29, 2024
27ead19
make response belongs to flask
TiagoTAlves Mar 1, 2024
c7bacb6
Revert "template out the help page. Not sure if this is cleaner or no…
jcpunk Mar 4, 2024
061e920
changed names of directories (so can be called for importing) and del…
TiagoTAlves Mar 1, 2024
943fff5
changed runnumber api to macth runregistry api
TiagoTAlves Mar 1, 2024
8d54507
changed the comments and returns to match oracle
TiagoTAlves Mar 1, 2024
214cd0f
some returns had to many nested []
TiagoTAlves Mar 1, 2024
71aca43
Merge pull request #90 from DUNE-DAQ/titavare/runservices_reorg
TiagoTAlves Mar 4, 2024
eb2eae2
changed all modules so they're all calling from microservices as the …
TiagoTAlves Mar 4, 2024
6b8df76
added pythonpath to dockerfile for module finding
TiagoTAlves Mar 4, 2024
250212e
changing app route / in api
TiagoTAlves Mar 4, 2024
b6bdcb3
Try to make the runregistry application fully replace runnumber.
jcpunk Mar 4, 2024
775d2d7
Fix path
jcpunk Mar 8, 2024
8c1920f
Drop default lines
jcpunk Mar 8, 2024
b91a5a2
combining some changes made in other PR to make it work in both postg…
TiagoTAlves Mar 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
*~
__pycache__/
*.py[cod]
*.sw[po]
2 changes: 2 additions & 0 deletions dockerfiles/Dockerfile.microservices
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,6 @@ FROM ghcr.io/dune-daq/microservices_dependencies:$DEPENDENCY_TAG
ARG BRANCH=develop
RUN git clone https://github.com/DUNE-DAQ/microservices -b $BRANCH

ENV PYTHONPATH=/microservices:$PYTHONPATH

ENTRYPOINT ["/microservices/entrypoint.sh"]
5 changes: 0 additions & 5 deletions dockerfiles/Dockerfile.microservices_dependencies
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,6 @@ RUN yum clean all \
&& yum -y install gcc make git unzip libaio libnsl libpq-devel libffi-devel python3-pip python3-wheel \
&& yum clean all

RUN curl -O https://download.oracle.com/otn_software/linux/instantclient/1919000/oracle-instantclient19.19-basic-19.19.0.0.0-1.el9.x86_64.rpm \
&& rpm -iv oracle-instantclient19.19-basic-19.19.0.0.0-1.el9.x86_64.rpm

COPY requirements.txt /
RUN python3 -m pip install --upgrade setuptools && \
python3 -m pip install -r requirements.txt && \
Expand All @@ -21,8 +18,6 @@ RUN python3 -m pip install --upgrade setuptools && \
# elisa_client_api needed by the logbook microservice
RUN git clone https://github.com/DUNE-DAQ/elisa_client_api.git && python3 -m pip install --upgrade setuptools && python3 -m pip install ./elisa_client_api



# protoc-24.3-linux-x86_64.zip is the latest zipfile available as of Sep-15-2023
# See also https://grpc.io/docs/protoc-installation/#install-pre-compiled-binaries-any-os

Expand Down
13 changes: 7 additions & 6 deletions dockerfiles/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
aniso8601==9.0.1
apispec==5.1.1
click==8.1.2
cx-Oracle==8.2.1
dataclasses==0.6
Flask==2.1.1
Flask==2.2.5
Flask-Caching==2.0.2
Flask-HTTPAuth==4.6.0
flask-redis==0.4.0
Flask-RESTful==0.3.9
Flask-SQLAlchemy==3.0.5
gevent==22.10.2
greenlet==2.0.2
gunicorn==20.1.0
Expand All @@ -16,17 +16,18 @@ influxdb==5.3.1
itsdangerous==2.0.1
Jinja2==3.1.1
kafka-python==2.0.2
MarkupSafe==2.0.1
MarkupSafe==2.1.1
psycopg2-binary==2.9.7
pymongo==4.0.2
pytz==2022.1
redis==3.5.3
setuptools==39.2.0
six==1.16.0
typing_extensions==4.1.1
Werkzeug==2.1.1
typing_extensions==4.2.0
Werkzeug==2.2.2
zipp==3.8.0
zope.event==4.6
zope.interface==5.5.2
protobuf==4.24.3

sqlalchemy==2.0.20
oracledb==2.0.1
14 changes: 0 additions & 14 deletions runnumber-rest/authentication.py

This file was deleted.

55 changes: 0 additions & 55 deletions runnumber-rest/backend.py

This file was deleted.

3 changes: 0 additions & 3 deletions runnumber-rest/credentials.py

This file was deleted.

15 changes: 0 additions & 15 deletions runnumber-rest/entrypoint.sh

This file was deleted.

8 changes: 0 additions & 8 deletions runnumber-rest/queries.py

This file was deleted.

128 changes: 0 additions & 128 deletions runnumber-rest/rest.py

This file was deleted.

23 changes: 0 additions & 23 deletions runnumber-rest/runnum_table.sql

This file was deleted.

Loading