Updated to use ETCD database instead of redis #85
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
t-persson
commented
Jan 11, 2024
| task_result.forget() | ||
| failure = release_environment(etos, jsontas, registry, sub_suite) | ||
| ETCDPath(metadata.get("key")).delete() | ||
| registry.testrun.delete_all() |
There was a problem hiding this comment.
This delete_all should be done by the ESR, just before shutting down.
t-persson
force-pushed
the
etcd-database
branch
from
a84c419 to
bfc7088
Compare
Also cleaned up a few deprecated things. Removed the SubSuite endpoint since it's no longer used. Added type-hints to everything. Removed copyright year from all files.
t-persson
force-pushed
the
etcd-database
branch
from
bfc7088 to
738a7ee
Compare
fredjn
requested changes
Jan 17, 2024
| failure = None | ||
| for suite in task_result.result.get("suites", {}): | ||
| registry = ProviderRegistry(etos, jsontas, suite_id) | ||
| for suite, metadata in registry.testrun.join("suite").read_all(): |
There was a problem hiding this comment.
If for some reason suite or metadata is malformed or missing, the exceptions will be thrown all the way up to on_get in the webserver. I'm not sure how this is handeled by falcon.
There was a problem hiding this comment.
Falcon will return a server error which, in my opinion, is correct since we are reading from the database and not receiving the problematic values from the user.
fredjn
approved these changes
Feb 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Applicable Issues
eiffel-community/etos#205
Description of the Change
Start using the ETCD database instead of redis.
Also cleaned up a few deprecated things.
Removed the SubSuite endpoint since it's no longer used. Added type-hints to everything.
Removed copyright year from all files.
Alternate Designs
We could've set up the ETCDPath as a database instead of using it as a path-ish object. I really did enjoy using the path-ish object since it really represents the representation in ETCD.
I added a few compatibility layers in the database, these could've been avoided but then I would've had to sync a change in the ESR at the same time which is prone to failures. Instead we should have the compatibility layers now and remove them when the ESR has been updated.
The ETCDPath object should also be added to the ETOS library. I wanted to use the object "for real" before making big changes to the library though, so moving it should be done in the future.
Benefits
We are now using a database that's intended as a database and we have now put more thought into the structure of the database. Something that we did not have when the redis-hack was implemented.
Possible Drawbacks
We are now running both redis and etcd in the cluster for a single service. We are looking into moving over from python to golang and that change may remove the need for redis since it's used by celery.
Sign-off
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
Signed-off-by: Tobias Persson [email protected]