DM-45005: In Prompt Processing, use microservice to check for already-arrived Rubin images #192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kfindeisen
force-pushed
the
tickets/DM-45005
branch
3 times, most recently
from
457dee0
to
db85f4b
Compare
ktlim
approved these changes
Aug 30, 2024
python/activator/raw.py
Outdated
| raise RuntimeError("Could not query raw microservice.") from e | ||
|
|
||
| if unpacked["error"]: | ||
| raise RuntimeError("Raw microservice had an unidentified error.") |
There was a problem hiding this comment.
If error is True, there will be a message element that gives more information (the exception that occurred in the server).
There was a problem hiding this comment.
Is message just a string? (An actual documented spec would be nice, BTW.)
Use of `moto.mock_aws` by itself can lead to authentication errors. The lsst.resources.s3utils.clean_test_environment_for_s3 context provides a clean slate to run in.
The mocks were previously only used for testing LSST cameras' sidecars, but upcoming HSC tests will test check_for_snap, which is bucket-based.
This decouples check_for_snap from the contents of activator.py. The behavior is unchanged. (Unfortunately, tbe boto3 API does not have a useful Bucket object, and making a proxy isn't worth it.)
check_for_snap is much more tightly coupled to the raws code (paths, parsing, etc.) than to the activator (bucket access). The activator is still responsible for initializing the S3 client.
The microservice tracks metadata for images that have been recently ingested, providing a backup for the bucket notifications.
The microservice maps metadata to filenames without requiring that the latter be a deterministic function of the former, which for Rubin instruments it isn't.
kfindeisen
force-pushed
the
tickets/DM-45005
branch
from
db85f4b
to
f47c4a4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for the raw image microservice added on DM-39022. The service is optional; if it is not initialized, the code will behave as it did previously.