DM-40823: Explore methods for calculating Prompt Processing preload timing #107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kfindeisen
force-pushed
the
tickets/DM-40823
branch
3 times, most recently
from
7973d41
to
3415e88
Compare
kfindeisen
force-pushed
the
tickets/DM-40823
branch
3 times, most recently
from
865fa64
to
b03247e
Compare
kfindeisen
force-pushed
the
tickets/DM-40823
branch
2 times, most recently
from
029f4e3
to
9f252c8
Compare
kfindeisen
force-pushed
the
tickets/DM-40823
branch
from
9f252c8
to
bf4c32e
Compare
hsinfang
approved these changes
Feb 1, 2024
This log helps clarify when the Flask app is ready to take requests.
The timer uses time_this instead of logInfo because raw timestamps are almost impossible to work with in Loki (in particular, they can't be parsed into something that allows arithmetic operations).
Timing of export_outputs, clean_local_repo, make_quantum_graph, and _filter_calibs_by_date should pin down the remaining I/O bottlenecks.
This context manager allows streamlined timing of arbitrary code blocks, placing the results in a MetricMeasurementBundle for upload to Sasquatch.
In light of Sasquatch's strict compatibility requirements, the metrics are limited to three times that will almost certainly survive future implementation changes: TotalTime (the running time of prep_butler, excluding the timing infrastructure itself), SearchTime (the time to identify the datasets to import), and TransferTime (the time to actually update the local repo's state).
The enforce_schema function allows the removal of timing metrics in the future without breaking Sasquatch rules. It's unlikely that this function will ever be ported to analysis_tools, as it's a workaround for the absence of a coherent timing action.
Configuring a dispatcher directly lets us sidestep the problem of using the Butler, and also gives us full control over the metadata associated with the upload.
Authentication is not yet supported server-side, but defining an environment variable will let us support it through the usual secrets mechanisms.
kfindeisen
force-pushed
the
tickets/DM-40823
branch
from
bf4c32e
to
eef4760
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds two timing frameworks to Prompt Processing. The first, based on the logger, is flexible and easy to modify, but makes it hard to track long-term changes. The second, based on a hybrid of
lsst.verifyandanalysis_tools, allows upload of timing metrics to Sasquatch, but individual metrics must be added and removed with care.