Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scaffolding for metriq-client integration #54

Open
vprusso opened this issue Dec 3, 2024 · 0 comments
Open

Scaffolding for metriq-client integration #54

vprusso opened this issue Dec 3, 2024 · 0 comments
Labels
feature New functionality

Comments

@vprusso
Copy link
Contributor

vprusso commented Dec 3, 2024

The full scope of the metriq-client integration (i.e. the ability to dispatch and update Metriq.info) is likely beyond this one issue and will almost surely change. Still, the general structure and scaffolding necessary to be in place to support this workflow can be achieved.

The content below is extracted from the discussion in #32:

We don't have much in the way of Metriq and Metriq-client integration, but we will likely want to subdivide some of the "client-specific" components into its directory. Something like:

metriq_gym/
    client/
        __init__.py
        metriq_metadata.py
        metriq_upload_result.py

Where:

  • metriq_upload_result.py: Taking a BenchJobResult and associated Metriq submission information and dissecting the relevant components to be uploaded to Metriq.info.

  • metriq_metadata.py contains the mapping of submission, result, etc. IDs on Metriq.info to specific benchmarks that we run. This is not exhaustive and needs to be updated as new results, submissions, etc., get added to the platform.

@vprusso vprusso added the feature New functionality label Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
Development

No branches or pull requests

1 participant