Added Eco-CI

[ArneTR]

Description

This pull requests is a bit out of the ordinary ... I hope :)

I mainly follows up on a discussion with the CNCF TAG ENV where @wbeckler encouraged us to integrate to make a PR here. But now for the details:

I work for an open source company called Green Coding Solutions in Germany and we started with the mission to increase awareness and actionability around digital CO2 emissions.

We have recently created an open source CI/CD plugin called Eco-CI.

• Project page
• Github Page

It integrates into the Github Actions pipeline and estimates the energy and CO2 consumption of the pipeline by utilizing a Machine Learning model trained on real server energy data from SPECpower

The tool creates awareness of the energy cost and carbon emissions of CI/CD pipelines and empoweres developers to create action for more sustainability.
It can separate steps of the pipeline and create detailed drill-downs which step has increased / decreased in carbon emissions and provider great insights into pipeline performance in general.

Since OpenSearch is a CNCF project and some members from your TAG ENV are active here I hope you find it adjacent to your work :)

In this PR I have made a sample integration into the .github/workflows/assemble.yml pipeline. It will show the current pipline energy cost and CO2 emissions (only for linux though atm) and will also provide an aggregate view on https://metrics.green-coding.io/ci-index.html

I hope this PR and the information that Eco-CI provides is interesting for you and I am super interested in your feedback on it.

Changelog

• Added Eco-CI into the .github/workflows/assemble.yml pipeline
• Activated the send-data: true flag. This will send data to be used as historical data view on https://metrics.green-coding.io/ci-index.html . If this is not desired it must be set to false.
• Nothing to be documented in the OpenSearch Changelog as it is only a CI change

Demo

• An example of where we have run it before making the PR in the forked repository to verify that the pipeline works

  • https://github.com/green-coding-solutions/OpenSearch/actions/runs/9566625147 (Since your pipeline skips some workflows it shows this status. Linux ran fine though)

• A possible upgrade could also be to show the energy and CO2 cost in every PR as message. See an example here: Bump psutil from 5.9.8 to 6.0.0 green-coding-solutions/green-metrics-tool#816 (comment)

  • Since this requires to set an additional permission to the workflow I omitted this for now. Let me know if you want to bring this in

@ribalba @MichelleGruene @wbeckler @by-d-sign @leonardpahlke @nikimanoledaki

Related Issues

• None

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
I have also signed my commit.

[dblock]

I think it's a great initiative, but I am a little concerned that in order to actually properly do this we'll have to add this kind of code in dozens (hundreds?) of .github workflows. The task being watched (./gradlew assemble) is a tiny portion of the build process.

I also saw a request to get org-wide access to a bot, which I don't think we should be taking lightly, cc: @sohami.

I believe an acceptable solution would be to enable a bot once for the entire org, and for it to automatically collect CI/CD data. We also have a very large (much larger than GHA) Jenkins instance for integration tests.

Since OpenSearch is a CNCF project

This is not the case, btw.

[ArneTR]

Hey @dblock

oh my, sry for the misunderstanding about the CNCF project. I got the pointers in a CNCF thread and then made a quick google to confirm and this page seem sufficient for me to make this claim. My bad ... but I assume no harm done :)

2. I do agree that bringing this to all your flows would involve a bit of manual copy & paste. My idea was mainly not to fully account for all workflows in the project but rather bring this concept into your project and then maybe take next steps from there.

Sady GitHub provides no easier integration or allows for encapsulating a whole workflow (other than self hosted runners of course). But as said: This PR is with the intention to bring sustainability / carbon awareness in and then see from there.

3. I am not sure what you mean regarding org wide bot accesss ... are you referring to the permissions I mentioned regarding pull_requests: write ? This I only mentioned. I did not add this to the PR to keep it minimal. Hope that clarifies things ... please follow up if I misunderstood.

In any case: Super happy that you consider it and happy to talk further. Already a win to understand your needs and concerrns.

[dblock]

I'd like to hear from other maintainers what they think about this and see if anyone is passionate about adding this workflow.

In my opinion this is not ready to merge. As written this PR doesn't actually "estimates the energy and CO2 consumption of the pipeline" because it covers a very small part of the pipeline. You'd have to modify all the workflows here. Then, wrapping every single workflow with two dozen lines of "start measure/stop measure" is a lot of noise for already complicated-enough workflows. For something like this I do think you want an org-wide, set it and forget it, bot, not modifying every workflow.

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[dblock]

Going to close, see above.