Repeatability of Energy measurements

[adrianco]

Power consumption of a system is highly variable, and some attempts I know of to measure the increase in power due to presence of absence of a workload have found that it's hard to get a repeatable result. The power consumption of an "idle" system depends on how long it is idle, and is perturbed by cron jobs and other daemon activities like security scanners. When a workload is added to the system, the power increase depends on other factors like how hot the system is, and what other workloads are also running on the system. Measuring the same workload on a different system (of the same type) will also get a different result. This is a Gage R&R (repeatability and reproducibility) problem, and there are standard statistical techniques for evaluating how many trials should be made to establish whether there is a good quality measurement available.

[atg-abhishek]

Numbers will vary widely when you measure without following this Gage R&R methodology, especially relevant in our case since there is variability in when the measurement is made (e.g., as the system ramps up to an idle state, as the system gets to the stage of processing the workload.)

Depending on the system, they can have very different starting points (e.g., what is the idle state - there is some debate around that)

Collect a large number of measurements for what you define as the idle or busy state and then apply statistical analysis to get results. We want to enable both repeatability and reproducibility in the measurements.

Might not need to include in the spec, but this should be done for worked out case studies / examples.

References:

1. https://www.spec.org/sert/
2. https://asq.org/quality-resources/gage-repeatability

Maybe R&R should be valid for the entire SCI which would necessitate having R&R at the level of each of the components of the SCI as well. It might not apply for something like M but it will apply to things like E.

[atg-abhishek]

in the case study done by Gadhu, the data that was collected varied over time (time series for the SCI). Say, you get an SCI score every 15 minutes, you can see how it varies over time, but getting a single SCI number from that would be difficult. The way the calculation is done in this case is that Gadhu is taking a point-in-time value for all the measurements that feed into the SCI score, aligning with the granularity provided by WattTime which is 15-minutes at the highest level of granularity. To further clarify, instead of calculating an average of the other measurement values (except the WattTime API value), Gadhu has just picked up the value at 815 pm rather than computing some average or other collapsing measure for all the data between 800 and 814.

[chrisxie-fw]

Repeatability and reproducibility (R&R) of SCI score in general is critical because it demonstrates the reliability or applicability of the score in certain use cases, where the SCI score is used to measure any meaningful improvements of a software system. Therefore, I would suggest an R&R study on SCI.

[atg-abhishek]

Could this be something that is explored as a project under the aegis of the SWG?