Better Plotting of Performance History Data

[qinsoon]

Changes:

• Update ci-perf-kit to 0.3.5
  • plot 10-point moving average with performance history data
  • plot +-1 standard deviation from 10p moving average
  • highlight the max/min number in the graph
  • normalize all the data to the min value
  • log timeline
• Plots for each branch are now stored in separate directories
  • Plots for a branch can co-exist with the plots for master

Examples for the new graph:

• https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/jikesrvm_semispace_history.html
• https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/jikesrvm_nogc_history.html
• https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/openjdk_semispace_history.html
• https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/openjdk_nogc_history.html

[github-actions]

JikesRVM

• binding_trunk: 3be34fc6fe3c37942813cf8aa051513c1ed936dc
• trunk: 767ae600b13798cea527480921bababebe2bb450
• binding_branch: 3be34fc6fe3c37942813cf8aa051513c1ed936dc
• branch: 7c86a85561160245b9662ccdf6497c2e4cf7fd30

NoGC (wrench-2020-08-17-Mon-174657)

Benchmark	Trunk(ms)			Branch(ms)			Diff
	mean	mean without outliers	median	mean	mean without outliers	median	mean	mean without outliers
antlr	783.95 ±2.70	783.95 ±2.70	786.5	781.15 ±7.17	778.23 ±4.18 ⚠️ 1 removed	778.0	-0.36%	-0.73%
fop	623.90 ±1.40	623.54 ±1.22 ⚠️ 1 removed	623.0	621.92 ±1.41	621.92 ±1.41	621.5	-0.32%	-0.26%
luindex	2312.72 ±5.33	2311.41 ±4.74 ⚠️ 1 removed	2314.5	2306.55 ±6.50	2304.82 ±5.63 ⚠️ 1 removed	2309.0	-0.27%	-0.29%

SemiSpace (wrench-2020-08-17-Mon-175858)

Benchmark	Trunk(ms)			Branch(ms)			Diff
	mean	mean without outliers	median	mean	mean without outliers	median	mean	mean without outliers
antlr	647.88 ±6.16	645.79 ±4.61 ⚠️ 1 removed	644.0	647.40 ±4.77	647.40 ±4.77	646.0	-0.07%	+0.25%
bloat	2472.78 ±25.54	2472.78 ±25.54	2470.0	2471.78 ±26.47	2471.78 ±26.47	2497.5	-0.04%	-0.04%
fop	586.55 ±1.31	586.55 ±1.31	587.0	586.75 ±2.00	586.03 ±1.40 ⚠️ 1 removed	586.5	+0.03%	-0.09%
hsqldb	1127.75 ±15.82	1127.75 ±15.82	1154.5	1138.83 ±19.76	1132.85 ±16.05 ⚠️ 1 removed	1156.5	+0.98%	+0.45%
jython	1841.97 ±7.55	1841.97 ±7.55	1844.5	1832.95 ±7.95	1830.85 ±6.89 ⚠️ 1 removed	1833.5	-0.49%	-0.60%
luindex	2381.69 ±10.02 ⚠️ 1/40 failed	2381.69 ±10.02	2385.0	2375.18 ±8.38 ⚠️ 1/40 failed	2375.18 ±8.38	2378.0	-0.27%	-0.27%
lusearch	686.90 ±4.21	685.72 ±3.55 ⚠️ 1 removed	681.5	685.85 ±3.82 ⚠️ 1/40 failed	683.86 ±2.74 ⚠️ 2 removed	685.0	-0.15%	-0.27%
pmd	1510.53 ±11.01	1510.53 ±11.01	1506.5	1505.28 ±7.02	1505.28 ±7.02	1502.0	-0.35%	-0.35%
xalan	501.15 ±2.88 ⚠️ 1/40 failed	501.15 ±2.88	503.0	505.13 ±2.63 ⚠️ 1/40 failed	505.13 ±2.63	505.0	+0.79%	+0.79%

[github-actions]

OpenJDK

• binding_trunk: bd3d73582f7706f61b2efa15964d0f95c20612fc
• trunk: 767ae600b13798cea527480921bababebe2bb450
• binding_branch: bd3d73582f7706f61b2efa15964d0f95c20612fc
• branch: 7c86a85561160245b9662ccdf6497c2e4cf7fd30

SemiSpace (wrench-2020-08-17-Mon-194451)

Benchmark	Trunk(ms)			Branch(ms)			Diff
	mean	mean without outliers	median	mean	mean without outliers	median	mean	mean without outliers
antlr	408.59 ±6.66 ⚠️ 6/40 failed	408.59 ±6.66	409.5	411.03 ±7.23 ⚠️ 6/40 failed	411.03 ±7.23	407.0	+0.60%	+0.60%
eclipse	7827.77 ±32.05 ⚠️ 1/40 failed	7827.77 ±32.05	7824.0	7814.54 ±37.29 ⚠️ 1/40 failed	7814.54 ±37.29	7787.0	-0.17%	-0.17%
fop	465.38 ±2.54	465.38 ±2.54	461.5	466.20 ±3.82	464.95 ±2.94 ⚠️ 1 removed	462.0	+0.18%	-0.09%
hsqldb	1113.69 ±13.87 ⚠️ 1/40 failed	1113.69 ±13.87	1117.0	1102.08 ±17.59	1102.08 ±17.59	1113.5	-1.04%	-1.04% 🟩
pmd	1401.92 ±11.53	1401.92 ±11.53	1400.0	1411.22 ±11.72	1411.22 ±11.72	1405.5	+0.66%	+0.66%

[qinsoon]

After a discussion with Steve, Steve suggested some changes (mmtk/ci-perf-kit#14). I will address those changes before we merge this PR.

[qinsoon]

Suggested changes are made, and I bumped ci-perf-kit to 0.3.5.

Can at least one of you review this PR and approve it? @steveblackburn @caizixian @wenyuzhao

[caizixian]

I'm looking at https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/jikesrvm_semispace_history.html , and it doesn't seem to render properly on Firefox 80.0/Chrome 83.0.4103.116 on macOS

[wenyuzhao]

I'm looking at https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/jikesrvm_semispace_history.html , and it doesn't seem to render properly on Firefox 80.0/Chrome 83.0.4103.116 on macOS

I think it's because the variance of some benchmark is too large, so the extra height for the subplot overlaps with others.

[qinsoon]

I'm looking at https://mmtk.github.io/ci-perf-result/refs_pull_125_merge/jikesrvm_semispace_history.html , and it doesn't seem to render properly on Firefox 80.0/Chrome 83.0.4103.116 on macOS

Yeah, the graph was rendered from the test branch of the result repository (https://github.com/mmtk/ci-perf-result/tree/test). It includes data from the Github hosted runner and our self hosted runner so the results vary a lot. The label of the subgraph below overlaps the the subgraph above. This should not happen if we have a 'normal' graph.