Incorrect Read (Avg) calculation in Local mining stats for coordinated mining

[kslazarev]

I've noticed an issue with the Read (Avg) calculation in the Local mining stats for coordinated mining. The Read (Avg) values seem to be incorrectly calculated across all metrics, while the Hash (Avg) appears to be correct (proportional Data Size).

Here are the stats from three nodes in our coordinated mining setup:

Node1 (exit_node):

• Data Size: 57.3 TiB
• Partitions: 20
• Read (Avg): 3829.9 MiB/s (expected 3482 MiB/s)
• Hash (Avg): 5046 h/s

Local mining stats:
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
| Partition | Data Size | % of Max |   Read (Cur)  |   Read (Avg)  |  Read (Ideal) | Hash (Cur) | Hash (Avg) | Hash (Ideal) |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
|     Total |  57.3 TiB |     30 % |  4457.2 MiB/s |  3829.9 MiB/s |  2056.2 MiB/s |   5616 h/s |   5046 h/s |     1989 h/s |
|         1 |   1.7 TiB |     52 % |    87.4 MiB/s |   109.6 MiB/s |    62.2 MiB/s |    182 h/s |    146 h/s |       60 h/s |
|         4 |   2.6 TiB |     78 % |   299.3 MiB/s |   176.1 MiB/s |    92.1 MiB/s |    335 h/s |    225 h/s |       89 h/s |
|         8 |   2.3 TiB |     70 % |   351.6 MiB/s |   163.4 MiB/s |    83.2 MiB/s |    574 h/s |    205 h/s |       80 h/s |
|         9 |   2.6 TiB |     79 % |   210.6 MiB/s |   182.3 MiB/s |    93.0 MiB/s |    196 h/s |    230 h/s |       90 h/s |
|        13 |   3.3 TiB |     99 % |   219.8 MiB/s |   226.9 MiB/s |   117.0 MiB/s |    249 h/s |    285 h/s |      113 h/s |
|        14 |   3.2 TiB |     99 % |   139.9 MiB/s |   186.8 MiB/s |   116.4 MiB/s |    124 h/s |    286 h/s |      112 h/s |
|        17 |   3.2 TiB |     99 % |   328.4 MiB/s |   229.2 MiB/s |   116.4 MiB/s |    472 h/s |    295 h/s |      112 h/s |
|        19 |   2.8 TiB |     86 % |   270.8 MiB/s |   195.4 MiB/s |   102.0 MiB/s |    336 h/s |    249 h/s |       98 h/s |
|        24 |   3.2 TiB |     98 % |   195.2 MiB/s |   221.9 MiB/s |   115.3 MiB/s |    322 h/s |    281 h/s |      111 h/s |
|        26 |   3.2 TiB |     98 % |   229.4 MiB/s |   216.0 MiB/s |   115.9 MiB/s |    285 h/s |    279 h/s |      112 h/s |
|        27 |   3.2 TiB |     98 % |   189.8 MiB/s |   218.7 MiB/s |   115.8 MiB/s |    276 h/s |    288 h/s |      112 h/s |
|        30 |   1.9 TiB |     58 % |   154.2 MiB/s |   130.5 MiB/s |    68.6 MiB/s |    196 h/s |    168 h/s |       66 h/s |
|        32 |   2.1 TiB |     65 % |   330.5 MiB/s |   146.3 MiB/s |    76.4 MiB/s |    228 h/s |    186 h/s |       73 h/s |
|        33 |   2.8 TiB |     85 % |   162.8 MiB/s |   192.6 MiB/s |    99.9 MiB/s |    123 h/s |    243 h/s |       96 h/s |
|        34 |   3.2 TiB |     96 % |   209.7 MiB/s |   216.3 MiB/s |   113.4 MiB/s |    225 h/s |    277 h/s |      109 h/s |
|        38 |   3.1 TiB |     94 % |   279.9 MiB/s |   217.1 MiB/s |   111.3 MiB/s |    269 h/s |    276 h/s |      107 h/s |
|        45 |   3.3 TiB |     99 % |   169.7 MiB/s |   218.9 MiB/s |   116.8 MiB/s |    247 h/s |    287 h/s |      113 h/s |
|        46 |   3.1 TiB |     94 % |   148.4 MiB/s |   167.5 MiB/s |   111.2 MiB/s |    339 h/s |    261 h/s |      107 h/s |
|        50 |   3.2 TiB |     98 % |   159.9 MiB/s |   201.1 MiB/s |   116.0 MiB/s |    164 h/s |    286 h/s |      112 h/s |
|        51 |   3.2 TiB |     96 % |   319.9 MiB/s |   213.2 MiB/s |   113.3 MiB/s |    464 h/s |    283 h/s |      109 h/s |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+

Coordinated mining cluster stats:
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                 Peer | H1 Out (Cur) | H1 Out (Avg) | H1 In (Cur) | H1 In (Avg) | H2 Out |  H2 In |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                  All |     3861 h/s |     3983 h/s |    4016 h/s |    3392 h/s |      0 |      0 |
|     95.54.82.50:1986 |     2272 h/s |     2218 h/s |    2035 h/s |    1560 h/s |      0 |      0 |
|     95.54.82.50:1985 |     1589 h/s |     1765 h/s |    1981 h/s |    1831 h/s |      0 |      0 |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+

Node2:

• Data Size: 59.1 TiB
• Partitions: 20
• Read (Avg): 3115.4 MiB/s (expected 3594 MiB/s)
• Hash (Avg): 5394 h/s

Local mining stats:
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
| Partition | Data Size | % of Max |   Read (Cur)  |   Read (Avg)  |  Read (Ideal) | Hash (Cur) | Hash (Avg) | Hash (Ideal) |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
|     Total |  59.1 TiB |     31 % |  3182.4 MiB/s |  3115.4 MiB/s |  2356.4 MiB/s |   5391 h/s |   5394 h/s |     2332 h/s |
|         0 |   1.9 TiB |     57 % |    88.3 MiB/s |   109.1 MiB/s |    75.4 MiB/s |    148 h/s |    172 h/s |       74 h/s |
|         2 |   2.0 TiB |     61 % |   108.3 MiB/s |   119.7 MiB/s |    80.3 MiB/s |    131 h/s |    181 h/s |       79 h/s |
|         3 |   2.7 TiB |     81 % |   112.5 MiB/s |   128.7 MiB/s |   106.1 MiB/s |    202 h/s |    242 h/s |      105 h/s |
|         7 |   2.4 TiB |     73 % |   151.2 MiB/s |   141.2 MiB/s |    95.6 MiB/s |    275 h/s |    224 h/s |       94 h/s |
|        10 |   3.0 TiB |     91 % |   217.9 MiB/s |   176.1 MiB/s |   120.0 MiB/s |    240 h/s |    280 h/s |      118 h/s |
|        11 |   3.2 TiB |     97 % |   138.4 MiB/s |   140.2 MiB/s |   126.7 MiB/s |    246 h/s |    291 h/s |      125 h/s |
|        18 |   3.2 TiB |     99 % |   156.5 MiB/s |   175.8 MiB/s |   129.4 MiB/s |    364 h/s |    296 h/s |      128 h/s |
|        20 |   3.2 TiB |     97 % |   168.5 MiB/s |   169.0 MiB/s |   127.4 MiB/s |    280 h/s |    290 h/s |      126 h/s |
|        22 |   3.2 TiB |     98 % |   195.3 MiB/s |   186.9 MiB/s |   128.6 MiB/s |    263 h/s |    299 h/s |      127 h/s |
|        25 |   3.2 TiB |     97 % |   148.6 MiB/s |   140.3 MiB/s |   127.7 MiB/s |    393 h/s |    285 h/s |      126 h/s |
|        28 |   3.2 TiB |     99 % |   228.1 MiB/s |   189.4 MiB/s |   129.2 MiB/s |    210 h/s |    304 h/s |      127 h/s |
|        29 |   3.2 TiB |     98 % |   185.9 MiB/s |   171.9 MiB/s |   128.1 MiB/s |    315 h/s |    290 h/s |      126 h/s |
|        31 |   2.0 TiB |     61 % |   111.3 MiB/s |   106.3 MiB/s |    80.5 MiB/s |    226 h/s |    182 h/s |       79 h/s |
|        36 |   3.2 TiB |     98 % |   138.5 MiB/s |   167.3 MiB/s |   128.7 MiB/s |    201 h/s |    284 h/s |      127 h/s |
|        40 |   3.3 TiB |     99 % |   158.7 MiB/s |   169.4 MiB/s |   129.7 MiB/s |    336 h/s |    303 h/s |      128 h/s |
|        41 |   3.2 TiB |     99 % |   158.6 MiB/s |   168.9 MiB/s |   129.3 MiB/s |    353 h/s |    295 h/s |      127 h/s |
|        42 |   3.2 TiB |     99 % |   222.1 MiB/s |   188.7 MiB/s |   129.3 MiB/s |    416 h/s |    295 h/s |      128 h/s |
|        49 |   3.2 TiB |     98 % |   168.6 MiB/s |   168.7 MiB/s |   128.4 MiB/s |    173 h/s |    296 h/s |      127 h/s |
|        52 |   3.2 TiB |     98 % |   157.6 MiB/s |   153.1 MiB/s |   128.9 MiB/s |    283 h/s |    288 h/s |      127 h/s |
|        53 |   3.2 TiB |     97 % |   167.6 MiB/s |   144.9 MiB/s |   127.3 MiB/s |    328 h/s |    288 h/s |      126 h/s |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+

Coordinated mining cluster stats:
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                 Peer | H1 Out (Cur) | H1 Out (Avg) | H1 In (Cur) | H1 In (Avg) | H2 Out |  H2 In |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                  All |     3288 h/s |     3420 h/s |    4015 h/s |    3885 h/s |      0 |      0 |
|     95.54.82.50:1985 |     1428 h/s |     1519 h/s |    1779 h/s |    1788 h/s |      0 |      0 |
|     95.54.82.50:1984 |     1859 h/s |     1900 h/s |    2235 h/s |    2096 h/s |      0 |      0 |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+

Node3:

• Data Size: 50.7 TiB
• Partitions: 16
• Read (Avg): 3333.9 MiB/s (expected 3078 MiB/s)
• Hash (Avg): 4472 h/s

Local mining stats:
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
| Partition | Data Size | % of Max |   Read (Cur)  |   Read (Avg)  |  Read (Ideal) | Hash (Cur) | Hash (Avg) | Hash (Ideal) |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+
|     Total |  50.7 TiB |     27 % |  3478.8 MiB/s |  3333.9 MiB/s |  1965.0 MiB/s |   4609 h/s |   4472 h/s |     1734 h/s |
|         5 |   2.8 TiB |     83 % |   230.0 MiB/s |   178.4 MiB/s |   106.6 MiB/s |    276 h/s |    240 h/s |       94 h/s |
|         6 |   2.8 TiB |     85 % |   185.7 MiB/s |   198.0 MiB/s |   109.0 MiB/s |    235 h/s |    249 h/s |       96 h/s |
|        12 |   3.2 TiB |     98 % |   179.2 MiB/s |   208.8 MiB/s |   125.1 MiB/s |    250 h/s |    283 h/s |      110 h/s |
|        15 |   3.3 TiB |     99 % |   169.6 MiB/s |   195.8 MiB/s |   126.3 MiB/s |    395 h/s |    286 h/s |      111 h/s |
|        16 |   3.2 TiB |     98 % |   208.8 MiB/s |   227.4 MiB/s |   125.5 MiB/s |    278 h/s |    288 h/s |      110 h/s |
|        21 |   3.3 TiB |     99 % |   219.8 MiB/s |   225.4 MiB/s |   126.2 MiB/s |    256 h/s |    291 h/s |      111 h/s |
|        23 |   3.2 TiB |     97 % |   228.5 MiB/s |   216.1 MiB/s |   123.4 MiB/s |    386 h/s |    277 h/s |      108 h/s |
|        35 |   3.2 TiB |     96 % |   235.7 MiB/s |   214.0 MiB/s |   122.5 MiB/s |    324 h/s |    277 h/s |      108 h/s |
|        37 |   3.2 TiB |     98 % |   249.7 MiB/s |   199.3 MiB/s |   125.0 MiB/s |    376 h/s |    287 h/s |      110 h/s |
|        39 |   3.3 TiB |     99 % |   189.6 MiB/s |   227.0 MiB/s |   126.6 MiB/s |    176 h/s |    287 h/s |      111 h/s |
|        43 |   3.3 TiB |     99 % |   139.9 MiB/s |   227.8 MiB/s |   126.8 MiB/s |    123 h/s |    292 h/s |      111 h/s |
|        44 |   3.2 TiB |     98 % |   339.5 MiB/s |   217.3 MiB/s |   125.6 MiB/s |    287 h/s |    285 h/s |      110 h/s |
|        47 |   3.1 TiB |     95 % |   263.6 MiB/s |   213.8 MiB/s |   121.7 MiB/s |    166 h/s |    275 h/s |      107 h/s |
|        48 |   3.3 TiB |     99 % |   189.8 MiB/s |   189.3 MiB/s |   126.1 MiB/s |    496 h/s |    287 h/s |      111 h/s |
|        54 |   3.3 TiB |     99 % |   239.8 MiB/s |   196.3 MiB/s |   126.4 MiB/s |    395 h/s |    290 h/s |      111 h/s |
|        55 |   3.1 TiB |     96 % |   209.8 MiB/s |   199.2 MiB/s |   122.1 MiB/s |    182 h/s |    271 h/s |      107 h/s |
+-----------+-----------+----------+---------------+---------------+---------------+------------+------------+--------------+

Coordinated mining cluster stats:
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                 Peer | H1 Out (Cur) | H1 Out (Avg) | H1 In (Cur) | H1 In (Avg) | H2 Out |  H2 In |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+
|                  All |     3670 h/s |     3775 h/s |    3224 h/s |    3042 h/s |      0 |      0 |
|     95.54.82.50:1986 |     1673 h/s |     1889 h/s |    1695 h/s |    1283 h/s |      0 |      0 |
|     95.54.82.50:1984 |     1997 h/s |     1885 h/s |    1528 h/s |    1759 h/s |      0 |      0 |
+----------------------+--------------+--------------+-------------+-------------+--------+--------+

As you can see, the Read (Avg) values don't match the expected values, while the Hash (Avg) seems to be correct. This discrepancy is consistent across all nodes in our setup.

Steps to reproduce:

1. Set up a coordinated mining environment with multiple nodes
2. Run the mining process for a sufficient period to gather stable metrics
3. Compare the reported Read (Avg) values with the expected values based on the node's specifications

Expected behavior:
The Read (Avg) values should closely match the expected values for each node, similar to how the Hash (Avg) values appear to be correct.

Actual behavior:
The Read (Avg) values are significantly different from the expected values, while Hash (Avg) values seem to be correct.

Additional information:

• Arweave version: master 55dbf97
• Operating system: ubuntu 22.04.3 lts

[JamesPiechota]

Yeah. I believe we're incorrectly reporting cached reads as reads. Peers batch H1 jobs and sometimes the same recall range will be represented in multiple batches. In those cases we cache the read to avoid going to disk repeatedly. The cached reads probably should not be shown in the report.

[bgmastermind]

Or they should as a separate stat named cache hit %