Erratic Network Performance

[DanielFroehlich]

Some MicroShift Demos need to tranfer large amounts of data (>1.6G) over the network.
During demos, I noticed a wildly different performance of that part - sometimes its lightning fast, sometimes slow as hell.
Hence I did some measurements using ipferf3 running in sender/receiver pods across different nodes.
The throughput measurements range from 19 GBit/sec (from inf8 to inf8, local) to only 0.4 GBit/sec (inf7 -> in44).
CPU utlisation of the nodes might play a factor (keep in mind, >50% CPU utili tips over the phys. core threshold).

See the measurements results here: https://docs.google.com/spreadsheets/d/1Pd0i9BmikUEwIfqRuDbmh3Skklvu__CimsBamZYnraQ/edit#gid=0

Tooling Deployment can be seen here: https://console-openshift-console.apps.isar.coe.muc.redhat.com/k8s/ns/dfroehli-perftst/apps~v1~Deployment

[DanielFroehlich]

deploy-isar.zip
tooling deployment yaml

[DanielFroehlich]

@rbo would you mind to take a look please?

[rbo]

Server	Switch
ucs56	arista-rj45
ucs57	arista-rj45
inf8	arista-rj45
inf7	arista-rj45
inf44	arista-sfp

[rbo]

I can reproduct...

[rbo]

How to run the tests: https://github.com/openshift-examples/ipferf3

Feel free to reopen if needed

[DanielFroehlich]

I did a re-test inf44->inf7
With CPU load <50% of both nodes, I get ~5 g/s
With CPU load >50% (by running pods with "stress", burning CPU cores away, I get ~1.5 g/s

I think this rely relates to #178
and hyperthreading
@rbo wdyt?

[rbo]

@DanielFroehlich can you provide more information how to you run your tests?

$ oc get pods -o wide
NAME                      READY   STATUS              RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
iperf-hostnetwork-4f2zm   0/1     ContainerCreating   0          7s    10.32.96.8    inf8    <none>           <none>
iperf-hostnetwork-8ssf6   0/1     ContainerCreating   0          7s    10.32.96.44   inf44   <none>           <none>
iperf-hostnetwork-ctpjm   1/1     Running             0          7s    10.32.96.7    inf7    <none>           <none>
iperf-hostnetwork-k84jt   0/1     ContainerCreating   0          7s    10.32.96.57   ucs57   <none>           <none>
iperf-hostnetwork-nsxtx   0/1     ContainerCreating   0          7s    10.32.96.56   ucs56   <none>           <none>
$ oc rsh iperf-hostnetwork-8ssf6
sh-5.1$ iperf3 --client 10.32.96.7 --time 60 --parallel 4
Connecting to host 10.32.96.7, port 5201
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  13.2 GBytes  1.89 Gbits/sec  11456             sender
[  5]   0.00-60.04  sec  13.2 GBytes  1.88 Gbits/sec                  receiver
[  7]   0.00-60.00  sec  15.8 GBytes  2.26 Gbits/sec  13341             sender
[  7]   0.00-60.04  sec  15.8 GBytes  2.26 Gbits/sec                  receiver
[  9]   0.00-60.00  sec  16.0 GBytes  2.28 Gbits/sec  12367             sender
[  9]   0.00-60.04  sec  16.0 GBytes  2.28 Gbits/sec                  receiver
[ 11]   0.00-60.00  sec  13.2 GBytes  1.88 Gbits/sec  10697             sender
[ 11]   0.00-60.04  sec  13.2 GBytes  1.88 Gbits/sec                  receiver
[SUM]   0.00-60.00  sec  58.1 GBytes  8.32 Gbits/sec  47861             sender
[SUM]   0.00-60.04  sec  58.1 GBytes  8.31 Gbits/sec                  receiver

iperf Done.
sh-5.1$ exit
$ oc adm top node
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
inf4    2962m        19%    21437Mi         8%        
inf44   10873m       34%    69532Mi         38%       
inf5    1352m        8%     19752Mi         7%        
inf6    2653m        17%    23314Mi         9%        
inf7    9806m        15%    56408Mi         22%       
inf8    3005m        4%     45632Mi         18%       
ucs56   26146m       21%    227540Mi        22%       
ucs57   6724m        5%     104793Mi        10%  

[rbo]

Pods are still running in rbohne-iperf namesapce.

[DanielFroehlich]

Archive.zip
I am running ipferf and stress using the attached manifests.