Skip to content

Commit

Permalink
ass envoy benchmark
Browse files Browse the repository at this point in the history 
Change-Id: I619d6e3d1c483cb48edcec50b024e3b246839c0d
  • Loading branch information
xiaosong yang committed Aug 28, 2024
1 parent 4e2a9b1 commit 71c6e50
Show file tree
Hide file tree
Showing 34 changed files with 398 additions and 130 deletions.
115 changes: 74 additions & 41 deletions benchmark/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ mkdir examples/certs && openssl req -x509 -newkey rsa:2048 -keyout examples/cert

# build monolake
cd $MONOLAKE_HOME
cargo build
cargo build --release
```

#### proxy/nginx/setup-once.sh
Expand Down Expand Up @@ -216,11 +216,11 @@ Some steps are manual steps.
* Update server-ip in the configuration files: running benchmark/proxy/update-server-ip.sh and follow the sed commands

```bash
export client_url=3.133.229.116
export proxy_url=18.217.152.113
export server_url=3.22.140.218
export proxy_private_url=172.31.2.253
export server_private_url=172.31.22.170
export client_url=<client-eip>
export proxy_url=<proxy-eip>
export server_url=<server-eip>
export proxy_private_url=<proxy-private-ip>
export server_private_url=<server-private-ip>
```

Then user can use benchmark-pipeline.sh to run all test in one script. User may need type "exit" to quit some finished jobs and go to the next step. User can early input "exit" when "Writing data to CSV file wrk-performance.csv..." prompts. Finally, user will get the results and visualized images.
Expand All @@ -231,11 +231,11 @@ Pipeline scripts contain plot scripts, so it is better to run on a host machine
# new_terminal=`osascript -e 'tell app "Terminal" to do script $1'`
# new_terminal='gnome-terminal -- $1'

export client_url=3.133.229.116
export proxy_url=18.217.152.113
export server_url=3.22.140.218
export proxy_private_url=172.31.2.253
export server_private_url=172.31.22.170
export client_url=<client-eip>
export proxy_url=<proxy-eip>
export server_url=<server-eip>
export proxy_private_url=<proxy-private-ip>
export server_private_url=<server-private-ip>

# manual update proxy configurations
echo "make sure proxy configurations are updated manually"
Expand Down Expand Up @@ -302,6 +302,24 @@ echo "stop proxy monolake"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url} -t 'cd ~/monolake/benchmark/proxy; ./stop-monolake.sh'
sleep 2

# then start proxy envoy
echo "start proxy envoy"
osascript -e 'tell app "Terminal" to do script "~/code/monolake/benchmark/pipeline-proxy-envoy.sh"'
sleep 5

# start client
echo "start client"
osascript -e 'tell app "Terminal" to do script "~/code/monolake/benchmark/pipeline-client-envoy.sh"'
sleep 2

echo "start client-metrics-collect"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${client_url} -t 'cd monolake/benchmark; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; echo "Please type exit to continue"; bash -l'

#stop proxy envoy
echo "stop proxy envoy"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url} -t 'cd ~/monolake/benchmark/proxy; ./stop-envoy.sh'
sleep 2

#stop server
echo "stop server"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${server_url} -t 'sudo service nginx stop'
Expand All @@ -324,22 +342,27 @@ echo "copy collected data from proxy"
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:nginx-performance.csv .
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:traefik-performance.csv .
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:monolake-performance.csv .
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:envoy-performance.csv .

#plot data
echo "plot data"
./performance-plot.sh nginx
./performance-plot.sh traefik
./performance-plot.sh monolake
./performance-plot.sh envoy
./performance-plot.sh server
./performance-plot.sh wrk
./nginx-http-latency-plot.sh
./traefik-http-latency-plot.sh
./monolake-http-latency-plot.sh
./envoy-http-latency-plot.sh
./all-http-latency-plot.sh
./nginx-https-latency-plot.sh
./traefik-https-latency-plot.sh
./monolake-https-latency-plot.sh
./envoy-https-latency-plot.sh
./all-https-latency-plot.sh
./proxies-performance-plot.sh
```

The visualized result example:
Expand All @@ -361,30 +384,38 @@ Throughput and requests per second compare:

| Case | Requests/sec | Transfer/sec | Server Error | Timeout |
|---------------------------------|--------------|---------------|--------------|---------|
| http-result-4c-monolake-tiny | 160969.16 | 54892953.60 | 0 | 0 |
| http-result-4c-monolake-small | 154858.08 | 539523809.28 | 0 | 0 |
| http-result-4c-monolake-medium | 90215.55 | 1030792151.04 | 0 | 0 |
| http-result-4c-monolake-large | 9190.98 | 1524713390.08 | 0 | 374 |
| http-result-4c-nginx-tiny | 190923.94 | 69688360.96 | 0 | 0 |
| http-result-4c-nginx-small | 179160.28 | 628495482.88 | 0 | 0 |
| http-result-4c-nginx-medium | 114646.49 | 1320702443.52 | 0 | 0 |
| http-result-4c-nginx-large | 9306.86 | 1546188226.56 | 0 | 29 |
| http-result-4c-traefik-tiny | 9985.18 | 3407872.00 | 0 | 0 |
| http-result-4c-traefik-small | 5133.71 | 17720934.40 | 3016 | 326 |
| http-result-4c-traefik-medium | 11988.72 | 137688514.56 | 0 | 0 |
| http-result-4c-traefik-large | 9078.24 | 1503238553.60 | 0 | 117 |
| https-result-4c-monolake-tiny | 143351.22 | 48884613.12 | 0 | 0 |
| https-result-4c-monolake-small | 121503.00 | 423320616.96 | 0 | 0 |
| https-result-4c-monolake-medium | 67396.11 | 774048317.44 | 0 | 0 |
| https-result-4c-monolake-large | 8023.95 | 1331439861.76 | 0 | 0 |
| https-result-4c-nginx-tiny | 130013.41 | 47458549.76 | 0 | 0 |
| https-result-4c-nginx-small | 117888.48 | 413547888.64 | 0 | 0 |
| https-result-4c-nginx-medium | 66158.46 | 761412976.64 | 0 | 0 |
| https-result-4c-nginx-large | 8040.46 | 1331439861.76 | 0 | 0 |
| https-result-4c-traefik-tiny | 9927.86 | 3386900.48 | 0 | 0 |
| https-result-4c-traefik-small | 11931.26 | 41565552.64 | 0 | 0 |
| https-result-4c-traefik-medium | 13899.86 | 159645696.00 | 0 | 0 |
| https-result-4c-traefik-large | 7787.11 | 1288490188.80 | 0 | 0 |
| http-result-4c-monolake-tiny | 154431.05 | 52659486.72 | 0 | 0 |
| http-result-4c-monolake-small | 145974.68 | 508580331.52 | 0 | 0 |
| http-result-4c-monolake-medium | 85856.89 | 987842478.08 | 0 | 0 |
| http-result-4c-monolake-large | 9185.21 | 1524713390.08 | 0 | 381 |
| http-result-4c-nginx-tiny | 183684.88 | 67045949.44 | 0 | 0 |
| http-result-4c-nginx-small | 176493.90 | 619142184.96 | 0 | 0 |
| http-result-4c-nginx-medium | 110590.35 | 1277752770.56 | 0 | 0 |
| http-result-4c-nginx-large | 9300.72 | 1546188226.56 | 0 | 24 |
| http-result-4c-traefik-tiny | 9984.86 | 3407872.00 | 0 | 0 |
| http-result-4c-traefik-small | 10992.20 | 38293995.52 | 0 | 0 |
| http-result-4c-traefik-medium | 11981.44 | 137604628.48 | 0 | 0 |
| http-result-4c-traefik-large | 9068.24 | 1503238553.60 | 0 | 7 |
| http-result-4c-envoy-tiny | 37543.33 | 13820231.68 | 0 | 0 |
| http-result-4c-envoy-small | 36367.61 | 127685099.52 | 0 | 0 |
| http-result-4c-envoy-medium | 31010.91 | 357008670.72 | 0 | 0 |
| http-result-4c-envoy-large | 9307.30 | 1546188226.56 | 0 | 0 |
| https-result-4c-monolake-tiny | 140520.84 | 47919923.20 | 0 | 0 |
| https-result-4c-monolake-small | 115614.24 | 402799984.64 | 0 | 0 |
| https-result-4c-monolake-medium | 63085.81 | 724545044.48 | 0 | 0 |
| https-result-4c-monolake-large | 8051.51 | 1342177280.00 | 0 | 2 |
| https-result-4c-nginx-tiny | 126128.05 | 46032486.40 | 0 | 0 |
| https-result-4c-nginx-small | 115107.62 | 403796131.84 | 0 | 0 |
| https-result-4c-nginx-medium | 63612.32 | 732115763.20 | 0 | 0 |
| https-result-4c-nginx-large | 8020.73 | 1331439861.76 | 0 | 3 |
| https-result-4c-traefik-tiny | 9938.96 | 3386900.48 | 0 | 0 |
| https-result-4c-traefik-small | 11888.26 | 41418752.00 | 0 | 0 |
| https-result-4c-traefik-medium | 13911.54 | 159771525.12 | 0 | 0 |
| https-result-4c-traefik-large | 7960.72 | 1320702443.52 | 0 | 0 |
| https-result-4c-envoy-tiny | 34888.23 | 12845056.00 | 0 | 0 |
| https-result-4c-envoy-small | 33313.77 | 116968652.80 | 0 | 0 |
| https-result-4c-envoy-medium | 27510.93 | 316722380.80 | 0 | 0 |
| https-result-4c-envoy-large | 8453.53 | 1406601789.44 | 0 | 0 |

![proxies-performance](images/README/proxies-performance.png)

Expand All @@ -393,11 +424,13 @@ Throughput and requests per second compare by payload size:

| Case | Tiny Requests/sec | Small Requests/sec | Medium Requests/sec | Large Requests/sec | Tiny Transfer/sec | Small Transfer/sec | Medium Transfer/sec | Large Transfer/sec |
|--------------------------|-------------------|--------------------|---------------------|--------------------|-------------------|--------------------|---------------------|--------------------|
| http-result-4c-monolake | 160969.16 | 154858.08 | 90215.55 | 9190.98 | 54892953.60 | 539523809.28 | 1030792151.04 | 1524713390.08 |
| http-result-4c-nginx | 190923.94 | 179160.28 | 114646.49 | 9306.86 | 69688360.96 | 628495482.88 | 1320702443.52 | 1546188226.56 |
| http-result-4c-traefik | 9985.18 | 5133.71 | 11988.72 | 9078.24 | 3407872.00 | 17720934.40 | 137688514.56 | 1503238553.60 |
| https-result-4c-monolake | 143351.22 | 121503.00 | 67396.11 | 8023.95 | 48884613.12 | 423320616.96 | 774048317.44 | 1331439861.76 |
| https-result-4c-nginx | 130013.41 | 117888.48 | 66158.46 | 8040.46 | 47458549.76 | 413547888.64 | 761412976.64 | 1331439861.76 |
| https-result-4c-traefik | 9927.86 | 11931.26 | 13899.86 | 7787.11 | 3386900.48 | 41565552.64 | 159645696.00 | 1288490188.80 |
| http-result-4c-monolake | 154431.05 | 145974.68 | 85856.89 | 9185.21 | 52659486.72 | 508580331.52 | 987842478.08 | 1524713390.08 |
| http-result-4c-nginx | 183684.88 | 176493.90 | 110590.35 | 9300.72 | 67045949.44 | 619142184.96 | 1277752770.56 | 1546188226.56 |
| http-result-4c-traefik | 9984.86 | 10992.20 | 11981.44 | 9068.24 | 3407872.00 | 38293995.52 | 137604628.48 | 1503238553.60 |
| http-result-4c-envoy | 37543.33 | 36367.61 | 31010.91 | 9307.30 | 13820231.68 | 127685099.52 | 357008670.72 | 1546188226.56 |
| https-result-4c-monolake | 140520.84 | 115614.24 | 63085.81 | 8051.51 | 47919923.20 | 402799984.64 | 724545044.48 | 1342177280.00 |
| https-result-4c-nginx | 126128.05 | 115107.62 | 63612.32 | 8020.73 | 46032486.40 | 403796131.84 | 732115763.20 | 1331439861.76 |
| https-result-4c-traefik | 9938.96 | 11888.26 | 13911.54 | 7960.72 | 3386900.48 | 41418752.00 | 159771525.12 | 1320702443.52 |
| https-result-4c-envoy | 34888.23 | 33313.77 | 27510.93 | 8453.53 | 12845056.00 | 116968652.80 | 316722380.80 | 1406601789.44 |

![proxies-performance-rotated](images/README/proxies-performance-rotated.png)
26 changes: 24 additions & 2 deletions benchmark/benchmark-pipeline.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
# new_terminal='gnome-terminal -- $1'

export client_url=3.133.229.116
export proxy_url=3.19.41.190
export proxy_url=18.217.152.113
export server_url=3.22.140.218
export proxy_private_url=172.31.7.16
export proxy_private_url=172.31.2.253
export server_private_url=172.31.22.170

# manual update proxy configurations
Expand Down Expand Up @@ -72,6 +72,24 @@ echo "stop proxy monolake"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url} -t 'cd ~/monolake/benchmark/proxy; ./stop-monolake.sh'
sleep 2

# then start proxy envoy
echo "start proxy envoy"
osascript -e 'tell app "Terminal" to do script "~/code/monolake/benchmark/pipeline-proxy-envoy.sh"'
sleep 5

# start client
echo "start client"
osascript -e 'tell app "Terminal" to do script "~/code/monolake/benchmark/pipeline-client-envoy.sh"'
sleep 2

echo "start client-metrics-collect"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${client_url} -t 'cd monolake/benchmark; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; ~/monolake/benchmark/performance-collect.sh wrk; echo "Please type exit to continue"; bash -l'

#stop proxy envoy
echo "stop proxy envoy"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url} -t 'cd ~/monolake/benchmark/proxy; ./stop-envoy.sh'
sleep 2

#stop server
echo "stop server"
ssh -i $HOME/ssh/monolake-benchmark.pem ec2-user@${server_url} -t 'sudo service nginx stop'
Expand All @@ -94,20 +112,24 @@ echo "copy collected data from proxy"
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:nginx-performance.csv .
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:traefik-performance.csv .
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:monolake-performance.csv .
scp -i $HOME/ssh/monolake-benchmark.pem ec2-user@${proxy_url}:envoy-performance.csv .

#plot data
echo "plot data"
./performance-plot.sh nginx
./performance-plot.sh traefik
./performance-plot.sh monolake
./performance-plot.sh envoy
./performance-plot.sh server
./performance-plot.sh wrk
./nginx-http-latency-plot.sh
./traefik-http-latency-plot.sh
./monolake-http-latency-plot.sh
./envoy-http-latency-plot.sh
./all-http-latency-plot.sh
./nginx-https-latency-plot.sh
./traefik-https-latency-plot.sh
./monolake-https-latency-plot.sh
./envoy-https-latency-plot.sh
./all-https-latency-plot.sh
./proxies-performance-plot.sh
20 changes: 20 additions & 0 deletions benchmark/client/benchmark-envoy-http.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# run benchmark: make sure proxy and server all are running; run this script from client
if [ -z "${MONOLAKE_HOME+set}" ]; then
export MONOLAKE_HOME=$HOME/monolake
fi

if [ -z "${MONOLAKE_BENCHMARK_PROXY_IP+set}" ]; then
export MONOLAKE_BENCHMARK_PROXY_IP=localhost
fi

if [ -z "${MONOLAKE_BENCHMARK_SERVER_IP+set}" ]; then
export MONOLAKE_BENCHMARK_SERVER_IP=localhost
fi

cd $HOME/wrk2

# http proxy for envoy
./wrk -d 1m -c 640 -t 64 -R 210000 --latency http://$MONOLAKE_BENCHMARK_PROXY_IP:8500/server2/ > http-result-4c-envoy-tiny.txt
./wrk -d 1m -c 640 -t 64 -R 200000 --latency http://$MONOLAKE_BENCHMARK_PROXY_IP:8500/server3/ > http-result-4c-envoy-small.txt
./wrk -d 1m -c 640 -t 64 -R 120000 --latency http://$MONOLAKE_BENCHMARK_PROXY_IP:8500/server4/ > http-result-4c-envoy-medium.txt
./wrk -d 1m -c 640 -t 64 -R 10000 --latency http://$MONOLAKE_BENCHMARK_PROXY_IP:8500/server5/ > http-result-4c-envoy-large.txt
20 changes: 20 additions & 0 deletions benchmark/client/benchmark-envoy-https.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# run benchmark: make sure proxy and server all are running; run this script from client
if [ -z "${MONOLAKE_HOME+set}" ]; then
export MONOLAKE_HOME=$HOME/monolake
fi

if [ -z "${MONOLAKE_BENCHMARK_PROXY_IP+set}" ]; then
export MONOLAKE_BENCHMARK_PROXY_IP=localhost
fi

if [ -z "${MONOLAKE_BENCHMARK_SERVER_IP+set}" ]; then
export MONOLAKE_BENCHMARK_SERVER_IP=localhost
fi

cd $HOME/wrk2

# https proxy for envoy
./wrk -d 1m -c 640 -t 64 -R 150000 --latency https://$MONOLAKE_BENCHMARK_PROXY_IP:5443/server2/ > https-result-4c-envoy-tiny.txt
./wrk -d 1m -c 640 -t 64 -R 140000 --latency https://$MONOLAKE_BENCHMARK_PROXY_IP:5443/server3/ > https-result-4c-envoy-small.txt
./wrk -d 1m -c 640 -t 64 -R 80000 --latency https://$MONOLAKE_BENCHMARK_PROXY_IP:5443/server4/ > https-result-4c-envoy-medium.txt
./wrk -d 1m -c 640 -t 64 -R 10000 --latency https://$MONOLAKE_BENCHMARK_PROXY_IP:5443/server5/ > https-result-4c-envoy-large.txt
4 changes: 4 additions & 0 deletions benchmark/client/verify-setup.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ curl -k http://$MONOLAKE_BENCHMARK_PROXY_IP:8100/server2
curl -k http://$MONOLAKE_BENCHMARK_PROXY_IP:8300/server2
# verify (monolake) proxy is ready
curl -k http://$MONOLAKE_BENCHMARK_PROXY_IP:8402
# verify (envoy) proxy is ready
curl -k http://$MONOLAKE_BENCHMARK_PROXY_IP:8500/server2/

# verify (nginx) tls proxy is ready
curl -k https://$MONOLAKE_BENCHMARK_PROXY_IP:8443/server2
Expand All @@ -29,3 +31,5 @@ curl -k https://$MONOLAKE_BENCHMARK_PROXY_IP:8443/server2
curl -k https://$MONOLAKE_BENCHMARK_PROXY_IP:9443/server2
# verify (monolake) tls proxy is ready
curl -k https://$MONOLAKE_BENCHMARK_PROXY_IP:6442
# verify (envoy) tls proxy is ready
curl -k https://$MONOLAKE_BENCHMARK_PROXY_IP:5443/server2/
Binary file modified benchmark/images/README/all-http-latency.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/images/README/all-latency-https.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/images/README/network-topology.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/images/README/proxies-performance-rotated.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/images/README/proxies-performance.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 71c6e50

Please sign in to comment.