Skip to content

OSIS Initial Performance Analysis

Jump to bottom
Sai Tharun Thodupunuru edited this page Sep 8, 2021 · 31 revisions

Objective

Checking the performance behavior of the OSIS service on scality cloud. As OSIS is not configured through NGINX, only one node will be configured in vCloudDirector.

  • Tenant Flow: Query/Create/List Tenant

  • User Flow: Query/Get/Create/List User

  • Credential Flow: Query/Get/Create/List Credentials

Test Cases:

  • Base Case:

    • Simple flow → Query/Create/List Tenant, Query/Get/Create/List User, Query/Get/Create/List Credentials

  • Medium load:

    • 1 client (threads)

      • (loop count) 100 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 1000 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 10000 x Tenant Flow, User Flow, Credential Flow

    • 2 clients (threads)

      • (loop count) 100 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 1000 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 10000 x Tenant Flow, User Flow, Credential Flow

    • 4 clients (threads)

      • (loop count) 100 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 1000 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 10000 x Tenant Flow, User Flow, Credential Flow

  • Ramp Up test - Stress failure test

    • Tenant Flow, User Flow, Credential Flow
Clients (threads) Loop Count(number of times to execute the test per thread) Rampup time: (Ramp-Up Period tells JMeter how long to delay before starting the next user) (in seconds)
128 10 300
256 10 300
512 10 300
128 10 600
256 10 600
512 10 600

Metrics to capture:

  • Elapsed time: Measures the elapsed time from just before sending the request to just after the last chunk of the response has been received,

  • Latency: Measures the latency from just before sending the request to just after the first chunk of the response has been received,

  • Median: Number which divides the samples into two equal halves,

  • 90% Line (90th Percentile): The elapsed time below which 90% of the samples fall

  • 95% Line (95th Percentile): The elapsed time below which 95% of the samples fall

  • Throughput: Calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. The formula is: Throughput = (number of requests) / (total time).

Storage Node metrics using RING grafana dashboards(http:///stats/):

  • Docker Image size

  • RAM/CPU/Disk usage

  • Report numbers to Product for tooling adjustments.

Test Results

1. Base Case:

Simple flow → Query/Create/List Tenant, Query/Get/Create/List User, Query/Get/Create/List Credentials

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 1 1095 1095 1095 1095 1095 1095 1095 0.000% 0.91324
List Tenants 1 204 204 204 204 204 204 204 0.000% 4.90196
Query Tenants 1 205 205 205 205 205 205 205 0.000% 4.87805
Create User 1 1432 1432 1432 1432 1432 1432 1432 0.000% 0.69832
List Users 1 292 292 292 292 292 292 292 0.000% 3.42466
Query Users 1 266 266 266 266 266 266 266 0.000% 3.75940
Get User 1 236 236 236 236 236 236 236 0.000% 4.23729
Create Credential 1 281 281 281 281 281 281 281 0.000% 3.55872
List Credentials 1 319 319 319 319 319 319 319 0.000% 3.13480
Query Credentials 1 278 278 278 278 278 278 278 0.000% 3.59712
Get Credential 1 237 237 237 237 237 237 237 0.000% 4.21941
TOTAL 11 440 278 1095 1095 1432 204 1432 0.000% 2.23532

2. Medium load

i. 1 client (threads)

a.(loop count) 100 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput(per second)
Create Tenant 100 228 214 252 294 385 192 813 0.000% 0.38506
List Tenants 100 222 216 251 265 276 196 279 0.000% 0.38624
Query Tenants 100 203 198 221 228 262 185 296 0.000% 0.38622
Create User 100 386 374 464 494 564 292 565 0.000% 0.38603
List Users 100 210 206 239 244 281 185 307 0.000% 0.38630
Query Users 100 232 228 259 279 284 206 353 0.000% 0.38635
Get User 100 208 203 225 243 282 189 312 0.000% 0.38639
Create Credential 100 246 229 281 295 345 202 1253 0.000% 0.38644
List Credentials 100 227 223 256 269 287 200 294 0.000% 0.38643
Query Credentials 100 218 212 245 262 283 192 291 0.000% 0.38647
Get Credential 100 222 213 253 273 366 195 427 0.000% 0.38650
TOTAL 1100 237 217 304 373 478 185 1253 0.000% 4.19926

b.(loop count) 1000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput(per second)
Create Tenant 1000 216 207 241 259 308 185 1446 0.000% 1.34937
List Tenants 1000 283 263 375 440 599 195 874 0.000% 1.35041
Query Tenants 1000 237 224 292 331 439 185 612 0.000% 1.35033
Create User 1000 299 283 352 378 490 240 1719 0.000% 0.98528
List Users 1000 299 292 365 385 489 195 922 0.000% 0.98535
Query Users 1000 207 202 229 241 266 182 563 0.000% 0.98540
Get User 1000 205 201 226 240 261 181 422 0.000% 0.98575
Create Credential 1000 229 221 254 272 310 196 1549 0.000% 0.49581
List Credentials 1000 1209 1226 1896 2125 2515 220 3220 0.000% 0.49534
Query Credentials 1000 289 288 350 373 425 193 493 0.000% 0.49531
Get Credential 1000 287 285 350 375 436 191 576 0.000% 0.49528
TOTAL 11000 342 247 410 1138 1856 181 3220 0.000% 2.91225

c.(loop count) 10000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput(per second)
Create Tenant 10000 219 211 243 257 322 183 1511 0.000% 0.49515
List Tenants 10000 940 1012 1198 1265 1410 190 2151 0.000% 0.49516
Query Tenants 10000 857 926 1103 1166 1317 183 2009 47.710% 0.49514
Create User 10000 399 379 497 557 754 258 2240 0.000% 0.53605
List Users 10000 513 510 617 664 791 210 1245 0.000% 0.53609
Query Users 10000 948 930 1118 1188 1365 486 3687 43.510% 0.53607
Get User 10000 189 186 201 211 238 174 2540 48.510% 0.39274
Create Credential 10000 211 206 234 248 288 187 408 0.000% 0.39274
List Credentials 10000 201 197 220 239 267 181 333 0.000% 0.39275
Query Credentials 10000 197 193 214 226 256 179 320 0.000% 0.39276
Get Credential 10000 196 192 213 223 248 178 313 0.000% 0.39277
TOTAL 110000 647 567 1079 1151 1314 183 3687 24.206% 1.54137

ii. 2 clients (threads)

a.(loop count) 100 x Tenant Flow, User Flow, Credential Flow

b.(loop count) 1000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average Median 90% Line 95% Line 99% Line Min Max Error % Throughput
Create Tenant 1998 244 236 281 299 364 193 1905 0.000% 1.70639
List Tenants 1998 508 386 908 990 1218 197 1633 0.000% 1.70511
Query Tenants 1998 414 288 788 865 1054 194 1888 0.000% 1.70387
Create User 2000 348 330 427 466 583 251 1873 0.050% 1.65971
List Users 1999 392 398 511 550 643 211 781 0.000% 1.66055
Get User 1999 226 221 257 278 331 185 403 0.000% 1.66064
Query Users 1999 232 224 265 284 339 184 1199 0.000% 1.66073
Set userID property 2 10 10 11 11 11 10 11 0.000% 45.45455
Create Credential 2000 263 250 314 343 434 203 1562 0.000% 1.10789
List Credentials 2000 817 856 1146 1248 1490 219 2914 0.000% 1.10755
Get Credential 2000 360 363 448 477 560 202 834 14.250% 1.10753
Query Credentials 2000 359 362 449 476 550 196 661 14.250% 1.10747
TOTAL 21995 378 303 728 909 1171 10 2914 2.596% 5.25439

c.(loop count) 10000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average Median 90% Line 95% Line 99% Line Min Max Error % Throughput
Create Tenant 20000 213 207 230 241 280 182 1524 0.000% 0.82181
List Tenants 20000 1136 1160 1404 1509 1791 191 3224 0.000% 0.82182
Query Tenants 20000 1081 1102 1346 1449 1743 183 2847 66.820% 0.82180
Create User 20000 341 332 408 441 531 230 1772 0.000% 1.74725
List Users 20000 392 392 462 494 577 188 850 0.000% 1.74738
Query Users 20000 202 197 218 227 260 178 1407 0.000% 1.74739
Get User 20000 203 198 226 241 275 177 602 0.000% 1.74750
Create Credential 20000 235 221 276 307 441 189 1756 0.000% 1.30582
List Credentials 20000 798 780 974 1068 1330 204 2902 0.000% 1.30579
Query Credentials 20000 418 408 508 559 705 191 1420 80.170% 1.30579
Get Credential 3966 372 361 489 544 707 192 1598 0.000% 0.25920
TOTAL 203966 499 345 1136 1246 1485 177 3224 14.413% 3.99129
Clone this wiki locally