Skip to content

Benchmark interpretation

Jump to bottom
Christopher Hunt edited this page May 7, 2019 · 4 revisions

Each build has a set of benchmarks that run to test (currently):

-------------------------------------------------------------------------------------------- benchmark: 8 tests -------------------------------------------------------------------------------------------
Name (time in ms)                            Min                 Max                Mean            StdDev              Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_python_spawn_time                   21.3215 (1.0)       27.4495 (1.0)       22.9800 (1.0)      1.5139 (1.35)      22.2420 (1.0)      2.4880 (1.40)         14;0  43.5161 (1.0)          37           1
test_quicken_import_time                 22.3037 (1.05)      27.6106 (1.01)      24.8583 (1.08)     1.5123 (1.35)      25.3876 (1.14)     2.8502 (1.60)         15;0  40.2279 (0.92)         42           1
test_quicken_cli_import_time             52.2955 (2.45)      62.7626 (2.29)      57.1451 (2.49)     3.4874 (3.12)      57.6780 (2.59)     6.5978 (3.71)          8;0  17.4993 (0.40)         19           1
test_quicken_start_after_first_time      61.6020 (2.89)      80.9876 (2.95)      66.5248 (2.89)     5.2323 (4.68)      64.7155 (2.91)     6.6298 (3.73)          1;1  15.0320 (0.35)         15           1
test_quicken_server_import_time          86.2055 (4.04)     102.1058 (3.72)      95.4627 (4.15)     5.1439 (4.60)      96.4755 (4.34)     7.3981 (4.16)          3;0  10.4753 (0.24)         10           1
test_python_program_time                123.0731 (5.77)     126.1379 (4.60)     124.3233 (5.41)     1.1320 (1.01)     123.7731 (5.56)     1.8471 (1.04)          3;0   8.0435 (0.18)          9           1
test_quicken_bypass_run_time            124.8132 (5.85)     127.8222 (4.66)     126.5076 (5.51)     1.1188 (1.0)      126.9804 (5.71)     1.8986 (1.07)          2;0   7.9047 (0.18)          8           1
test_quicken_start_first_time           243.0368 (11.40)    247.7039 (9.02)     245.0443 (10.66)    1.7013 (1.52)     244.8607 (11.01)    1.7771 (1.0)           2;0   4.0809 (0.09)          5           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Under test is a script that sleeps for 100ms at the module level and has a main function that just returns 0.

This represents an application that has 100ms of imports or module initialization code. Actual applications may have some overlap in imports with quicken itself which would speed things up a little, but not much.

These test:

  • test_python_spawn_time - how long does it take to spawn a Python process that runs nothing and imports the default site?
  • test_quicken_import_time - how long to run from quicken.lib import quicken and nothing else
  • test_quicken_cli_import_time - how long to import our actual client handler
  • test_quicken_start_after_first_time - how long does it take to start and finish the script after the server is up
  • test_quicken_server_import_time - import server
  • test_python_program_time - run the script directly - no quicken imports
  • test_quicken_bypass_run_time - run the script wrapped with quicken but bypass - do not try to connect to or start the server
  • test_quicken_start_first_time - run the script wrapped with quicken the first time

From which we can extrapolate:

User questions:

How fast can a Python program run (without -S)? min(test_python_spawn_time)

What is the average time that importing quicken will add to my app? mean(test_quicken_import_time) - mean(test_python_spawn_time)

What is the average time that using quicken will add to my app on incompatible platforms? mean(test_quicken_bypass_time) - mean(test_python_program_time)

What is the expected time-to-user-code when using quicken on a compatible platform? mean(test_quicken_start_after_first_time) - this is expected to be the same regardless of setup or number of modules

What is the expected overhead of using quicken for process start the first time? mean(test_quicken_start_first_time) - mean(test_python_program_time)

Possible areas of improvement:

  1. test_quicken_server_import_time - test_quicken_cli_import_time - time we spend importing things not used in client
  2. test_quicken_start_first_time - (test_python_program_time + test_quicken_server_import_time) - overhead of starting and connecting to server not taken up by script or server imports/module initialization

Benchmark shortcomings:

  1. (makes test look bad) - Assumes an application with a static 100ms import time with no overlap with required quicken imports
  2. (makes test look good) - Some amount of time for an application's imports will be taken by filesystem access, which would be less on re-invocation like this benchmark due to OS-level caching
Clone this wiki locally