Speedup test_car_interfaces

[adeebshihadeh]

Takes ~1-2s per car. That's a big hit since we support nearly 300 different cars.

Goal is <=0.2s avg and <1s max on the CI machine, with the same test coverage. There's likely two or three things slowing this test down that'll be obvious with a bit of profiling.

Command to run:

pytest --durations=0 selfdrive/car/tests/test_car_interfaces.py 

Current output on my workstation

batman:sixpilot$ pytest --durations=10 selfdrive/car/tests/test_car_interfaces.py
=============================================================== test session starts ===============================================================
platform linux -- Python 3.11.4, pytest-8.2.0, pluggy-1.5.0
Using --randomly-seed=431984017
rootdir: /dev/shm/openpilot/sixpilot
configfile: pyproject.toml
plugins: randomly-3.15.0, subtests-0.12.1, anyio-4.3.0, cov-5.0.0, xdist-3.6.1, cpp-2.5.0, hypothesis-6.47.5, mock-3.14.0, timeout-2.3.1, flaky-3.8.1, asyncio-0.23.6
asyncio: mode=Mode.STRICT
32 workers [209 items]
........................................................................................................................................... [ 65%]
......................................................................                                                                      [100%]
============================================================== slowest 10 durations ===============================================================
2.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_FOCUS_MK4
2.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_138_MAZDA_CX9_2021
2.15s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_037_GENESIS_GV60_EV_1ST_GEN
2.05s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_147_SKODA_FABIA_MK4
1.97s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_094_HYUNDAI_VELOSTER
1.97s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_005_AUDI_Q3_MK2
1.87s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_059_HONDA_PILOT
1.83s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_113_KIA_OPTIMA_H_G4_FL
1.83s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_076_HYUNDAI_IONIQ_PHEV_2019
1.78s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_095_JEEP_GRAND_CHEROKEE
============================================================== 209 passed in 13.32s ===============================================================

[sshane]

A good place to optimize is the hypothesis example generation. Speeds are ~2x faster than our current version in version 4.51.0 and get slower somewhere between then and 4.55.3. The latest version (6.102.6) is even slower compared to our current version (6.47). You can root cause and open a PR to hypothesis or openpilot depending on where the optimization lies.

[deanlee]

Another option is to optimize get_fuzzy_car_interface_args by caching params_strategy to avoid regenerating strategies each time. Additionally, drawing st.dictionaries is quite slow; according to my tests, it can take up to 0.3 seconds.

[lsrugo]

Another option is to optimize get_fuzzy_car_interface_args by caching params_strategy to avoid regenerating strategies each time.

I tried this and it didn't seem to work.

Additionally, drawing st.dictionaries is quite slow; according to my tests, it can take up to 0.3 seconds.

Right so it's likely a problem in the underlying library

[geeth345]

Is this bounty still available? I'm looking to work on it but am confused if it has already been claimed by one of the PRs above

[c-gamble]

I'm interested in working on this but it's looking like this bounty has been (partially) solved? Is the bounty still open to go from~0.7s to <= 0.2s?

[sshane]

Getting this result on a AMD Ryzen Threadripper PRO 3955WX 16-Cores:

Using --randomly-seed=4229886755
rootdir: /home/batman/openpilot
configfile: pyproject.toml
plugins: timeout-2.3.1, xdist-3.6.1, cov-5.0.0, mock-3.14.0, asyncio-0.24.0, sugar-1.0.0, randomly-3.15.0, cpp-2.6.0, flaky-3.8.1, hypothesis-6.47.5, anyio-4.4.0, subtests-0.13.1, repeat-0.9.3
asyncio: mode=Mode.STRICT, default_loop_scope=function
16 workers [211 items]    collecting ... 

 selfdrive/car/tests/test_car_interfaces.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 60% ██████    
                                            ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                         100% ██████████
================================================================================= slowest 10 durations ==================================================================================
1.27s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_001_ACURA_RDX
1.22s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_032_FORD_MUSTANG_MACH_E_MK1
1.20s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_EXPLORER_MK6
1.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_026_FORD_ESCAPE_MK4
1.16s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_028_FORD_FOCUS_MK4
1.16s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_042_GENESIS_GV80
1.15s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_017_CHEVROLET_VOLT
1.14s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_047_HONDA_CIVIC_2022
1.12s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_025_FORD_BRONCO_SPORT_MK1
1.12s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_202_VOLKSWAGEN_POLO_MK6

Results (14.30s):
     211 passed
batman@workstation-shane:~/openpilot$

So to achieve the original goal, we need to see about a ~5-6x speedup still.

[c-gamble]

Getting this result on a AMD Ryzen Threadripper PRO 3955WX 16-Cores:

I ran on m3 MBP—appreciate the clarification!

[JIBSIL]

Hey guys,

This bounty appears open under Projects so I attempted it. I believe with a few changes I have improved it to your specifications without making the tests less effective (about a 8x improvement on my machine, i7-7700k). I'll open a PR in a bit, can you guys lock the bounty if my PR is satisfactory? First PR I've done on this repository by the way, so please let me know if there's anything I am missing.

[sshane]

You can put up a PR and we will review it.