Refactor CAN Parser

[deanlee]

This PR refactors the existing code by shifting most of the logic to C++, significantly simplifying the update process, improving performance, and resolving issues related to the mixed update logic that was previously split between Cython and C++. The parser_pyx.pyx now functions as a streamlined interface wrapper around the C++ functions.

Key Changes:

1. Removed query_latest() Function: The query_latest() function, which previously copied updated values to Cython and was invoked with empty CAN strings in Cython's __init__ method to initialize value dictionaries, has been removed. This led to unnecessary complexity and has now been streamlined.
2. Exposed MessageState to Cython: MessageState is now directly accessible from Cython, allowing values to be read directly from the C++ MessageState object without the need for intermediate copying. This change streamlines data access, reduces overhead, and minimizes complexity and potential bugs.
3. Enhanced Message Tracking: Updates are now tracked using and returning a set of updated addresses in MessageState::update, ensuring comprehensive capture of every update, independent of can frame's timestamp.

By centralizing most of the logic in C++, this PR simplifies the Update Logic, Reduces the complexity of the code by removing redundant operations and unnecessary data copying.The new implementation is more efficient, as demonstrated by performance comparisons.

Issues Resolved:

This PR resolves multiple issues, and the test cases have been updated to catch them:

• Issue 1068
• Issue 913
• Issue 1066 (partially resolved)

Performance Comparison:

master:

6000 CAN packets, 10 runs
327.69 mean ms, 334.36 max ms, 321.73 min ms, 3.74 std ms
0.0546 mean ms / CAN packet

this branch:

6000 CAN packets, 10 runs
113.71 mean ms, 125.21 max ms, 109.85 min ms, 4.42 std ms
0.0189 mean ms / CAN packet

Additional Notes:

PlotJuggler will be broken after this PR because data is now read directly from MessageState instead of being copied using query_latest. However, it is quite easy to modify PlotJuggler to use the new API.

[sshane]

👀

batman@workstation-shane:~/openpilot/selfdrive/debug$ ./check_can_parser_performance.py 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  7.92it/s]
6000 CAN packets, 10 runs
99.36 mean ms, 99.86 max ms, 98.82 min ms, 0.33 std ms
0.0166 mean ms / CAN packet
batman@workstation-shane:~/openpilot/selfdrive/debug$ ./check_can_parser_performance.py 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 20.73it/s]
6000 CAN packets, 10 runs
23.19 mean ms, 23.77 max ms, 22.84 min ms, 0.25 std ms
0.0039 mean ms / CAN packet

[deanlee]

👀

batman@workstation-shane:~/openpilot/selfdrive/debug$ ./check_can_parser_performance.py 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  7.92it/s]
6000 CAN packets, 10 runs
99.36 mean ms, 99.86 max ms, 98.82 min ms, 0.33 std ms
0.0166 mean ms / CAN packet
batman@workstation-shane:~/openpilot/selfdrive/debug$ ./check_can_parser_performance.py 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 20.73it/s]
6000 CAN packets, 10 runs
23.19 mean ms, 23.77 max ms, 22.84 min ms, 0.25 std ms
0.0039 mean ms / CAN packet

Yes, this test should be accurate. The test results from my master version are outdated.

[sshane]

After this PR, we can simplify car/interface by using only one CanParser. With a single call to update_string for parsing all messages, there's no need to repeatedly feed identical can_strings to multiple instances of CanParser's update_string method for message filtering. This straightforward approach greatly enhances car/interface and parsing efficiency.

Nice, you mean we can pass in a dictionary with bus as the key, or similar?

[deanlee]

After this PR, we can simplify car/interface by using only one CanParser. With a single call to update_string for parsing all messages, there's no need to repeatedly feed identical can_strings to multiple instances of CanParser's update_string method for message filtering. This straightforward approach greatly enhances car/interface and parsing efficiency.

Nice, you mean we can pass in a dictionary with bus as the key, or similar?

y, or may something like {bus: [(msg1, freq1), (msg2, freq2)...], bus2: [...]}

[github-actions]

Thanks for contributing to opendbc! In order for us to review your PR as quickly as possible, check the following:

• Convert your PR to a draft unless it's ready to review
• Read the contributing docs
• Before marking as "ready for review", ensure:
  • the goal is clearly stated in the description
  • all the tests are passing
  • include a route or your device' dongle ID if relevant

[deanlee]

This PR is too complicated. Closed. Some functions have been moved to PR #1382.