video doesn't work in simulator.py

[justjake]

...but it does in the legacy drc-sim.py

[zhuowei]

simulator.py only reads one set of packets per frame, which causes packets to be lost when the kernel's UDP packet queue fills up. drc-sim.py instead constantly reads packets, resulting in no lost packets.

I don't know much about networking, so I tried three solutions, neither of which worked:

First, I tried making drc-sim-keyboard also read all packets every frame (in a while loop until select no longer returns any ready sockets): video works then with no missing packets, but the entire application (including the video) renders at like 1fps.

Then, I tried moving the reading to a separate thread: same result as the first; I'm guessing that Python's global interpreter lock prevents threads from simultaneously running.

Finally, I tried disabling all services in the main app except for HID, and enabling all except HID in the standalone drc-sim.py in the hopes that the two could then be ran simultaneously. Didn't work since H264 in your repo has a different constructor that took a surface and I couldn't figure out how to interface with it

Two other approaches I haven't tried are to process packets for a set amount of length per frame and modifying drc-sim to ignore the inevitable dropped frame errors and attempt to render them anyways.

Edit: I see your solution on the maximum yolo branch was even crazier.

[justjake]

The slowness of python/pygame is why I eventually abandoned this project. That -- and the math required for proper mouse <--> joystick emulation. I am vaguely interested in a re-write of the control portion in Golang (since I don't know C++) but life is short...

[seveirein]

Justjake, got video working no problem. The main problem is that you were calling pygame.display.flip() every time through the main loop, which is an intensive operation. It should only be called when a new screen is ready or at a reasonable frame rate (not as fast as possible). It was taking up so much time that you were dropping udp packets. If you render and flip at a much more reasonable rate (or only when new screen is available from VSTRM) the video works great. Still plenty of other bugs though. The original code only called pygame.display.flip when a new screen was ready.