tools/rerun: Rerun 0.18

[bongbui321]

This logs everything within 30-35s on my machine and final memory usage is about 2.5GB

[github-actions]

Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:

• Convert your PR to a draft unless it's ready to review
• Read the contributing docs
• Before marking as "ready for review", ensure:
  • the goal is clearly stated in the description
  • all the tests are passing
  • the change is something we merge
  • include a route or your device' dongle ID if relevant

[bongbui321]

These are services that don't have log data

preferrably, I like to only display services available within the logs, and send the blueprint after seeing every log messages, but rerun doesn't seem apply that sent blueprint when the logs are logged using subprocesses

[adeebshihadeh]

Got this error:

(.venv) macbookair:rerun adeebshihadeh$ ./run.py --demo --qcam
You're requesting more than 1 segment, please be aware that might take a lot of memory
Do you wish to continue? (Y/n): y
[2024-08-06T21:19:23Z INFO  re_sdk::spawn] A process is already listening at this address. Assuming it's a Rerun Viewer. addr=0.0.0.0:9876
  0%|                                                                             | 0/13 [00:19<?, ?it/s]
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/adeebshihadeh/openpilot/openpilot/tools/lib/logreader.py", line 294, in _run_on_segment
    return func(self._get_lr(i))
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adeebshihadeh/openpilot/.venv/lib/python3.12/site-packages/rerun_sdk/rerun/__init__.py", line 378, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adeebshihadeh/openpilot/tools/rerun/run.py", line 121, in _process_log_msgs
    rr.log_temporal_batch(
    ^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'rerun' has no attribute 'log_temporal_batch'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/adeebshihadeh/openpilot/tools/rerun/./run.py", line 174, in <module>
    rerunner.load_data()
  File "/Users/adeebshihadeh/openpilot/tools/rerun/./run.py", line 141, in load_data
    self.lr.run_across_segments(NUM_CPUS, partial(self._process_log_msgs, self.blueprint))
  File "/Users/adeebshihadeh/openpilot/openpilot/tools/lib/logreader.py", line 300, in run_across_segments
    for p in tqdm.tqdm(pool.imap(partial(self._run_on_segment, func), range(num_segs)), total=num_segs):
  File "/Users/adeebshihadeh/openpilot/.venv/lib/python3.12/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
AttributeError: module 'rerun' has no attribute 'log_temporal_batch'

I'm assuming it's because this doesn't install 0.18? Let's just ship that hack you had in the other PR, then remove it once 0.18 is released.

[adeebshihadeh]

Ah didn't realize you marked that other one as ready for review. Next time you can just do it all in one PR.

[adeebshihadeh]

Updated to 0.18, feedback so far:

• can we silence [2024-08-06T21:24:22Z INFO re_sdk_comms::server] New SDK client connected from: 127.0.0.1:62200?
• my shell gets messed up after rerun finishes, pic below. reset fixes it
• no prompt unless requested the big cam files for >1 segment, and a better explanation of why there's a prompt. currently it's not clear as a user why i'm seeing the prompt.
• time ./run.py --demo takes real 1m3.265s on my mac, which is several times slower than PJ. is this still expected?
• can we collapse all the services under "streams" by default? it'll make it easier to scroll
• can we make the plot empty by default? it's initialized with a bunch of junk now
• the whole viewer is pretty laggy. are you also seeing this?

[bongbui321]

can we silence [2024-08-06T21:24:22Z INFO re_sdk_comms::server] New SDK client connected from: 127.0.0.1:62200?

Currently no interface to turn off logging, but to use RUST_LOG=warn ./run --demo. I can run another process of rerun and set something like os.environ['RUST_LOG'] = "warn" but that would slow down the logging process since all the processors would have to pipe its output to parent process. abt 40s slower if do that. I have talked to rerun maintainer abt having an interface to turn it off so it should be here soon

my shell gets messed up after rerun finishes, pic below. reset fixes it

Fixed

no prompt unless requested the big cam files for >1 segment, and a better explanation of why there's a prompt. currently it's not clear as a user why i'm seeing the prompt.

updated explanation

time ./run.py --demo takes real 1m3.265s on my mac, which is several times slower than PJ. is this still expected?

This is my juggle.py
./juggle.py --demo 62.10s user 6.64s system 159% cpu 43.139 total

This is my run.py
tools/rerun/run.py --demo 166.24s user 5.78s system 490% cpu 35.081 total

can we collapse all the services under "streams" by default? it'll make it easier to scroll

Do you mean like this?

This seems pretty clean to me

can we make the plot empty by default? it's initialized with a bunch of junk now

This is my startup screen from scratch

Rerun saves your session from last time so that was maybe something to do with your last session. There is button you can press to reset the preview:

the whole viewer is pretty laggy. are you also seeing this?

Nope, it is very smooth for me