Aim tensorboard tracker stop logging metrics after certain number of epochs

[aswanthkrishna]

❓tensorboard tracker stop logging metrics after certain number of epochs

I am using the tensorboard logger inorder to track my metrics. in the UI run is showing as in-progress but the metrics are not getting updated after 1-2 epochs. I am using a remote server to store my logs.

     train_process = subprocess.Popen(['python3', '/root/workspace/mmdetection/tools/train.py', config_file, "--work-dir", work_dir])
     time.sleep(60)
     run = AimRun(repo="aim://192.168.9.14:53800", sync_tensorboard_log_dir=work_dir, system_tracking_interval=10, force_resume=True, experiment='my-mmdetection-trial')
     train_process.wait()

Have anyone faced similiar issues, and how can i try to resolve this?

[SGevorg]

Hi @aswanthkrishna are there any traces in the terminal. Maybe some exception or something? How can we reproduce this?

[aswanthkrishna]

@SGevorg i could not find annything particular in the logs

I am running this mmdetection tutorial

this create tensorboard logs of the following format

./work_dirs/20240621_150242/vis_data
│
├── 20240621_150242.json
├── config.py
├── events.out.tfevents.1718982168.gpu02.2571.0
└── scalars.json

i am using following code for aim tensorboard tracker based on the tutorial

from aim.ext.tensorboard_tracker import Run as AimRun
work_dir="./work_dirs"
run = AimRun(sync_tensorboard_log_dir=work_dir, system_tracking_interval=10, force_resume=True, experiment='my-mmdetection-trial')