Evaluating the pre-trained model on ScanNet test set yield NaN results.

[rcffc]

This is what I encounter if I run CUDA_VISIBLE_DEVICES=0 python test.py --config config/pointgroup_default_scannet.yaml --pretrain pointgroup.pth:

/app/util/config.py:20: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(f)
[2021-04-19 10:14:10,618  INFO  log.py  line 40  36]  ************************ Start Logging ************************
[2021-04-19 10:14:10,630  INFO  test.py  line 32  36]  Namespace(TEST_NMS_THRESH=0.3, TEST_NPOINT_THRESH=100, TEST_SCORE_THRESH=0.09, batch_size=4, bg_thresh=0.25, block_reps=2, block_residual=True, classes=20, cluster_meanActive=50, cluster_npoint_thre=50, cluster_radius=0.03, cluster_shift_meanActive=300, config='config/pointgroup_default_scannet.yaml', data_root='dataset', dataset='scannetv2', dataset_dir='data/scannetv2_inst.py', epochs=384, eval=True, exp_path='exp/scannetv2/pointgroup/pointgroup_default_scannet', fg_thresh=0.75, filename_suffix='_inst_nostuff.pth', fix_module=[], full_scale=[128, 512], ignore_label=-100, input_channel=3, loss_weight=[1.0, 1.0, 1.0, 1.0], lr=0.001, m=16, manual_seed=123, max_npoint=250000, mode=4, model_dir='model/pointgroup/pointgroup.py', model_name='pointgroup', momentum=0.9, multiplier=0.5, optim='Adam', prepare_epochs=128, pretrain='pointgroup.pth', pretrain_module=[], pretrain_path=None, save_freq=16, save_instance=False, save_pt_offsets=False, save_semantic=False, scale=50, score_fullscale=14, score_mode=4, score_scale=50, split='val', step_epoch=384, task='test', test_epoch=384, test_seed=567, test_workers=16, train_workers=16, use_coords=True, weight_decay=0.0001)
[2021-04-19 10:14:10,631  INFO  test.py  line 188  36]  => creating model ...
[2021-04-19 10:14:10,631  INFO  test.py  line 189  36]  Classes: 20
[2021-04-19 10:14:11,659  INFO  test.py  line 200  36]  cuda available: True
[2021-04-19 10:25:29,652  INFO  test.py  line 205  36]  #classifier parameters (model): 7715016
[2021-04-19 10:25:29,661  INFO  utils.py  line 61  36]  Restore from pointgroup.pth
[2021-04-19 10:25:29,762  INFO  test.py  line 41  36]  >>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>
[2021-04-19 10:25:29,844  INFO  scannetv2_inst.py  line 65  36]  Testing samples (val): 0
/app/util/eval.py:190: RuntimeWarning: Mean of empty slice
  avg_dict['all_ap']     = np.nanmean(aps[ d_inf,:,oAllBut25])
/app/util/eval.py:191: RuntimeWarning: Mean of empty slice
  avg_dict['all_ap_50%'] = np.nanmean(aps[ d_inf,:,o50])
/app/util/eval.py:192: RuntimeWarning: Mean of empty slice
  avg_dict['all_ap_25%'] = np.nanmean(aps[ d_inf,:,o25])
[2021-04-19 10:25:30,212  INFO  eval.py  line 274  36]  
[2021-04-19 10:25:30,212  INFO  eval.py  line 275  36]  ################################################################
[2021-04-19 10:25:30,212  INFO  eval.py  line 281  36]  what           :             AP         AP_50%         AP_25%
[2021-04-19 10:25:30,212  INFO  eval.py  line 282  36]  ################################################################
[2021-04-19 10:25:30,212  INFO  eval.py  line 292  36]  cabinet        :            nan            nan            nan
[2021-04-19 10:25:30,212  INFO  eval.py  line 292  36]  bed            :            nan            nan            nan
[2021-04-19 10:25:30,212  INFO  eval.py  line 292  36]  chair          :            nan            nan            nan
[2021-04-19 10:25:30,212  INFO  eval.py  line 292  36]  sofa           :            nan            nan            nan
[2021-04-19 10:25:30,212  INFO  eval.py  line 292  36]  table          :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  door           :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  window         :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  bookshelf      :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  picture        :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  counter        :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  desk           :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  curtain        :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  refrigerator   :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  shower curtain :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  toilet         :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  sink           :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  bathtub        :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 292  36]  otherfurniture :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 298  36]  ----------------------------------------------------------------
[2021-04-19 10:25:30,213  INFO  eval.py  line 303  36]  average        :            nan            nan            nan
[2021-04-19 10:25:30,213  INFO  eval.py  line 304  36]  

[dddlr]

I'm having this same issue too - did you ever find a solution? @rcffc

[dddlr]

Found the cause - I needed to complete the Data Preparation section before evaluating the pre-trained model Silly me 😄

[XIAOGUOY]

找到原因 - 我需要完成数据准备部分，然后再评估预先训练的模型愚蠢的我😄

请问你的数据准备部分就是按照作者说的把数据集放进去就可以么，还需要别的操作么

[dddlr]

嗯，把數據集放進去後，只要執行這些命令就可以吧

cd dataset/scannetv2
python prepare_data_inst.py --data_split train
python prepare_data_inst.py --data_split val
python prepare_data_inst.py --data_split test

I haven't used this library for a long time though, so I don't fully remember @XIAOGUOY