Something about reimplementation of the results for ultraquery model.

[leapxcheng]

Hi,

ULTRA and ULTRAQUERY are both impressive works.

However, when attempting to reimplement the experimental results for ULTRAQUERY (specifically those shown in Table 9) by following the instructions in README.md, I noticed that the performance (as shown following) was lower than expected.

python script/run_query.py -c config/ultraquery/transductive.yaml --dataset FB15kLogicalQuery --epochs 0 --bpe null --gpus [0] --bs 32 --threshold 0.0 --ultra_ckpt null --qe_ckpt /path/to/ckpts/ultraquery.pth

# FB15k237LogicalQuery
hits@1: 0.0088
hits@10: 0.0332
hits@3: 0.0174
mrr: 0.0181

# NELL995LogicalQuery
hits@1: 0.0242
hits@10: 0.0822
hits@3: 0.0471
mrr: 0.0445

I wonder if there might have been some issues during my reimplementation of the experiments. I would greatly appreciate your help with this.

[migalkin]

Did you compile rspmm CUDA kernels and which CUDA toolkit version did you use?

[leapxcheng]

Thank you for your feedback. It seems that the rspmm CUDA kernel is automatically compiled upon the first launch, as described. However, I would like to know how to manually compile the rspmm CUDA kernel and confirm whether it has been compiled successfully. I am currently using CUDA toolkit version 11.8.

[leapxcheng]

I resolved the issue by reinstalling the environment for Ultra, as it appears the rspmm CUDA kernel wasn't successfully compiled before. Thank you for your help!