rank 0 is not on the local node

[Delaunay]

rdv is cn-n002.server.mila.quebec:29400 but RANK 0 is on cn-n001

vjepa-nodes.cn-n002.server.mila.quebec [start] /tmp/workspace/cuda/results/venv/torch/bin/benchrun --nnodes=2 --rdzv-backend=c10d --rdzv-endpoint=cn-n002.server.mila.quebec:29400 --nproc-per-node=8 -- 
/tmp/workspace/cuda/results/venv/torch/bin/voir /home/mila/d/delaunap/milabench/benchmarks/vjepa/main.py --dataset /tmp/workspace/cuda/results/data/FakeVideo/video_metainfo.csv --output 
/tmp/workspace/cuda/results/extra/vjepa-nodes [at 2024-09-11 13:36:52.011579]
vjepa-nodes.cn-n001.server.mila.quebec.nolog [start] ssh -oCheckHostIP=no -oStrictHostKeyChecking=no -oPasswordAuthentication=no -oPasswordAuthentication=no -p 22 [email protected] 
/tmp/workspace/cuda/results/venv/torch/bin/benchrun --nnodes=2 --rdzv-backend=c10d --rdzv-endpoint=cn-n002.server.mila.quebec:29400 --nproc-per-node=8 -- /tmp/workspace/cuda/results/venv/torch/bin/voir 
/home/mila/d/delaunap/milabench/benchmarks/vjepa/main.py --dataset /tmp/workspace/cuda/results/data/FakeVideo/video_metainfo.csv --output /tmp/workspace/cuda/results/extra/vjepa-nodes [at 2024-09-11 
13:36:57.705778]

vjepa-nodes.cn-n002.server.mila.quebec [stdout] 10
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 12
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 11
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 13
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 9
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 14
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 15
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 8
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 3
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 4
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 7
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 2
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 6
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 0
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 5
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 1

[Delaunay]

Adding --node-rank does not help

/tmp/workspace/cuda/results/venv/torch/bin/benchrun --nnodes=2 --rdzv-backend=c10d 
--rdzv-endpoint=cn-n002.server.mila.quebec:29400 --node-rank 0 --nproc-per-node=8 -- /tmp/workspace/cuda/results/venv/torch/bin/voir 
/home/mila/d/delaunap/milabench/benchmarks/vjepa/main.py --dataset /tmp/workspace/cuda/results/data/FakeVideo/video_metainfo.csv --output 
/tmp/workspace/cuda/results/extra/vjepa-nodes [at 2024-09-11 13:50:06.594776]
vjepa-nodes.cn-n001.server.mila.quebec.nolog [start] ssh -oCheckHostIP=no -oStrictHostKeyChecking=no -oPasswordAuthentication=no -oPasswordAuthentication=no -p 22 
[email protected] /tmp/workspace/cuda/results/venv/torch/bin/benchrun --nnodes=2 --rdzv-backend=c10d --rdzv-endpoint=cn-n002.server.mila.quebec:29400 
--node-rank 1 --nproc-per-node=8 -- /tmp/workspace/cuda/results/venv/torch/bin/voir /home/mila/d/delaunap/milabench/benchmarks/vjepa/main.py --dataset 
/tmp/workspace/cuda/results/data/FakeVideo/video_metainfo.csv --output /tmp/workspace/cuda/results/extra/vjepa-nodes [at 2024-09-11 13:50:14.261659]
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stderr] W0911 13:50:18.524000 140623254692992 torch/distributed/run.py:779]
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stderr] W0911 13:50:18.524000 140623254692992 torch/distributed/run.py:779] *****************************************
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stderr] W0911 13:50:18.524000 140623254692992 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable 
for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stderr] W0911 13:50:18.524000 140623254692992 torch/distributed/run.py:779] *****************************************
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 6
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 5
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 2
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 0
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 4
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 7
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 1
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 4/16
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 1/16
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 7/16
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] 3
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 3/16
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 6/16
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 5/16
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n001.server.mila.quebec.nolog [stdout] INFO:__main__:Initialized (rank/world-size) 2/16
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 10
vjepa-nodes.cn-n002.server.mila.quebec [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n002.server.mila.quebec [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n002.server.mila.quebec [stdout] INFO:__main__:Initialized (rank/world-size) 10/16
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 12
vjepa-nodes.cn-n002.server.mila.quebec [stdout] INFO:__main__:loaded params...
vjepa-nodes.cn-n002.server.mila.quebec [stdout] INFO:__main__:which_dtype='bfloat16'
vjepa-nodes.cn-n002.server.mila.quebec [stdout] INFO:__main__:Initialized (rank/world-size) 12/16
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 14
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 11
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 9
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 13
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 15
vjepa-nodes.cn-n002.server.mila.quebec [stdout] 8

[Delaunay]

needed to change --rdzv-backend=static