no attribute for multi-processing

[connection-ai]

Do you support mpiexec ? :-(
op module has no attribute for 'allreduce_nccl'

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
--sparcity matrix : 160 X 160
--sparcity matrix : 160 X 160
Traceback (most recent call last):
File "/storage/blocksparse/examples/transformer/enwik8_psy.py", line 498, in
train_loss, train_op, gn, ns = model(X, Y, loss_scale, train=True)
File "/storage/blocksparse/examples/transformer/enwik8_psy.py", line 365, in model
loss = bs.allreduce(loss)
File "/usr/local/lib/python3.6/dist-packages/blocksparse/nccl.py", line 42, in allreduce
ret = _op_module.allreduce_nccl(x, op_num=op_counter, sync_size=sync_size, num_comms=num_comms, prereduce=prereduce, logfile=logfile, name=name)
AttributeError: module 'b4313feb600581ca30dcf30777c2167e' has no attribute 'allreduce_nccl'
Traceback (most recent call last):
File "/storage/blocksparse/examples/transformer/enwik8_psy.py", line 498, in
train_loss, train_op, gn, ns = model(X, Y, loss_scale, train=True)
File "/storage/blocksparse/examples/transformer/enwik8_psy.py", line 365, in model
loss = bs.allreduce(loss)
File "/usr/local/lib/python3.6/dist-packages/blocksparse/nccl.py", line 42, in allreduce
ret = _op_module.allreduce_nccl(x, op_num=op_counter, sync_size=sync_size, num_comms=num_comms, prereduce=prereduce, logfile=logfile, name=name)
AttributeError: module 'b4313feb600581ca30dcf30777c2167e' has no attribute 'allreduce_nccl'