question to updated code

[flobotics]

hi,
i saw your code changes for tensorpack. You write its for tensorpack==0.8.9, but then i get the error:
ImportError: cannot import name 'MovingAverageSummary'

If i install tensorpack==0.9.4 it works. (with tensorflow 1.13)

---

But with multi-gpu it is not faster than with one gpu ? I changed line 78 in GAN.py to
def init(self, input, model, num_gpu=2):

and in main.py i changed SeparateGANTrainer( to GANTrainer(

Is that correct, or do i something wrong ? Is it possible to use more than 2 GPUs ? e.g. 8 GPUs ?
thanks
flo

[flobotics]

hi,
it does NOT work faster with 2 GPUs (2x V100)

But with more than 2 GPUs, it hangs after :

[0427 10:25:10 @base.py:275] Start Epoch 1 ...
0%| |0/1000[00:00<?,?it/s][0427 10:25:10 @input_source.py:554] Pre-filling StagingArea ...
[0427 10:25:14 @input_source.py:558] 1 element was put into StagingArea on each tower.

Is it possible to train this model on more than 1 GPU ?

used:
tensorflow-gpu 1.13.1
tensorpack 0.9.4
cudnn 10

thanks
flo

[flobotics]

when i start training, this error output comes before it starts to train, then this error does not happen again, but always at startup? Perhaps its interresting

2019-04-28 14:42:57.642632: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_2. Error: Pack node (LossB/stack) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642726: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_6. Error: Pack node (LossB/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642775: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_10. Error: Pack node (LossB/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642808: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_2. Error: Pack node (LossA/stack) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642837: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_6. Error: Pack node (LossA/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642869: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_10. Error: Pack node (LossA/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642916: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice. Error: Pack node (LossB/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642944: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_4. Error: Pack node (LossB/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:42:57.642968: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_8. Error: Pack node (LossB/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:42:57.643011: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice. Error: Pack node (LossA/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:42:57.643046: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_4. Error: Pack node (LossA/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:42:57.643069: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_8. Error: Pack node (LossA/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:43:00.569966: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_2. Error: Pack node (LossB/stack) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570046: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_6. Error: Pack node (LossB/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570095: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_10. Error: Pack node (LossB/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570126: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_2. Error: Pack node (LossA/stack) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570156: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_6. Error: Pack node (LossA/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570184: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_10. Error: Pack node (LossA/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570216: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice. Error: Pack node (LossB/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570245: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_4. Error: Pack node (LossB/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570281: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_8. Error: Pack node (LossB/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570308: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice. Error: Pack node (LossA/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570335: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_4. Error: Pack node (LossA/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:43:00.570363: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_8. Error: Pack node (LossA/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095399: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_2. Error: Pack node (LossB/stack) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095527: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_6. Error: Pack node (LossB/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095604: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_10. Error: Pack node (LossB/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095696: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_2. Error: Pack node (LossA/stack) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095784: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_6. Error: Pack node (LossA/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095853: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_10. Error: Pack node (LossA/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:43:21.095975: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice. Error: Pack node (LossB/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:43:21.096043: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_4. Error: Pack node (LossB/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:43:21.096110: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_8. Error: Pack node (LossB/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:43:21.096174: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice. Error: Pack node (LossA/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:43:21.096238: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_4. Error: Pack node (LossA/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:43:21.096300: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_8. Error: Pack node (LossA/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:43:40.817740: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_2. Error: Pack node (LossB/stack) axis attribute is out of bounds: 0
2019-04-28 14:43:40.817880: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_6. Error: Pack node (LossB/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:43:40.817942: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_10. Error: Pack node (LossB/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818002: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_2. Error: Pack node (LossA/stack) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818069: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_6. Error: Pack node (LossA/stack_2) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818118: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_10. Error: Pack node (LossA/stack_4) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818230: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice. Error: Pack node (LossB/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818307: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_4. Error: Pack node (LossB/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818377: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossB/strided_slice_8. Error: Pack node (LossB/stack_5) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818447: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice. Error: Pack node (LossA/stack_1) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818523: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_4. Error: Pack node (LossA/stack_3) axis attribute is out of bounds: 0
2019-04-28 14:43:40.818589: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node LossA/strided_slice_8. Error: Pack node (LossA/stack_5) axis attribute is out of bounds: 0