Bad accuraccy + Crash after saving weights4.h5

[pasquierjb]

In the last epoch of the last loop, I get:

Epoch 50/50
 - 23s - loss: 0.8512 - dice_coef: 1.1176e-04 - dice_coef_rounded: 1.0000 - binary_crossentropy: 0.0086 - val_loss: 0.8512 - val_dice_coef: 1.1405e-04 - val_dice_coef_rounded: 1.0000 - val_binary_crossentropy: 0.0084
steps_per_epoch 19 validation_steps 6

And at the end of the training, I get this strange error. However the unet_weights4.h5 got saved correctly.

None
Traceback (most recent call last):
  File "/opt/conda/envs/jupyter_env/bin/pixel_decoder", line 11, in <module>
    load_entry_point('pixel-decoder', 'console_scripts', 'pixel_decoder')()
  File "/example/pixel_decoder/main.py", line 75, in cli
    main(args.pop('command'), **args)
  File "/example/pixel_decoder/main.py", line 56, in main
    train(**kwargs)
  File "/example/pixel_decoder/train.py", line 62, in train
    metrics=[dice_coef, dice_coef_rounded, metrics.binary_crossentropy])
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/engine/training.py", line 830, in compile
    sample_weight, mask)
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/engine/training.py", line 429, in weighted
    score_array = fn(y_true, y_pred)
  File "/example/pixel_decoder/loss.py", line 28, in dice_logloss3
    return binary_crossentropy(y_true, y_pred) * 0.15 + dice_coef_loss(y_true, y_pred) * 0.85
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/losses.py", line 77, in binary_crossentropy
    return K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3068, in binary_crossentropy
    output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py", line 61, in clip_by_value
    [t, clip_value_min, clip_value_max]) as name:
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5770, in __enter__
    g = _get_graph_from_inputs(self._values)
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5428, in _get_graph_from_inputs
    _assert_same_graph(original_graph_element, graph_element)
  File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5364, in _assert_same_graph
    original_item))
ValueError: Tensor("loss/conv2d_11_loss/Const:0", shape=(), dtype=float32) must be from the same graph as Tensor("conv2d_11/Sigmoid:0", shape=(?, 256, 256, 1), dtype=float32).

I attach the log file of the training.

log.txt

[pasquierjb]

Solved with #1 . Still get the final strange error but now the model learns correctly. Here are the final metrics I get:

Epoch 50/50
 - 23s - loss: 0.4352 - dice_coef: 0.5432 - dice_coef_rounded: 0.5629 - binary_crossentropy: 0.3129 - val_loss: 0.4929 - val_dice_coef: 0.4875 - val_dice_coef_rounded: 0.5036 - val_binary_crossentropy: 0.3815
steps_per_epoch 19 validation_steps 6

[Geoyi]

Glad you fix it and the metrics look correct to me, @pasquierjb.

[pasquierjb]

Good! Do you have a code to vectorize the predictions (into a geojson with multipe lines for example)? @Geoyi

[Geoyi]

Sorry, I totally missed out your questions here @pasquierjb. Yes, I do have a vectorization of road prediction, but currently, it's in Javascript, and we will write it in python and open source with company version of Pixel Decoder.

I don't have much build in in this version, but only for the presentation of FOSS4G.

[pasquierjb]

Thanks! Let me know when it's ready