Skip to content

Evaluating deep learning models with float16 dtype in Keras, float16 inference

License

Notifications You must be signed in to change notification settings

knowledge3/keras-FP16-test

 
 

Repository files navigation

Keras float16 vs float32

Requirements

keras==2.1.0

tensorflow==1.8.0

opencv==3.2.0

pycocotools, BeautifulSoup4, lxml, tqdm

How to set float16 compute mode.

from keras import backend as K

K.set_floatx('float16)

Image Classification Part

Usage

Firstly, download ImageNet val data and model pre-trained weights file.

An example for testing vgg16 with float16.

python eval_image_classification.py --model='vgg' --dtype='float16'

An example for testing mobilenet with a width multiplier 1.0.

python eval_image_classification.py --model='mobilenet' --dtype='float16' --alpha=1.0

ImageNet Datatset

ImageNet val data provided by aaron-xichen, sincerely thanks to aaron-xichen for sharing this processed ImageNet val data.

Results

TOP1 acc and TOP5 acc on ImageNet val data.

Pre-trained weight files are downloaded from deep learning models, DenseNet121 weight file is downloaded from DenseNet-Keras Squeezenet weight file is downloaded from keras-squeezenet

There is a backup of weights in baiduyun(百度云)

float32 float16 diff
Model Top1 acc Top5 acc Top1 acc Top5 acc Top1 acc Top5 acc
VGG16 0.70786 0.89794 0.7082 0.89802 0.00034 0.00008
ResNet50 0.74366 0.91806 0.70508 0.89466 -0.03858 -0.0234
Inceptionv3 0.76518 0.92854 0.765 0.92854 -0.00018 0.00
Inception-ResNet 0.789 0.94426 0.7888 0.94436 -0.0002 0.0001
DenseNet121 0.74234 0.91868 0.74206 0.91868 -0.00028 0.000
Xception 0.77446 0.93618 0.77392 0.93596 -0.00054 -0.00049
Squeezenet 0.52294 0.76312 0.52172 0.76226 -0.00122 -0.00086
MobileNet-1-0 0.69856 0.89174 0.6966 0.8898 -0.00196 -0.00194
MobileNet-7-5 0.67726 0.87838 0.6726 0.87652 -0.00466 -0.00186
MobileNet-5-0 0.6352 0.85006 0.62944 0.84644 -0.00576 -0.00362
MobileNet-2-5 0.5134 0.75546 0.50272 0.74648 -0.01068 -0.00898

Object Detection Part

Usage

Firstly, download VOC2007 test set and COCO2017 val set, COCO2017 val set annotations datasets, then extract them and modify the path in script.

Secondly, download SSD pre-trained weights and put them in 'weights' directory.

SSD300 VOC weights, SSD300 COCO weights, SSD512 VOC weights, SSD512 COCO weights

The method for converting the original YOLOv3 model to a keras model can be found in this repo.

An example for evaluating SSD300 on VOC2007 test set

python eval_object_detection.py --model='ssd300' --dtype='float16' --eval-dataset='voc2007'

Results

Notice that SSD models suffer significant accuracy loss.

SSD results on VOC2007 test set

mAP
Model float32 float16 diff
SSD300 0.782 0.769 -0.013
SSD512 0.91 0.868 -0.042

The AP of each category can be found in this doc

SSD and YOLOv3 results on COCO val2017.

mAP
Model float32 float16 diff
SSD300 0.424 0.374 -0.050
SSD512 0.481 0.448 -0.033
YOLO320 to do to do to do
YOLO416 to do to do to do
YOLO608 to do to do to do

Semantic Segmentation Part

In this part, I evaluate semantic segmentation with float16 dtype.

U-net is adopted in this test.

HumanParsing-Dataset is adopted in this test.

The tested models are trained by my-self. Training details can be found in this repo: Person-Segmentation-Keras.

Usage

For person segmentation (binary classification) task.

python eval_segmentation.py --model='unet' --dtype='float16' --nClasses=2

For human parsing (multi-class classification) task.

python eval_segmentation.py --model='unet' --dtype='float16' --nClasses=5

Results

Person segmentation

mIoU
Model float32 float16 diff
Unet 0.8920 0.8918 -0.0002

Human parsing

mIoU
Part float32 float16 diff
Unet head 0.66476 0.66463 -0.00013
upper body 0.48639 0.48640 0.00001
both hands 0.27016 0.27005 -0.00011
lower body 0.66536 0.66520 -0.00016
mean 0.52167 0.52157 -0.0001

PointNet Evaluation

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

In this evaluation, I use a pre-trained PointNet reference to PointNet-Keras.

Results

3D classification

acc
Model float32 float16 diff
PointNet_cls 0.87824 0.87784 -0.0004

Reference

deep learing models

DenseNet-Keras

keras-squeezenet

ssd_keras

keras-yolo3

SegNet-Tutorial

Tensorflow-SegNet

image-segmentation-keras

About

Evaluating deep learning models with float16 dtype in Keras, float16 inference

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%