Face recognition with Google FaceNet
architecture and retrained model by David Sandberg
(github.com/davidsandberg/facenet)
using TensorRT and OpenCV.
This project is based on the
implementation of l2norm helper functions which are needed in the output
layer of the FaceNet model. Link to the repo:
github.com/r7vme/tensorrt_l2norm_helper.
Moreover, this project uses an adapted version of PKUZHOU's implementation
of the mtCNN for face detection. More info below.
- NVIDIA Jetson Nano
- Raspberry Pi v2 camera
If you want to use a USB camera instead of Raspi Camera set the boolean isCSICam to false in main.cpp.
cuda 10.2 + cudnn 8.0
TensorRT 7.x
OpenCV 4.1.1
TensorFlow r1.14 (for Python to convert model from .pb to .uff)
This master branch now uses Jetpack 4.4, so dependencies have slightly changed and tensorflow is not preinstalled anymore. So there is an extra step that takes a few minutes more than before.
In case you would like to use older versions of Jetpack there is a tag jp4.2.2, that can links to the older implementation.
You can check NVIDIA website for help.
Installation procedures are very well documented.
If you are
using NVIDIA Jetson (Nano, TX1/2, Xavier) with Jetpack 4.4, most needed packages
should be installed if the Jetson was correctly flashed using SDK
Manager or the SD card image, you will only need to install cmake, openblas and tensorflow:
sudo apt install cmake libopenblas-devThe following shows the steps to install Tensorflow for Jetpack 4.4. This was copied from the official NVIDIA documentation. I'm assuming you don't need to install it in a virtual environment. If yes, please refer to the documentation linked above. If you are not installing this on a jetson, please refer to the official tensorflow documentation.
# Install system packages required by TensorFlow:
sudo apt update
sudo apt install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev liblapack-dev libblas-dev gfortran
# Install and upgrade pip3
sudo apt install python3-pip
sudo pip3 install -U pip testresources setuptools
# Install the Python package dependencies
sudo pip3 install -U numpy==1.16.1 future==0.18.2 mock==3.0.5 h5py==2.10.0 keras_preprocessing==1.1.1 keras_applications==1.0.8 gast==0.2.2 futures protobuf pybind11
# Install TensorFlow using the pip3 command. This command will install the latest version of TensorFlow compatible with JetPack 4.4.
sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v44 'tensorflow<2'The inputs to the original model are an input tensor consisting of a single or multiple faces and a phase train tensor telling all batch normalisation layers that model is not in train mode. Batch normalisation uses a switch layer to decide if the model is currently trained or just used for inference. This switch layer cannot be processed in TensorRT which is why it needs to be removed. Apparently this can be done using freeze_graph from TensorFlow, but here is a link to model where the phase train tensor has already been removed from the saved model github.com/apollo-time/facenet/raw/master/model/resnet/facenet.pb
Use the convert-to-uff tool which is installed with tensorflow installation to convert the *.pb model to *.uff. The script will replace unsupported layers with custom layers implemented by github.com/r7vme/tensorrt_l2norm_helper. Please check the file for the user defined values and update them if needed. Do not worry if there are a few warnings about the TRT_L2NORM_HELPER plugin.
cd path/to/project
python3 step01_pb_to_uff.pyYou should now have a facenet.uff file in the facenetModels folder which will be used as the input model to TensorRT.
This repo uses an implementation by PKUZHOU of the multi-task Cascaded Convolutional Neural Network (mtCNN) for face detection. The original implementation was adapted to return the bounding boxes such that it can be used as input to my FaceNet TensorRT implementation. You will need all models from the repo in the mtCNNModels folder so please do this to download them:
# go to one above project,
cd path/to/project/..
# clone PKUZHOUs repo,
git clone https://github.com/PKUZHOU/MTCNN_FaceDetection_TensorRT
# and move models into mtCNNModels folder
mv MTCNN_FaceDetection_TensorRT/det* path/to/project/mtCNNModelsAfter doing so you should have the following files in your mtCNNModels folder:
- det1_relu.caffemodel
- det1_relu.prototxt
- det2_relu.caffemodel
- det2_relu.prototxt
- det3_relu.caffemodel
- det3_relu.prototxt
- README.md
Done you are ready to build the project!
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j${nproc}If not run on Jetson platform set the path to your CUDA and TensorRT installation using -DCUDA_TOOLKIT_ROOTDIR=path/to/cuda and -DTENSORRT_ROOT=path/to/tensorRT.
.uff and .engine files are GPU specific, so if you use want to run this project on a different GPU or on another machine, always start over at step 3. above.
Put images of people in the imgs folder. Please only use images that contain one face.
NEW FEATURE:You can now add faces while the algorithm is running. When you see
the OpenCV GUI, press "N" on your keyboard to add a new face. The camera input will stop until
you have opened your terminal and put in the name of the person you want to add.
./mtcnn_facenet_cpp_tensorRTPress "Q" to quit and to show the stats (fps).
NOTE: This step might take a while when done the first time. TensorRT now parses and serializes the model from .uff to a runtime engine (.engine file).
Performance on NVIDIA Jetson Nano
- ~60ms +/- 20ms for face detection using mtCNN
- ~22ms +/- 2ms per face for facenet inference
- Total: ~15fps
Performance on NVIDIA Jetson AGX Xavier:
- ~40ms +/- 20ms for mtCNN
- ~9ms +/- 1ms per face for inference of facenet
- Total: ~22fps
Please respect all licenses of OpenCV and the data the machine learning models (mtCNN and Google FaceNet) were trained on.
Sometimes the camera driver doesn't close properly that means you will have to restart the nvargus-daemon:
sudo systemctl restart nvargus-daemonNiclas Wesemann
[email protected]