This demo uses Milvus to detect objects in a video based on a dataset of object images with known information. To get images of objects in videos, it uses OpenCV to extract video frames and then uses Yolov3 to detect objects in each frame. It uses ResNet50 to get feature vectors of images for both known objects in dataset and objects detected in video. Finally, it can detect object and get object information easily by similarity search in Milvus. Let's have fun playing with it!
The video object detection system will use Milvus to store and search the feature vector data. Mysql is used to store the correspondence between the ids returned by Milvus and the object information (name & image path). So you need to start Milvus and Mysql first.
-
Start Milvus v2.0
First, refer Install Milvus 2.0 for how to run Milvus with docker.
Note the version of Milvus should be consistent with pymilvus in requirements.txt.
-
Start MySQL
$ docker run -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 -d mysql:5.7
The next step is to start the system server. It provides HTTP backend services, you can run source code or use docker image to start it.
-
Install the Python packages
$ cd server $ pip install -r requirements.txt -
Install additional package if using MacOS
$ brew install ffmpeg # If you are using Ubuntu, you can use # $ apt install ffmpeg
-
Download Yolov3 Model
$ cd server/src/yolov3_detector/data $ ./prepare_model.shYou will get a folder yolov3_darknet containing 3 files:
├── yolov3_darknet │ ├── __model__ │ ├── __params__ │ └── yolo.yml -
Set configuration
$ vim server/src/config.py
Please modify the parameters according to your own environment. Here listing some parameters that need to be set, for more information please refer to config.py.
Parameter Description Default setting MILVUS_HOST The IP address of Milvus, you can get it by ifconfig. localhost MILVUS_PORT Port of Milvus. 19530 VECTOR_DIMENSION Dimension of the vectors. 2048 MYSQL_HOST The IP address of Mysql. localhost MYSQL_PORT Port of Milvus. 3306 DEFAULT_TABLE The milvus and mysql default collection name. video_obj_det DATA_PATH The folder path of known object images to insert. /data/example_object UPLOAD_PATH The folder path of the video. /data/example_video DISTANCE_LIMIT Maximum distance to return object information. If set as "None", then no limit on distance. 0.6 - DATA_PATH & UPLOAD_PATH: modify to your own ABSOLUTE paths for object images & video respectively
- DISTANCE_LIMIT: change to some number so that results with larger distances will not be shown in response
-
Set Parameters
$ export DATAPATH1='/absolute/image/folder/path' $ export DATAPATH2='/absolute/video/path' $ export Milvus_HOST='xxx.xxx.x.xx' $ export Milvus_PORT='19530' $ export Mysql_HOST='xxx.xxx.x.xx'
Note:: modify 'xxx.xxx.x.xx' to your own IP address
-
Run Docker
$ docker run -d \ -v ${DATAPATH1}:${DATAPATH1} \ -v ${DATAPATH2}:/data/example_video \ -p 5000:5000 \ -e "MILVUS_HOST=${Milvus_HOST}" \ -e "MILVUS_PORT=${Milvus_PORT}" \ -e "MYSQL_HOST=${Mysql_HOST}" \ milvusbootcamp/video-object-detect-server:2.0
Note: -v ${DATAPATH1}:${DATAPATH1} means that you can mount the directory into the container. If needed, you can load the parent directory or more directories.
-
Run the code
Then start the server with Fastapi.
$ cd src $ python main.py -
The API docs
Type localhost:5000/docs in your browser to see all the APIs.
/data
Return the object image by path.
/video/getVideo
Return the video by path.
/progress
Check the progress when loading.
/image/count
Return the number of vectors in Milvus.
/image/load
Load images of known objects by the folder path.
/video/search
Pass in a video to search for similar images of objects detected.
-
Code structure
If you are interested in our code or would like to contribute code, feel free to learn more about our code structure.
└───server │ │ Dockerfile │ │ requirements.txt │ │ main.py # File for starting the program. │ │ │ └───src │ │ config.py # Configuration file. │ │ encode.py # Covert image/video/questions/... to embeddings. │ │ milvus_helpers.py # Connect to Milvus server and insert/drop/query vectors in Milvus. │ │ mysql_helpers.py # Connect to MySQL server, and add/delete/query IDs and object information. │ │ │ └───operations # Call methods in milvus_helpers.py and mysql_helperes.py to insert/search/delete objects. │ │ insert.py │ │ search.py │ │ delete.py │ │ count.py
- Start the front-end
# Modify API_URL to the IP address and port of the server.
$ export API_URL='http://xxx.xx.xx.xx:5000' # change xxx.xx.xx.xx to your own IP address
$ docker run -d -p 8001:80 \
-e API_URL=${API_URL} \
milvusbootcamp/video-object-detect-client:2.0In this command,
API_URLmeans the query service address.
- How to use
Visit WEBCLIENT_IP:8001 in the browser to open the interface for reverse image search.
WEBCLIENT_IPspecifies the IP address that runs video_object_detection client docker.
Click UPLOAD DATA SET & enter the folder path of object images, then click CONFIRM to insert pictures. The following screenshot shows the loading process:
Note: The path entered should be consistent with DATA_PATH in config.py
The loading process may take a while depending on data size. The following screenshot shows the interface with images loading in progress.
Only support jpg pictures.
Then click UPLOAD A VIDEO TO SEARCH to upload a video to detect objects.
Note: The video should under the UPLOAD_PATH in config.py
Only support avi video.
The loading process may take a while. After video is successfully loaded, click Play button to play video and detected objects will be displayed with its image, name, distance (The smaller the distance the higher similarity between the object detected in video and the object image stored in Milvus).
