Project Components

This project focuses on two major components:

1. Video enhancement system and image processing.
2. Data extraction and analytics specific about vehicles and license plates.

Repositories:

1. Real-ESRGAN - Enhance low-resolution images with super-resolution techniques.

   • GitHub Repository

2. RVRT - Techniques for de-blurring images and videos.

   • GitHub Repository

3. RealBasicVSR - A repository dedicated to video super-resolution.

   • GitHub Repository

4. NonUniformBlurKernelEstimation - Estimate the blur kernel in images for de-blurring.

   • GitHub Repository

5. Grounded-Segment-Anything (SAM) - Segmentation techniques based on text prompts.

   • GitHub Repository

6. Yolo Tracking - Object detection, tracking, and REID using Yolo.

   • GitHub Repository

7. Co-Tracker Optical Flow - Optical flow-based tracking system.

   • GitHub Repository

8. Frame Interpolation - Interpolate frames in videos for smooth transitions.

   • GitHub Repository

9. Inpaint-Anything - Techniques for inpainting objects in images and videos.

   • GitHub Repository

10. Yolo detection - Techniques for detections segmentation.

    • GitHub Repository

11 **Segment anything ** - Techniques for segmentation. - GitHub Repository

Models:

1. Lavis Model - A model for question-answering based on images.

   • GitHub Model

2. ViLT Model - Another model for image-based question-answering, hosted on Hugging Face.

   • Hugging Face Model

Additional Resources:

1. PyTorch - An open-source machine learning library to accelerate the path from research to production.

   • Official Website

2. Plate recognizer - A company provide ability to recognize license plate.

   • Official Website

---

---

1. Video Enhancement System

Image/Video Stabilization:

• Image stabilization by classical calculations:

  • Options: Directory of images input -> processing -> image result

• Reticle image stabilization using optical flow and averaging:

  • Options:
    • Video input -> processing -> image result
    • Video input -> processing -> video result with averaging
    • Video input -> processing -> video result without averaging

---

Video Enhancement:

1. Averaging and stabilization - ECC by choosing a segmentation map for an object that we want to stabilize.

   • Options:
     • Directory of images + directory of segmentation mask -> processing -> image result

2. De-noise - Repository

   • Options:
     • Video + ROI User selection -> processing -> denoised video result

3. De-blur - Repository

   • Options: Video + ROI User selection -> processing -> De-blure video result

4. De-jpeg - Repository

   • Options: Video + ROI User selection -> processing -> De-jpeg video result

---

Video Processing

1. Blur kernel estimation - Repository

   • Options: Image + ROI User selection -> processing -> Kernel blur estimation

2. Object segmentation based on text prompt - Repository

   • Options: Directory of images + text prompt -> processing -> segmentation map

3. Object Detection & Tracking & REID - YOLO - Repository

   • Options: Video + classes and other terms -> processing -> crop and other stuff

4. Optical Flow - Repository

   • Options: Video + ROI User selection -> processing -> Video & PT

5 Frame Interpolation - Repository * Options: Image a + image b -> processing -> crop and other stuff

6. Questions and Answering - Based on the Lavis Model and ViLT Model

   • Options:
     • Person file questions + image -> processing -> Answer
     • Car file questions + image -> processing -> Answer
     • Free text question + image -> processing -> Answer

7. Create Motion Blur for Objects - Based on optical flow and some logic. (Coming Soon)

8. Remove/Inpaint Any Object

   • Options:
     • Video + ROI User selection -> processing (Model) -> Inpaint object inside video, Repository
     • Video + ROI User selection -> processing (Classic) -> Inpaint object inside video

9. Segmentation based on Optical Flow - Get segmentation (BB) based on the optical flow points track

   • Options: Video + ROI User selection -> processing -> cropped object, bb

---

Video Compression:

1. Video Compression: Compress video via H264 using YOLO Detector

---

---

2. Data Extraction and Analytics

Specifically regarding vehicles and license plates. Follow the provided web page for results for cars and license plates according to your terms. The results are provided as a JSON file per object (Car and its license) with the best score.

Note -> to use this web page u should to buy and install the license for - PlateRecognizerCompany

Get Started

1. Clone the Repository

git clone https://github.com/NehorayMelamed/VPAAE.git

2. Install PyTorch

pip3 install torch torchvision torchaudio

3. Setup the Project

Navigate to the project's root directory and run:

pip install -r requirements.txt
sudo apt install python3-tk
sudo apt install tk-dev

4. Download and Place the .pt and .pth Files

1. Download the Files:
   Download the pt_and_pth_files.zip from this Drive link and place it in the base directory of the project.

2. Place the .pt Files in Their Locations:

   • Ensure you are in the root directory of the project (where the zip file should be located).
   • Navigate to the SystemBuilding directory:

     cd SystemBuilding
     

   • Run the code inside the SystemBuildingForUser.py script:

     python SystemBuildingForUser.py

5. Run the Main Script

python gui_video_processing_via_tkniret_3.py

6. You should see the main window

Remember

The way to get into the system for the vehicle and license plate is by clicking the below link of the window

TODO:

• Add an option to upload a video or a folder of photos, uniformly for everyone.

• To create a connection between the different services.

• Make VIDEO COMPRESSION so that it is also possible to choose with a simple YOLO, and not just upload a folder of FRAMES, and MASKS.

• Support the choice of CPU or CUDA for:

  • Chain_demo_main_interface -(By co-track)
  • FRAME INTERPOLATION
  • DENOISE
  • DEBLUR