Bad latency on CPU

[tilman67]

Hello,

docTR is quite slow on my computer. It takes about 1.6 to 10 seconds to analyse one page, depending on how much text is on the page. The pages are scanned documents. I don't have a GPU. I also tested with a GPU (on a different machine). The times are better, but my goal is to achive good performance without a GPU.

Is there anything I can di to make docTR faster?

[felixdittrich92]

Hi @tilman67 👋 ,

Would be great if you can provide some more information :)
you can use the following for this:

wget https://raw.githubusercontent.com/mindee/doctr/main/scripts/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

Deep Learning backend
Please run the following snippet and paste the output below.

from doctr.file_utils import is_tf_available, is_torch_available
print(f"is_tf_available: {is_tf_available()}")
print(f"is_torch_available: {is_torch_available()}")

Which model combination have you used ?
Have you tried to pass assume_straight_pages=True to the predictor ?
Maybe attatching an example if possible would be great 👍

[tilman67]

Hi Felix,

thank you for your reply!

Here is the output of collect_env.py:

DocTR version: 0.4.1f
TensorFlow version: 2.7.0
PyTorch version: 1.11.0+cu102 (torchvision 0.12.0+cu102)
OpenCV version: 4.5.1
OS: Ubuntu 20.04.4 LTS
Python version: 3.8.10
Is CUDA available (TensorFlow): No
Is CUDA available (PyTorch): No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

I also tried a newer docTR version, 0.6.0a0. Both were cloned from github.

I used model combinations db_resnet50 + crnn_vgg16_bn and db_resnet50 + crnn_mobilenet_v3_small.

I tried both the TensorFlow and PyTorch backend.

I didn't set assume_straight_pages=True, at least not deliberately. But that's an interesting hint, I'll try that asap.

Here is a typical test script:

import os
import time

os.environ["USE_TF"] = "1"
os.environ["USE_TORCH"] = "0"

from doctr.io import DocumentFile
from doctr.models import ocr_predictor

import argparse

arg_parser = argparse.ArgumentParser()
arg_parser.add_argument("image")
args = arg_parser.parse_args()

image_path = args.image

predictor = ocr_predictor("db_resnet50", "crnn_vgg16_bn", pretrained=True)

start_time = time.time()
doc = DocumentFile.from_images(image_path)
result = predictor(doc)
time_sec = time.time() - start_time
print(f"{time_sec:.2f}")

I'm going to send you a sample image, but must first find one that conforms privacy regulations.

Thank you so far!

[tilman67]

Hi Felix,

here are some sample images I tested with.

[felixdittrich92]

8Hi @tilman67 👋

Can you try the following combination if it's work better for you ?
predictor = ocr_predictor(det_arch='db_mobilenet_v3_large', reco_arch='crnn_mobilenet_v3_small', pretrained=True, assume_straight_pages=True)

my cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
first page: 0.84 sec
sec page: 0.81 sec

Otherwise you can also try the following suggestions from #814
Let me know if there are further questions 🤗

EDIT: tested with PyTorch / batch inference will also speed up the whole process (pass a list of images to .from_images([...]) / you can also play a bit with passing different batch sizes to the predictor maybe det_bs=4, rec_bs=256

EDIT2:

with following it takes ~2.8sec on my machine (CPU):

image_path = ["/home/felix/Desktop/1.jpg", "/home/felix/Desktop/2.jpg", "/home/felix/Desktop/3.jpg", "/home/felix/Desktop/4.jpg"]

predictor = ocr_predictor(det_arch='db_mobilenet_v3_large',
                          reco_arch='crnn_mobilenet_v3_small',
                          pretrained=True,
                          assume_straight_pages=True,
                          det_bs=1,
                          reco_bs=32)

start_time = time.time()
doc = DocumentFile.from_images(image_path)
result = predictor(doc)
time_sec = time.time() - start_time
print(f"{time_sec:.2f}")

and (GPU) ~ 0.8sec

image_path = ["/home/felix/Desktop/1.jpg", "/home/felix/Desktop/2.jpg", "/home/felix/Desktop/3.jpg", "/home/felix/Desktop/4.jpg"]

predictor = ocr_predictor(det_arch='db_mobilenet_v3_large',
                          reco_arch='crnn_mobilenet_v3_small',
                          pretrained=True,
                          assume_straight_pages=True,
                          det_bs=4,
                          reco_bs=128).cuda()

[tilman67]

Hi Felix,

following your instructions, I was able to achieve similar good times on my machine! :-)

Thank you very much for your help!

Best regards,
Tilman

[frgfm]

Hi @tilman67 👋

Arriving a bit late to the party, my apologies! To elaborate on @felixdittrich92 suggestion, there are multiple architectures available for each task. The reason this suggestion is working better for you is that you're on CPU, and mobilenet (serving as feature extractor here) are designed for fast inference on CPU, much more than resnets :)

Happy coding ✌️

[assistechsol]

predictor=ocr_predictor(det_arch='db_mobilenet_v3_large',
                          reco_arch='crnn_mobilenet_v3_small',
                          pretrained=True,
                          assume_straight_pages=True,
                          det_bs=4,
                          reco_bs=128).cuda()

AttributeError: 'OCRPredictor' object has no attribute 'cuda'

[assistechsol]

gpu code show this error

[felixdittrich92]

@assistechsol Hi 👋 ,
do you use PyTorch as framework ? Otherwise no need to do this Tensorflow will use your GPU automatically 👍