Pre-processing steps before docTR OCR

[khawar-islam]

I have different types of korean documents like color passport, black and white docs, national security card and many more documents like above. During OCR, I have to hide the personal information in documents.
In this way, I will first read the document with OCR and then hide the information. Sometimes OCR failed to hide the information which creates trouble for me.

Can i do some processioning steps which can enhance the docTR performance?

In below pictures, OCR missed to hide the "1234" security number which is very important

[frgfm]

Hello @khawar-islam 👋

Interesting use case!
Would you mind sharing the code that you use for that?

On my end, I just tried with the default option on the live demo app
And I had no problem retrieving all the test! Here is the JSON output:

{
  "page_idx": 0,
  "dimensions": [
    330,
    520
  ],
  "orientation": {
    "value": null,
    "confidence": null
  },
  "language": {
    "value": null,
    "confidence": null
  },
  "blocks": [
    {
      "geometry": [
        [
          0.0400390625,
          0.068359375
        ],
        [
          0.33984375,
          0.169921875
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.0400390625,
              0.068359375
            ],
            [
              0.33984375,
              0.169921875
            ]
          ],
          "words": [
            {
              "value": "2701585",
              "confidence": 0.046277422457933426,
              "geometry": [
                [
                  0.0400390625,
                  0.068359375
                ],
                [
                  0.33984375,
                  0.169921875
                ]
              ]
            },
            {
              "value": ":",
              "confidence": 0.7796680331230164,
              "geometry": [
                [
                  0.2255859375,
                  0.1279296875
                ],
                [
                  0.263671875,
                  0.1474609375
                ]
              ]
            },
            {
              "value": "-",
              "confidence": 0.49894943833351135,
              "geometry": [
                [
                  0.2509765625,
                  0.125
                ],
                [
                  0.3134765625,
                  0.154296875
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.0498046875,
          0.1611328125
        ],
        [
          0.3828125,
          0.2001953125
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.0498046875,
              0.1611328125
            ],
            [
              0.3828125,
              0.2001953125
            ]
          ],
          "words": [
            {
              "value": "ALIEN",
              "confidence": 0.7344251871109009,
              "geometry": [
                [
                  0.0498046875,
                  0.162109375
                ],
                [
                  0.126953125,
                  0.2001953125
                ]
              ]
            },
            {
              "value": "REGISTRATION",
              "confidence": 0.9981516599655151,
              "geometry": [
                [
                  0.1328125,
                  0.1650390625
                ],
                [
                  0.306640625,
                  0.1962890625
                ]
              ]
            },
            {
              "value": "CARD",
              "confidence": 0.9989686608314514,
              "geometry": [
                [
                  0.3095703125,
                  0.1611328125
                ],
                [
                  0.3828125,
                  0.1982421875
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.7890625,
          0.09375
        ],
        [
          0.9619140625,
          0.203125
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.7890625,
              0.09375
            ],
            [
              0.9619140625,
              0.203125
            ]
          ],
          "words": [
            {
              "value": "KOR",
              "confidence": 0.9629258513450623,
              "geometry": [
                [
                  0.7890625,
                  0.09375
                ],
                [
                  0.9619140625,
                  0.203125
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.3505859375,
          0.232421875
        ],
        [
          0.7431640625,
          0.2841796875
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.3505859375,
              0.232421875
            ],
            [
              0.7431640625,
              0.2841796875
            ]
          ],
          "words": [
            {
              "value": ":2",
              "confidence": 0.060839954763650894,
              "geometry": [
                [
                  0.3505859375,
                  0.232421875
                ],
                [
                  0.44921875,
                  0.2763671875
                ]
              ]
            },
            {
              "value": "123456-1234567",
              "confidence": 0.9305340051651001,
              "geometry": [
                [
                  0.45703125,
                  0.232421875
                ],
                [
                  0.7431640625,
                  0.2841796875
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.8271484375,
          0.2294921875
        ],
        [
          0.91015625,
          0.283203125
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.8271484375,
              0.2294921875
            ],
            [
              0.91015625,
              0.283203125
            ]
          ],
          "words": [
            {
              "value": "-",
              "confidence": 0.778566837310791,
              "geometry": [
                [
                  0.8271484375,
                  0.24609375
                ],
                [
                  0.853515625,
                  0.267578125
                ]
              ]
            },
            {
              "value": "F",
              "confidence": 0.775295615196228,
              "geometry": [
                [
                  0.8369140625,
                  0.2294921875
                ],
                [
                  0.91015625,
                  0.283203125
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.3505859375,
          0.2685546875
        ],
        [
          0.4375,
          0.30859375
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.3505859375,
              0.2685546875
            ],
            [
              0.4375,
              0.30859375
            ]
          ],
          "words": [
            {
              "value": "SRus",
              "confidence": 0.2719847857952118,
              "geometry": [
                [
                  0.3505859375,
                  0.2685546875
                ],
                [
                  0.4375,
                  0.30859375
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.3525390625,
          0.396484375
        ],
        [
          0.3818359375,
          0.431640625
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.3525390625,
              0.396484375
            ],
            [
              0.3818359375,
              0.431640625
            ]
          ],
          "words": [
            {
              "value": "d0",
              "confidence": 0.3320925235748291,
              "geometry": [
                [
                  0.3525390625,
                  0.396484375
                ],
                [
                  0.3818359375,
                  0.431640625
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.4189453125,
          0.3916015625
        ],
        [
          0.736328125,
          0.4404296875
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.4189453125,
              0.3916015625
            ],
            [
              0.736328125,
              0.4404296875
            ]
          ],
          "words": [
            {
              "value": "91",
              "confidence": 0.2635422945022583,
              "geometry": [
                [
                  0.4189453125,
                  0.3916015625
                ],
                [
                  0.458984375,
                  0.43359375
                ]
              ]
            },
            {
              "value": "HONG",
              "confidence": 0.9962020516395569,
              "geometry": [
                [
                  0.455078125,
                  0.3916015625
                ],
                [
                  0.5556640625,
                  0.4404296875
                ]
              ]
            },
            {
              "value": "SPECIMEN",
              "confidence": 0.9071111083030701,
              "geometry": [
                [
                  0.5693359375,
                  0.392578125
                ],
                [
                  0.736328125,
                  0.4375
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.4208984375,
          0.5185546875
        ],
        [
          0.7978515625,
          0.5693359375
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.4208984375,
              0.5185546875
            ],
            [
              0.7978515625,
              0.5693359375
            ]
          ],
          "words": [
            {
              "value": "21",
              "confidence": 0.3850867748260498,
              "geometry": [
                [
                  0.4208984375,
                  0.5234375
                ],
                [
                  0.44921875,
                  0.5517578125
                ]
              ]
            },
            {
              "value": "REPUBLIC",
              "confidence": 0.997722864151001,
              "geometry": [
                [
                  0.4560546875,
                  0.521484375
                ],
                [
                  0.6103515625,
                  0.5673828125
                ]
              ]
            },
            {
              "value": "OF",
              "confidence": 0.9974988102912903,
              "geometry": [
                [
                  0.62109375,
                  0.5185546875
                ],
                [
                  0.673828125,
                  0.5693359375
                ]
              ]
            },
            {
              "value": "UTOPIA",
              "confidence": 0.9955006837844849,
              "geometry": [
                [
                  0.6806640625,
                  0.5205078125
                ],
                [
                  0.7978515625,
                  0.568359375
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.353515625,
          0.6533203125
        ],
        [
          0.4462890625,
          0.6982421875
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.353515625,
              0.6533203125
            ],
            [
              0.4462890625,
              0.6982421875
            ]
          ],
          "words": [
            {
              "value": "NIFX",
              "confidence": 0.021382195875048637,
              "geometry": [
                [
                  0.353515625,
                  0.6533203125
                ],
                [
                  0.4462890625,
                  0.6982421875
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.6572265625,
          0.7646484375
        ],
        [
          0.75390625,
          0.8125
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.6572265625,
              0.7646484375
            ],
            [
              0.75390625,
              0.8125
            ]
          ],
          "words": [
            {
              "value": "\"BOXH",
              "confidence": 0.10252692550420761,
              "geometry": [
                [
                  0.6572265625,
                  0.7646484375
                ],
                [
                  0.75390625,
                  0.8125
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.8076171875,
          0.76953125
        ],
        [
          0.9169921875,
          0.8134765625
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.8076171875,
              0.76953125
            ],
            [
              0.9169921875,
              0.8134765625
            ]
          ],
          "words": [
            {
              "value": "2011.1.1",
              "confidence": 0.9881366491317749,
              "geometry": [
                [
                  0.8076171875,
                  0.76953125
                ],
                [
                  0.9169921875,
                  0.8134765625
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    },
    {
      "geometry": [
        [
          0.541015625,
          0.8203125
        ],
        [
          0.921875,
          0.93359375
        ]
      ],
      "lines": [
        {
          "geometry": [
            [
              0.55078125,
              0.8203125
            ],
            [
              0.9189453125,
              0.90234375
            ]
          ],
          "words": [
            {
              "value": "M889RE2A9A8",
              "confidence": 0.0017843696987256408,
              "geometry": [
                [
                  0.55078125,
                  0.8203125
                ],
                [
                  0.9189453125,
                  0.90234375
                ]
              ]
            }
          ]
        },
        {
          "geometry": [
            [
              0.541015625,
              0.890625
            ],
            [
              0.921875,
              0.93359375
            ]
          ],
          "words": [
            {
              "value": "CHIEF,SEOUL",
              "confidence": 0.7693777084350586,
              "geometry": [
                [
                  0.541015625,
                  0.8955078125
                ],
                [
                  0.6875,
                  0.93359375
                ]
              ]
            },
            {
              "value": "IMMIGRATION",
              "confidence": 0.5678054690361023,
              "geometry": [
                [
                  0.69140625,
                  0.8955078125
                ],
                [
                  0.8330078125,
                  0.9267578125
                ]
              ]
            },
            {
              "value": "OFFICE",
              "confidence": 0.9976310133934021,
              "geometry": [
                [
                  0.8408203125,
                  0.890625
                ],
                [
                  0.921875,
                  0.9267578125
                ]
              ]
            }
          ]
        }
      ],
      "artefacts": []
    }
  ]
}

The only part that it missed is the grey text at the bottom left. Let me know if that helps!

Cheers ✌️

[khawar-islam]

@frgfm I have noted one thing when i checked single image through on your live app. It works fine but when i put loop on several image then accuracy fall down. The code is extremely simple and i have just put a loop to iterate on all docs.

Here's is Code

import os

os.environ['USE_TORCH'] = '1'
import glob
import torch
import cv2
import os
from doctr.io import DocumentFile
from doctr.models import ocr_predictor, crnn_mobilenet_v3_large

print(torch.cuda.is_available())

face_cascade = cv2.CascadeClassifier('/media/cvpr/CM_1/pytesseract/haarcascade_frontalface_default.xml')

testing_path = "/media/cvpr/CM_1/COREMAX/testing/"
result_path = "/media/cvpr/CM_1/COREMAX/docTR/"

model = ocr_predictor(det_arch='db_resnet50', reco_arch='crnn_mobilenet_v3_large', pretrained=True)

for img in glob.glob(testing_path + '*.*'):
    if img.endswith(".pdf"):
        doc = DocumentFile.from_pdf(img)
    else:
        doc = DocumentFile.from_images(img)

    img_bgr_rgb = cv2.imread(img)
    file_Name = os.path.basename(img)
    image = DocumentFile.from_images(img)

    result = model(image)
    export = result.export()
    # print(result)

    page_words = [[word for block in page['blocks'] for line in block['lines'] for word in line['words']] for page
                  in export['pages']]

    page_dims = [page['dimensions'] for page in export['pages']]

    words_abs_coords = [
        [[int(round(word['geometry'][0][0] * dims[1])), int(round(word['geometry'][0][1] * dims[0])),
          int(round(word['geometry'][1][0] * dims[1])), int(round(word['geometry'][1][1] * dims[0]))] for word in words]
        for words, dims in zip(page_words, page_dims)
    ]

    faces = face_cascade.detectMultiScale(img_bgr_rgb, scaleFactor=1.2, minNeighbors=5)

    for (x, y, w, h) in faces:
        cv2.rectangle(img_bgr_rgb, (x, y), (x + w, y + h), (224, 224, 153), cv2.FILLED)

    for words, dims in zip(page_words, page_dims):
        for word in page_words:
            for single_word in word:
                if '-' in single_word['value']:
                    print(single_word)
                    xmin = int(single_word['geometry'][0][0] * dims[1])
                    ymin = int(single_word['geometry'][0][1] * dims[0])
                    xmax = int(single_word['geometry'][1][0] * dims[1])
                    ymax = int(single_word['geometry'][1][1] * dims[0])
                    image = cv2.rectangle(img_bgr_rgb, (xmin, ymin), (xmax, ymax), (224, 224, 153), cv2.FILLED)
                    cv2.imwrite(os.path.join(result_path, file_Name), img_bgr_rgb)
                else:
                    cv2.imwrite(os.path.join(result_path, file_Name), img_bgr_rgb)

[khawar-islam]

Through above code

testing.zip

[frgfm]

There is no way to know if there is a problem with this script 😅

If looping on the same image produces an accuracy drop, there is a problem. Otherwise it's just a coincidence that the order of files found by glob and the reduced performances align :)

But either way, one thing after another: did you solve the problem of this discussion topic or do you still have some issues?

[khawar-islam]

@frgfm Please give me sometime, i am still working on above problem and try to understand to where performance degrades during OCR operation