Using pypdfium2 instead of pypdf as the default document loader for langchain.document_loaders

[jerrytigerxu]

The PyPDFLoader() module, which is based on the pypdf.PdfReader() method, is considerably slower than using the pypdfium2.PdfDocument() method, with PyPDFLoader taking (on average), 1000% more time to load PDFs than pypdfium2.

Using pydfium2 instead of pypdf would save time on a exponential level, especially when handling PDFs files with over 500 pages.

[sandorkonya]

@jerrytigerxu , the pdfloader saves the page number as metadata, could we also save the document's absolute path with it?
Use case: i write articles for which i use multiple dozens of referece articles as base. I would like to see the page itself, where the resulting chunks originate from visually from the pdf (like a semantic search). What do you think, is this feasible within langchain?

[jerrytigerxu]

@sandorkonya That's a good question. I'm not entirely sure if that's feasible within LangChain, but I do think getting the absolute path is possible.

[sandorkonya]

@jerrytigerxu yes, the File reader API seems to be able to do this on file upload,
so basicly either one more metadata along the chunk or some kind of storage that creates a uuid for each file name and assigns those uuids to the chunk (to avoid bloating the chunk metadata with the same string (path) over and over again).

[vader-valencia]

Leaving the comment here in case it helps anyone else.

@jerrytigerxu , this was a fantastic idea! It saved me literally hours of debugging work!

Package versions used:

python 3.12
langchain_community==0.3.7
pypdfium2==4.30.0

Loader code:

loader = PyPDFium2Loader(file_path)
pages = loader.load()

[jerrytigerxu]

Appreciate it!