What is the difference between LlamaIndex text splitters and node parsers?

[SinghCoder]

Aren't they both the same thing - given a document, chunks them down into nodes.

[dosubot[bot]]

LlamaIndex text splitters and node parsers serve different purposes in document processing. Text splitters primarily focus on dividing text into smaller segments without directly considering the document's metadata. They operate by taking a string of text and outputting a list of smaller text segments. This process is abstract and requires specific implementation details to be defined in subclasses.

On the other hand, node parsers are designed to parse documents into nodes, which involves not just splitting text but also understanding the document's structure, semantics, and metadata. Node parsers work by parsing documents to establish node relationships, update metadata, and consider the document's overall structure. This process is more complex and involves methods that handle the parsing of nodes from documents, including considering metadata and relationships between nodes.

The key difference lies in the approach to handling text and metadata: text splitters are simpler and focus solely on breaking down text, while node parsers take a more holistic approach, considering the document's structure and metadata to parse it into nodes. The MetadataAwareTextSplitter subclass bridges the gap between these two by using metadata to inform how text is split, offering a nuanced approach that leans towards the complexity of node parsing.

Details

Sources

• Customized text chunking
• llama-index-core/llama_index/core/node_parser/interface.py
• llama-index-core/llama_index/core/node_parser/init.py
• llama-index-core/llama_index/core/node_parser/text/init.py
• llama-index-legacy/llama_index/legacy/node_parser/interface.py
• llama-index-legacy/llama_index/legacy/node_parser/text/sentence.py
• llama-index-legacy/llama_index/legacy/node_parser/text/token.py
• llama-index-core/llama_index/core/node_parser/text/token.py
• experimental/splitter_playground/app.py
• llama-index-legacy/llama_index/legacy/node_parser/text/semantic_splitter.py
• llama-index-core/llama_index/core/node_parser/text/semantic_splitter.py
• llama-index-core/llama_index/core/node_parser/file/json.py
• llama-index-legacy/llama_index/legacy/node_parser/file/json.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[jay-joshy]

Is it possible to combine text splitters and node parsers?
For example, I have a set of pdf documents - some which include tables. I've loaded the data using Llama parse.
I like the functionality of the node parser MarkdownElementNodeParser() specifically for its ability to extract out tables, but I find the chunk sizes otherwise for the main text is too large and 'pollutes' the context I want to send to another model with too much extra information.

In contrast, the regular text splitters (ie Sentence Splitter) can be configured to have smaller more granular chunks of context, but do not have the capability to extract a table accurately...

Can these two approaches be combined? @dosu