Proposal of pingcap/autoflow framework

[sykp241095]

Proposal for a New Open-Source RAG Framework Project: Seeking Community Feedback!

Hello, everyone!

We’re excited to introduce a proposal for a new open-source project hosted temporarily at pingcap/xxxrag: a flexible, lightweight Retrieval-Augmented Generation (RAG) framework. This project is designed as a developer-friendly library (similar to Django), tailored to meet the needs of AI application developers by providing streamlined RAG capabilities without a heavy user interface.

Project Goals and Scope of the RAG Framework

The framework is being built with a focus on adaptability and efficiency, addressing core needs for AI applications. Its scope includes essential tools for managing RAG workflows, as well as a unique "Search Plan" module that optimizes search strategies based on the context of each query. Key planned capabilities include:

• Knowledge Base Management – Tools for handling and updating domain-specific knowledge bases, allowing developers to keep content relevant and accessible.
• Data Source Management – Capabilities for organizing and integrating various data sources to support reliable retrieval processes.
• Index Management – Flexible management of vector, graph, and full-text search indexes, allowing developers to build or update indexes as needed.
• Basic Abstractions – Foundational abstractions that make it easy to extend and customize the framework for specific applications.

Key Modules of the RAG Framework

To ensure that this framework meets the diverse needs of AI application developers, we are structuring it into core modules focused on data import and indexing, optimized search strategies, and answer evaluation:

1. Data Import and Indexing

This module simplifies the process of bringing external data into the framework and managing it effectively:

• Data Crawling and Importing – Supports various data ingestion methods, including web crawling, API connections, and file-based imports, allowing developers to consolidate multiple data sources.
• Indexing – After data is imported, the framework allows developers to create and manage indexes tailored to the type of data, including:
  • Vector Indexes for similarity-based retrieval,
  • Graph Indexes for relationship-focused data structures, and
  • Full-Text Indexes for keyword and phrase-based searches.

This module provides centralized management for data sources and indexes, ensuring that data remains organized, accessible, and ready for efficient retrieval.

2. Search and Retrieval Optimization

The framework includes a Search Plan Module to intelligently optimize how data is retrieved, based on the requirements of each query:

• Search Plan Module – This engine examines the query context and generates an optimized “search plan,” selecting the most suitable index (or combination of indexes) to provide the best results. By dynamically choosing between vector, graph, and full-text search, the module can:
  • Prioritize relevant data sources based on query needs,
  • Minimize search time by directing the query to the most efficient retrieval method, and
  • Combine multiple index types for complex or hybrid queries requiring layered retrieval.

This module maximizes efficiency, enabling precise and contextually relevant searches across multiple data types and sources.

3. Answer Evaluation and Feedback

Once data has been retrieved and responses generated, the framework provides tools for ongoing evaluation and improvement:

• Evaluation Metrics – Standard metrics are provided to assess relevance, accuracy, and response quality, offering insights into how well the framework’s responses match user expectations.
• Feedback Loop – A continuous feedback module allows developers to track user interactions, fine-tuning the model over time to improve performance. This helps in refining index selection and search strategies, optimizing the RAG workflow for better results.
• User-Defined Evaluation Criteria – For specific applications, developers can customize evaluation metrics and feedback mechanisms, ensuring that performance aligns with domain-specific quality standards.

These modules are designed to work together seamlessly, offering a flexible and efficient workflow for data ingestion, optimized search, and rigorous evaluation, making it easier for developers to build reliable RAG-based applications.

Draft of the User Guide

The user guide will ensure that developers can make the most of the framework, with sections covering:

• Installation and Setup: Quick, clear instructions for deploying all modules, including setup for search plan and management tools.
• Basic Usage: Sample workflows demonstrating how to utilize the search plan, knowledge base, and index management modules.
• Customization Tips: Tips for adapting the framework to unique knowledge bases, data sources, and index configurations.
• Troubleshooting and Optimization: Common issues and tips for optimizing performance across data and index management.

[634750802]

tirag 🤣

[sykp241095]

We have renamed pingcap/tidb.ai to pingcap/autoflow, it will be the unified repo in the future, acting as a RAG (but not only RAG) framework.