RALM_Survey

This is a repository of RALM surveys containing a summary of state-of-the-art RAG and other technologies according to according to our survey paper: RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing . In this repository, we will present the most central research approach of our thesis as well as keep up-to-date with work on RALM in the most accessible way possible. For more detailed information, please read our papers. Please cite our papers if you think they can help you with your research!

News

This project is under development. You can hit the STAR and WATCH to follow the updates.

• Our survey：RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing on RALM is now public.

Citation Information:

@article{hu2024rag,
  title={RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing},
  author={Hu, Yucheng and Lu, Yuxing},
  journal={arXiv preprint arXiv:2404.19543},
  year={2024}
}

Table of Contents

• RALM_Survey
  • News
  • Overview
  • Table of Contents
  • Related Surveys
  • Definition
    • Sequential Single Interaction
    • Sequential Multiple Interactions
    • Parallel Interaction
  • Retriever
    • Sparse Retrieval
    • Dense Retrieval
    • Internet Retrieval
    • Hybrid Retrieval
  • Language Models
    • AutoEncoder Language Model
    • AutoRegressive Language Model
    • Encoder-Decoder Language Model
  • RALM Enhancement
    • Retriever Enhancement
    • LM Enhancement
    • Overall Enhancement
  • Data Source
    • Structured Data
    • Unstructured Data
  • Applications
    • RALM on NLG
    • RALM on NLU
    • RALM on Both NLU and NLG
  • Evaluation

Overview

This SURVEY of ours summarizes multiple aspects of RALM, including: definition, retriever, LM, enhancement, data source, application, evaluation, and more.

We hope this repository can help researchers and practitioners to get a better understanding of RALM.

Related Surveys

• Retrieval-Augmented Generation for AI-Generated Content: A Survey(Arxiv, 2024)[paper]
• A Survey on Retrieval-Augmented Text Generation(Arxiv, 2022)[paper]
• Retrieving Multimodal Information for Augmented Generation: A Survey(Arxiv, 2023)[paper]
• Retrieval-Augmented Generation for Large Language Models: A Survey(Arxiv, 2024)[paper]

Definition

Sequential Single Interaction

• Corrective Retrieval Augmented Generation(Arxiv, 2024)[paper]
• SELF-RAG: LEARNING TO RETRIEVE, GENERATE, AND CRITIQUE THROUGH SELF-REFLECTION(Arxiv, 2023)[paper]
• Atlas: Few-shot Learning with Retrieval Augmented Language Models(Arxiv, 2023)[paper]
• Efficient Retrieval Augmented Generationfrom Unstructured Knowledge for Task-Oriented Dialog(Arxiv, 2021)[paper]
• FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation(Arxiv, 2024)[paper]
• FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation(acm, 2023)[paper]
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering(mit, 2024)[paper]
• End-to-End Training of Neural Retrievers for Open-Domain Question Answering(Arxiv, 2021)[paper]
• REALM: Retrieval-Augmented Language Model Pre-Training(mlr, 2020)[paper]
• In-Context Retrieval-Augmented Language Models(mit, 2023)[paper]
• Learning to Filter Context for Retrieval-Augmented Generation(Arxiv, 2023)[paper]
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text(Arxiv, 2022)[paper]
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(neurips, 2020)[paper]
• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering(Arxiv, 2021)[paper]
• Improving Language Models by Retrieving from Trillions of Tokens(mlr, 2022)[paper]
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories(Arxiv, 2023)[paper]
• Check Your Facts and Try Again: Improving Large Language Models(Arxiv, 2023)[paper]
• RA-DIT: RETRIEVAL-AUGMENTED DUAL INSTRUCTION TUNING(Arxiv, 2023)[paper]
• SAIL: Search-Augmented Instruction Learning(Arxiv, 2023)[paper]
• MAKING RETRIEVAL-AUGMENTED LANGUAGE MODELS ROBUST TO IRRELEVANT CONTEXT(Arxiv, 2023)[paper]
• RECOMP: IMPROVING RETRIEVAL-AUGMENTED LMS WITH COMPRESSION AND SELECTIVE AUGMENTATION(Arxiv, 2023)[paper]
• Latent Retrieval for Weakly Supervised Open Domain Question Answering(Arxiv, 2019)[paper]
• End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering(neurips, 2021)[paper]
• DISTILLING KNOWLEDGE FROM READER TO RETRIEVER FOR QUESTION ANSWERING(Arxiv, 2022)[paper]
• REPLUG: Retrieval-Augmented Black-Box Language Models(Arxiv, 2023)[paper]
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory(thecvf, 2023)[paper]
• Neural Argument Generation Augmented with Externally Retrieved Evidence(Arxiv, 2018)[paper]

Sequential Multiple Interactions

• Active Retrieval Augmented Generation(Arxiv, 2023)[paper]
• Rethinking with Retrieval: Faithful Large Language Model Inference(Arxiv, 2023)[paper]
• DEMONSTRATE–SEARCH–PREDICT:Composing retrieval and language models for knowledge-intensive NLP(Arxiv, 2023)[paper]
• Improving Language Models via Plug-and-Play Retrieval Feedback(Arxiv, 2023)[paper]
• Retrieval Augmentation Reduces Hallucination in Conversation(Arxiv, 2021)[paper]
• KAUCUS - Knowledgeable User Simulators for Training Large Language Models [paper]
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia (Findings of EMNLP 2023) [paper] [code] [demo]

Parallel Interaction

• GENERALIZATION THROUGH MEMORIZATION:NEAREST NEIGHBOR LANGUAGE MODELS(Arxiv, 2020)[paper]
• Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval(mlr, 2022)[paper]
• Efficient Nearest Neighbor Language Models(Arxiv, 2021)[paper]
• You can’t pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM(Arxiv, 2022)[paper]

Retriever

Sparse Retrieval

• Okapi at TREC-3(google, 1995)[paper]
• Learning to retrieve passages without supervision(Arxiv, 2022)[paper]
• Generation-Augmented Retrieval for Open-Domain Question Answering(Arxiv, 2021)[paper]
• GENERALIZATION THROUGH MEMORIZATION:NEAREST NEIGHBOR LANGUAGE MODELS(Arxiv, 2020)[paper]
• Adaptive Semiparametric Language Models(mit, 2021)[paper]
• MemPrompt: Memory-assisted Prompt Editing with User Feedback(Arxiv, 2023)[paper]

Dense Retrieval

• Unsupervised Dense Information Retrieval with Contrastive Learning(Arxiv, 2022)[paper]
• Large Dual Encoders Are Generalizable Retrievers(Arxiv, 2021)[paper]
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction(Arxiv, 2022)[paper]
• How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval(Arxiv, 2023)[paper]
• Dense Passage Retrieval for Open-Domain Question Answering(Arxiv, 2020)[paper]
• REPLUG: Retrieval-Augmented Black-Box Language Models(Arxiv, 2023)[paper]
• End-to-End Training of Neural Retrievers for Open-Domain Question Answering(Arxiv, 2021)[paper]
• REALM: Retrieval-Augmented Language Model Pre-Training(mlr, 2020)[paper]
• Latent Retrieval for Weakly Supervised Open Domain Question Answering(Arxiv, 2019)[paper]
• End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering(neurips, 2021)[paper]
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text(Arxiv, 2022)[paper]
• RE-IMAGEN: RETRIEVAL-AUGMENTED TEXT-TO-IMAGE GENERATOR(Arxiv, 2022)[paper]
• MEMORY-DRIVEN TEXT-TO-IMAGE GENERATION(Arxiv, 2022)[paper]
• Retrieval-Augmented Diffusion Models(neurips, 2022)[paper]

Internet Retrieval

• Active Retrieval Augmented Generation(Arxiv, 2023)[paper]
• MAKING RETRIEVAL-AUGMENTED LANGUAGE MODELS ROBUST TO IRRELEVANT CONTEXT(Arxiv, 2023)[paper]
• Internet-augmented dialogue generation(Arxiv, 2021)[paper]
• Webgpt: Browser-assisted question-answering with human feedback(Arxiv, 2022)[paper]

Hybrid Retrieval

• SAIL: Search-Augmented Instruction Learning(Arxiv, 2023)[paper]
• Internet-augmented language models through few-shot prompting for open-domain question answering(Arxiv, 2022)[paper]
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory(thecvf, 2023)[paper]
• Neural Argument Generation Augmented with Externally Retrieved Evidence(Arxiv, 2018)[paper]
• Boosting search engines with interactive agents(Arxiv, 2022)[paper]
• Off the beaten path: Let’s replace term-based retrieval with k-NN search(Arxiv, 2016)[paper]

Language Models

AutoEncoder Language Model

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(Arxiv, 2019)[paper]
• RoBERTa: A Robustly Optimized BERT Pretraining Approach(Arxiv, 2019)[paper]
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter(Arxiv, 2020)[paper]
• ConvBERT: Improving BERT with Span-based Dynamic Convolution(neurips, 2020)[paper]

AutoRegressive Language Model

• Llama 2: Open Foundation and Fine-Tuned Chat Models(Arxiv, 2023)[paper]
• GPT-4 Technical Report(Arxiv, 2024)[paper]
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model(Arxiv, 2022)[paper]
• OPT: Open Pre-trained Transformer Language Models(Arxiv, 2022)[paper]
• LLaMA: Open and Efficient Foundation Language Models(Arxiv, 2023)[paper]
• Few-shot Learning with Multilingual Generative Language Models(aclanthology, 2022)[paper]
• QWEN TECHNICAL REPORT(Arxiv, 2022)[paper]
• Language Models are Unsupervised Multitask Learners(amazonaws, 2019)[paper]
• ADAPTIVE INPUT REPRESENTATIONS FOR NEURAL LANGUAGE MODELING(Arxiv, 2019)[paper]
• Mistral 7B(Arxiv, 2023)[paper]
• Language models are few-shot learners(neurips, 2020)[paper]
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model(hal open science, 2023)[paper]

Encoder-Decoder Language Model

• Attention Is All You Need(neurips, 2017)[paper]
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer(jmlr, 2020)[paper]
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension(Arxiv, 2019)[paper]

RALM Enhancement

Retriever Enhancement

• Corrective Retrieval Augmented Generation(Arxiv, 2024)[paper]
• SELF-RAG: LEARNING TO RETRIEVE, GENERATE, AND CRITIQUE THROUGH SELF-REFLECTION(Arxiv, 2023)[paper]
• RA-DIT: RETRIEVAL-AUGMENTED DUAL INSTRUCTION TUNING(Arxiv, 2023)[paper]
• MAKING RETRIEVAL-AUGMENTED LANGUAGE MODELS ROBUST TO IRRELEVANT CONTEXT(Arxiv, 2023)[paper]
• RECOMP: IMPROVING RETRIEVAL-AUGMENTED LMS WITH COMPRESSION AND SELECTIVE AUGMENTATION(Arxiv, 2023)[paper]
• Learning to Filter Context for Retrieval-Augmented Generation(Arxiv, 2023)[paper]
• Active Retrieval Augmented Generation(Arxiv, 2023)[paper]
• In-Context Retrieval-Augmented Language Models(mit, 2023)[paper]
• Improving Language Models via Plug-and-Play Retrieval Feedback(Arxiv, 2023)[paper]
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories(Arxiv, 2023)[paper]
• Rethinking with Retrieval: Faithful Large Language Model Inference(Arxiv, 2023)[paper]
• DEMONSTRATE–SEARCH–PREDICT:Composing retrieval and language models for knowledge-intensive NLP(Arxiv, 2023)[paper]

LM Enhancement

• FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation(acm, 2023)[paper]
• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering(Arxiv, 2021)[paper]
• End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering(neurips, 2021)[paper]
• DISTILLING KNOWLEDGE FROM READER TO RETRIEVER FOR QUESTION ANSWERING(Arxiv, 2022)[paper]
• Scaling Instruction-Finetuned Language Models(jmlr, 2024)[paper]
• RA-DIT: RETRIEVAL-AUGMENTED DUAL INSTRUCTION TUNING(Arxiv, 2023)[paper]
• SAIL: Search-Augmented Instruction Learning(Arxiv, 2023)[paper]
• MemPrompt: Memory-assisted Prompt Editing with User Feedback(Arxiv, 2023)[paper]
• Internet-augmented language models through few-shot prompting for open-domain question answering(Arxiv, 2022)[paper]
• GENERALIZATION THROUGH MEMORIZATION:NEAREST NEIGHBOR LANGUAGE MODELS(Arxiv, 2020)[paper]
• Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval(mlr, 2022)[paper]
• Efficient Nearest Neighbor Language Models(Arxiv, 2021)[paper]
• You can’t pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM(Arxiv, 2022)[paper]
• Training Language Models with Memory Augmentation(Arxiv, 2022)[paper]
• IMPROVING NEURAL LANGUAGE MODELS WITH A CONTINUOUS CACHE(Arxiv, 2016)[paper]
• Adaptive Semiparametric Language Models(mit, 2021)[paper]

Overall Enhancement

• REALM: Retrieval-Augmented Language Model Pre-Training(mlr, 2020)[paper]
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(neurips, 2020)[paper]
• End-to-End Training of Neural Retrievers for Open-Domain Question Answering(Arxiv, 2021)[paper]
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering(mit, 2024)[paper]
• End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering(neurips, 2021)[paper]
• Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory(neurips, 2023)[paper]
• Check Your Facts and Try Again: Improving Large Language Models(Arxiv, 2023)[paper]

Data Source

Structured Data

• Natural Questions: A Benchmark for Question Answering Research(mit, 2019)[paper]
• HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering(Arxiv, 2018)[paper]
• Wikidata5M-SI(madata, 2023)[dataset]
• OGB-LSC: WIKIKG90MV2 TECHNICAL REPORT(stanford, 2023)[paper]
• OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs(aclanthology, 2019)[paper]

Unstructured Data

• SQuAD: 100,000+ Questions for Machine Comprehension of Text(Arxiv, 2016)[paper]
• FEVER: a large-scale dataset for Fact Extraction and VERification(Arxiv, 2018)[paper]
• MULTIMODALQA: COMPLEX QUESTION ANSWERING OVER TEXT, TABLES AND IMAGES(Arxiv, 2021)[paper]
• LAION-5B: An open large-scale dataset for training next generation image-text models(neurips, 2021)[paper]
• AudioCaps: Generating Captions for Audios in The Wild(aclanthology, 2019)[paper]
• AUDIO SET: AN ONTOLOGY AND HUMAN-LABELED DATASET FOR AUDIO EVENTS(googleapis, 2022)[paper]
• Clotho: an Audio Captioning Dataset(academia, 2020)[paper]
• VideoQA: question answering on news video(academia, 2003)[paper]

Applications

RALM on NLG

• Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory(neurips, 2023)[paper]
• Training Language Models with Memory Augmentation(Arxiv, 2022)[paper]
• Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference(Arxiv, 2023)[paper]
• Learning to Filter Context for Retrieval-Augmented Generation(Arxiv, 2023)[paper]
• RA-DIT: RETRIEVAL-AUGMENTED DUAL INSTRUCTION TUNING(Arxiv, 2023)[paper]
• REPLUG: Retrieval-Augmented Black-Box Language Models(Arxiv, 2023)[paper]
• Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation(Arxiv, 2023)[paper]

RALM on NLU

• Robust Retrieval Augmented Generation for Zero-shot Slot Filling(Arxiv, 2021)[paper]
• MEMORY-DRIVEN TEXT-TO-IMAGE GENERATION(Arxiv, 2022)[paper]
• Retrieval-Augmented Diffusion Models(neurips, 2022)[paper]
• RE-IMAGEN: RETRIEVAL-AUGMENTED TEXT-TO-IMAGE GENERATOR(Arxiv, 2022)[paper]
• KNN-Diffusion: Image Generation via Large-Scale Retrieval(Arxiv, 2022)[paper]
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(neurips, 2020)[paper]
• Atlas: Few-shot Learning with Retrieval Augmented Language Models(Arxiv, 2023)[paper]
• Learning to Filter Context for Retrieval-Augmented Generation(Arxiv, 2023)[paper]
• Retrieval-Enhanced Generative Model for Large-Scale Knowledge Graph Completion(ACM, 2023)[paper]
• Active Retrieval Augmented Generation(Arxiv, 2023)[paper]
• Learning to retrieve in-context examples for large language models(Arxiv, 2023)[paper]
• Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training(Arxiv, 2022)[paper]
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy(Arxiv, 2023)[paper]

RALM on Both NLU and NLG

• RA-DIT: RETRIEVAL-AUGMENTED DUAL INSTRUCTION TUNING(Arxiv, 2023)[paper]
• Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory(neurips, 2023)[paper]
• Learning to retrieve in-context examples for large language models(Arxiv, 2023)[paper]
• Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training(Arxiv, 2022)[paper]
• Augmented Large Language Models with Parametric Knowledge Guiding(Arxiv, 2023)[paper]
• Think and Retrieval: A Hypothesis Knowledge Graph Enhanced Medical Large Language Models(Arxiv, 2023)[paper]
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(neurips, 2020)[paper]
• In-Context Retrieval-Augmented Language Models(mit, 2023)[paper]
• Learning to Filter Context for Retrieval-Augmented Generation(Arxiv, 2023)[paper]
• REPLUG: Retrieval-Augmented Black-Box Language Models(Arxiv, 2023)[paper]
• Active Retrieval Augmented Generation(Arxiv, 2023)[paper]
• Rethinking with Retrieval: Faithful Large Language Model Inference(Arxiv, 2023)[paper]
• DEMONSTRATE–SEARCH–PREDICT:Composing retrieval and language models for knowledge-intensive NLP(Arxiv, 2023)[paper]
• Retrieval Augmented Code Generation and Summarization(Arxiv, 2021)[paper]
• When language model meets private library(Arxiv, 2022)[paper]
• RACE: Retrieval-Augmented Commit Message Generation(Arxiv, 2022)[paper]
• RETRIEVAL-AUGMENTED GENERATION FOR CODE SUMMARIZATION VIA HYBRID GNN(Arxiv, 2021)[paper]

Evaluation

• RAGAS: Automated Evaluation of Retrieval Augmented Generation(Arxiv, 2023)[paper]
• Benchmarking Large Language Models in Retrieval-Augmented Generation(AAAI, 2024)[paper]
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models(Arxiv, 2024)[paper]
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems(Arxiv, 2024)[paper]
• Recall: A benchmark for llms robustness against external counterfactual knowledge(Arxiv, 2023)[paper]
• Benchmarking Retrieval-Augmented Generation for Medicine(Arxiv, 2024)[paper]