[NAACL 2025 Main] Prompt Compression for Large Language Models: A Survey

Content

🚀 News • ✏️ Todo • ✨ Introduction

👀 Examples • 🌳 Tree Overview • 📖 Paper List 🎨 Visualisations

📌 Citation • 🔖 License

 

Links

Project Page • Paper

🚀 News

• [2025.01.22] This paper was accepted by NAACL 2025 Main!
• [2024.10.16] The paper was uploaded to Arxiv.

 

 

 

✏️ Todo

• Release the full README page.

 

 

 

✨ Introduction

 

 

 

👀 Examples

 

 

 

🌳 Tree Overview

 

 

 

📖 Paper List

Hard Prompt Methods:

• Filtering:

  • General:

    • [SelectiveContext] Compressing Context to Enhance Inference Efficiency of Large Language Models
    • [LLMLingua] Compressing Prompts for Accelerated Inference of Large Language Models
    • [LongLLMLingua] Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
    • [AdaComp] Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models

  • Distillation Enhanced:

    • [LLMLingua-2] Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

  • RL Enhanced:

    • [TACO-RL] Task Aware Prompt Compression Optimization with Reinforcement Learning
    • [PCRL] Priority Convention Reinforcement Learning for Microscopically Sequencable Multi-agent Problems

  • Embedding Enhanced:

    • [CPC] Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
    • [TCRA-LLM] Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction

• Paraphrasing:

  • (No sub category)

    • [Nano-Capsulator] Learning to Compress Prompt in Natural Language Formats
    • [CompAct] Compressing Retrieved Documents Actively for Question Answering
    • [FAVICOMP] Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation

Soft Prompt Methods:

• Decoder Only:

  • Not Finetuned:

    • [CC] Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

  • Finetuned:

    • [GIST] Learning to Compress Prompts with Gist Tokens
    • [AutoCompressor] Adapting Language Models to Compress Contexts

• Encoder-decoder:

  • Both Finetuned:

    • [COCOM] Context Embeddings for Efficient Answer Generation in RAG
    • [LLoCO] Learning Long Contexts Offline

  • Finetuned Encoder:

    • [ICAE] In-context Autoencoder for Context Compression in a Large Language Model
    • [500xCompressor] Generalized Prompt Compression for Large Language Models
    • [QGC] Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs

  • Embedding Encoder:

    • [xRAG] Extreme Context Compression for Retrieval-augmented Generation with One Token

  • Projector:

    • [UniICL] Unifying Demonstration Selection and Compression for In-Context Learning

Applications:

• RAG:

  • (No sub category)

    • [xRAG] Extreme Context Compression for Retrieval-augmented Generation with One Token
    • [RECOMP] Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation
    • [COCOM] Context Embeddings for Efficient Answer Generation in RAG
    • [CompAct] Compressing Retrieved Documents Actively for Question Answering
    • [FAVICOMP] Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation
    • [AdaComp] Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models
    • [LLoCO] Learning Long Contexts Offline
    • [TCRA-LLM] Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction

• Agents:

  • (No sub category)

    • [HD-Gist] Hierarchical and Dynamic Prompt Compression for Efficient Zero-shot API Usage
    • [Link] Concise and Precise Context Compression for Tool-Using Language Models

• Domain-specific tasks:

  • (No sub category)

    • [Tag-llm] Repurposing General-Purpose LLMs for Specialized Domains
    • [CoLLEGe] Concept Embedding Generation for Large Language Models

• Others:

  • (No sub category)

    • [ICL] Unifying Demonstration Selection and Compression for In-Context Learning
    • [Role Playing] Extensible Prompts for Language Models on Zero-shot Language Style Customization
    • [Functions] Function Vectors in Large Language Models

 

 

 

🎨 Visualisations

 

 

 

📌 Citation

@misc{li2024promptcompressionlargelanguage,
      title={Prompt Compression for Large Language Models: A Survey}, 
      author={Zongqian Li and Yinhong Liu and Yixuan Su and Nigel Collier},
      year={2024},
      eprint={2410.12388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.12388}, 
}

 

 

 

🔖 License

This project is licensed under the Creative Commons Attribution 4.0 International License - see the LICENSE for details.