Contextual compression is a technique used to reduce the size of the input context for a Large Language Model (LLM) while retaining its most relevant information, typically by summarizing or filtering.
2 min readupdated 2026-08-04
/ quick answer
A set of techniques applied to raw input context (e.g., retrieved documents, conversation history) to reduce its token count before feeding it to a Large Language Model, typically involving summarization, filtering, or selective extraction of the most relevant information. Contextual compression is a technique used to reduce the size of the input context for a Large Language Model…
Large Language Models have finite context windows, and providing too much irrelevant or verbose information can lead to degraded performance, increased latency, and higher costs. Contextual compression addresses this challenge by intelligently reducing the volume of text fed to the LLM. This process isn't just about truncation; it involves methods like extractive summarization (selecting key sentences) or abstractive summarization (generating new, concise summaries) to preserve meaning and relevance. It can also include filtering out redundant or low-relevance information identified through semantic similarity or keyword matching. Effective contextual compression is vital for maintaining the efficiency and accuracy of LLM applications, especially in long-running conversations or when dealing with extensive knowledge bases.
Definition
A set of techniques applied to raw input context (e.g., retrieved documents, conversation history) to reduce its token count before feeding it to a Large Language Model, typically involving summarization, filtering, or selective extraction of the most relevant information.
Example
Instead of passing a full 20-page document found via RAG to the LLM, contextual compression might use a mini-LLM or a summarization algorithm to condense it into a 200-word summary, or extract only the three most semantically relevant paragraphs, before sending it along with the user query to the main LLM.
What's the difference between compression and chunking?
Chunking is the process of breaking down a large document into smaller, predefined segments (chunks) for easier storage and retrieval. Compression, on the other hand, actively reduces the content within those chunks or retrieved segments by summarizing or filtering, aiming to maintain core information while reducing token count.
Can contextual compression lead to loss of important details?
Yes, there's always a trade-off. Over-compression can lead to the loss of nuanced or specific details. The key is to balance reduction in token count with the preservation of critical information. Advanced compression methods often prioritize information based on its relevance to the current user query or task.