Corrective Retrieval-Augmented Generation (CRAG)

An implementation of Corrective Retrieval-Augmented Generation (CRAG) based on the research paper Corrective Retrieval Augmented Generation (arXiv: 2401.15884) using LangGraph, Ollama (Qwen2.5-Coder:7b), and monoT5-Large for document and strip-level evaluation.

Architecture

This pipeline implements the core concepts from the CRAG paper:

1. Lightweight Retrieval Evaluator: Integrates a local castorini/monot5-large-msmarco pointwise reranker.
2. Confidence-Based Action Triggering:
   • CORRECT: Refines retrieved internal documents.
   • INCORRECT: Ignores internal documents and triggers a web search fallback.
   • AMBIGUOUS: Merges and refines both internal and external web search documents.
3. Decompose-then-Recompose (Knowledge Refinement): Sentence-level decomposition and filtering using the T5 evaluator.
4. Query Rewriter: Translates natural language questions into search engine keywords.

Installation

1. Install the required dependencies:

   pip install -r requirements.txt

2. Create a .env file based on .env.example and add your Tavily API Key:

   TAVILY_API_KEY=your_tavily_api_key_here

3. Ensure Ollama is running locally with the target model:

   ollama run qwen2.5-coder:7b

4. Place your PDF files under a directory named documents/.

5. Verify your environment setup before running:

   python verify_env.py

6. Run the pipeline:

   python crag_pipeline.py

Tuning Thresholds

The pipeline's behavior can be tuned in the crag_pipeline.py configuration section using the following parameters:

• UPPER_TH (default 0.70): Documents scoring above this threshold are deemed highly relevant, triggering the CORRECT path.
• LOWER_TH (default 0.30): If all documents score below this threshold, the INCORRECT path is triggered, initiating web search fallback.
• STRIP_TH (default 0.40): Used during the decompose-then-recompose step to filter out individual sentence strips.