RAG Accuracy Optimizer#

A skill for optimizing end-to-end accuracy in RAG systems.

Workflow Overview#

text

Data Design → Chunking → Indexing → Retrieval → Generation → Testing → Monitoring

Each step impacts accuracy. Optimize each step in order.

---

1. Structured Data Design#

SQL vs Vector DB — When to Use What?#

CriteriaSQL (PostgreSQL, MySQL)Vector DB (Pinecone, Qdrant, Weaviate)Exact facts (price, date, product code)✅ Optimal❌ Not suitableSemantic search (query meaning)❌ Not supported✅ OptimalAggregation (SUM, COUNT, AVG)✅ Native❌ Not supportedFuzzy matching ("similar to...")⚠️ Limited✅ Optimal**Hybrid (recommended)**pgvector for bothVector DB + SQL metadata store

Principle: Clearly structured data → SQL. Unstructured data requiring semantic understanding → Vector DB. Most production systems need both.

Schema Design Patterns by Domain#

Insurance:

text

policies(policy_id, product_type, effective_date)
clauses(clause_id, policy_id, clause_number, title, content)
exclusions(exclusion_id, clause_id, description)
-- Vector: embedding for clause.content + exclusion.description

Finance:

text

securities(ticker, name, sector, exchange)
reports(report_id, ticker, period, report_type)
sections(section_id, report_id, heading, content)
-- Vector: embedding for section.content, metadata: ticker + period

Healthcare:

text

drugs(drug_id, generic_name, brand_name, category)
guidelines(guideline_id, condition, recommendation, evidence_level)
interactions(drug_a_id, drug_b_id, severity, description)
-- Vector: embedding for guidelines.recommendation

E-commerce:

text

products(product_id, name, category, brand, price)
reviews(review_id, product_id, rating, content)
specs(product_id, attribute, value)
-- Vector: embedding for review.content + product description

Metadata Tagging Strategy#

Each chunk/document needs at minimum:

python

metadata = {
    "source": "policy_doc_v2.pdf",       # Origin
    "source_type": "pdf",                 # File type
    "domain": "insurance",                # Domain
    "category": "life_insurance",          # Classification
    "entity_id": "POL-2024-001",          # Related entity ID
    "section": "exclusions",              # Section in doc
    "chunk_index": 3,                      # Chunk position
    "total_chunks": 12,                    # Total chunks in doc
    "created_at": "2024-01-15",           # Creation date
    "version": "2.0",                      # Version
    "language": "en"                       # Language
}

Metadata principles:

• Always include source for traceability and citation
• entity_id enables pre-filtering before search → reduces noise
• chunk_index + total_chunks enables fetching surrounding context
• Domain-specific fields (clause_number, ticker, drug_id) vary by use case

Normalization vs Denormalization#

NormalizedDenormalizedProsLess duplication, easy to updateFaster queries, fewer JOINsConsRequires JOINs, slowerDuplication, harder to syncUse whenSource of truth (SQL)Vector store chunks

Recommendation: Normalized for SQL source → Denormalized when creating chunks for Vector DB. Each chunk should contain sufficient context, no JOINs needed at retrieval time.

---

2. Chunking Strategies#

Detailed code examples: read references/chunking-patterns.md

Choosing the Right Strategy#

text

Data has clear structure (clauses, sections)?
  → Semantic chunking (by heading/section)

Long, continuous data (articles, transcripts)?
  → Fixed size + overlap (512 tokens, 10-20% overlap)

Need both overview + detail?
  → Hierarchical chunking (parent-child)

Domain-specific with its own logical units?
  → Domain-specific chunking

Chunk Size Guidelines#

SizeUse caseTrade-off128-256 tokensFAQ, short definitionsHigh precision, less context256-512 tokensRecommended defaultGood balance512-1024 tokensComplex text, legal docsMore context, potential noise>1024 tokensRarely usedToo much noise

Semantic Chunking#

Split by meaning (section, topic) instead of fixed size:

python

# Split by markdown headings
# Split by paragraph breaks (\n\n)
# Split by topic change (using NLP or LLM detection)

Overlap Strategy#

• 10-20% overlap between adjacent chunks
• Ensures information at boundaries is not lost
• Chunk N ends with 1-2 opening sentences of chunk N+1

Hierarchical Chunking (Parent-Child)#

text

Document (summary)
  └── Section (heading + key points)
        └── Paragraph (details)

• Search at paragraph level (most detailed)
• When matched, pull parent section for additional context
• Keep parent_id in metadata

Domain-Specific Chunking#

• Insurance: 1 chunk = 1 clause
• Finance: 1 chunk = 1 report section, metadata = ticker + period
• Healthcare: 1 chunk = 1 guideline/recommendation
• E-commerce: 1 chunk = 1 review or 1 product description
• Legal: 1 chunk = 1 article/clause/section

Metadata Enrichment Per Chunk#

Each chunk should be enriched with:

• Summary: 1-2 sentence content summary (LLM-generated)
• Keywords: Key terms (supports BM25)
• Questions: 2-3 questions this chunk can answer (hypothetical questions)
• Entities: Named entities (product names, codes, dates)

---

3. Retrieval Optimization#

Detailed code examples: read references/retrieval-patterns.md

Recommended Retrieval Pipeline#

text

User Query
  → Query Rewriting (expand/reformulate)
  → Multi-Query Generation (3-5 variants)
  → Metadata Filtering (narrow scope)
  → Hybrid Search (Vector + BM25)
  → Merge & Deduplicate
  → Reranking (top 20 → top 5)
  → Contextual Compression
  → LLM Generation (with citations)

Hybrid Search (Vector + BM25)#

• Vector search: Find by meaning (semantic similarity)
• BM25 (keyword): Find by exact keywords (product names, codes)
• Combined: Weighted fusion or Reciprocal Rank Fusion (RRF)

text

final_score = α × vector_score + (1-α) × bm25_score
# α = 0.7 is a good starting point, tune per domain

Query Rewriting#

Use LLM to reformulate the user question for clarity:

text

User: "does insurance pay?"
→ Rewritten: "Under what circumstances does life insurance pay out benefits?"

Multi-Query#

From 1 question, generate 3-5 variants → search each variant → merge results:

text

Original: "Which bank has the highest savings rate?"
Query 1: "Compare savings interest rates across banks 2024"
Query 2: "Bank with highest deposit rate currently"
Query 3: "Top banks with best deposit interest rates"

Reranking#

After retrieval, use a reranking model to re-sort by relevance:

• Cohere Rerank: Simple API, highly effective
• Cross-encoder: More accurate than bi-encoder, but slower
• GPT Rerank: Use LLM to evaluate relevance (expensive but flexible)

Retrieve top 20 → rerank → take top 3-5 for generation.

Contextual Compression#

After reranking, compress each chunk: keep only the part relevant to the question.

text

Original chunk (500 tokens) → Compressed (150 tokens, relevant part only)

Reduces noise, saves context window, improves accuracy.

Metadata Filtering#

Narrow the search space BEFORE vector search:

python

# Instead of searching all 1M chunks:
filter = {"domain": "insurance", "product_type": "life"}
# Only search within ~50K relevant chunks
results = vector_db.search(query, filter=filter, top_k=20)

---

4. Accuracy Testing & Monitoring#

Test Suite Design#

Create ground truth Q&A pairs:

json

{
    "test_cases": [
        {
            "question": "Does life insurance pay out for suicide?",
            "expected_answer": "No payout within the first 2 years",
            "expected_source": "clause_15_exclusions.pdf",
            "category": "exclusions",
            "difficulty": "medium"
        }
    ]
}

Recommendation: Minimum 50-100 test cases, evenly distributed across categories and difficulty levels.

Metrics#

MetricMeaningTargetPrecision@K% relevant results in top K>0.8Recall@K% ground truth found in top K>0.9F1Harmonic mean of Precision and Recall>0.85MRRMean Reciprocal Rank — average position of first correct result>0.8NDCGNormalized Discounted Cumulative Gain — ranking quality>0.85Answer Accuracy% correct answers (human eval or LLM judge)>0.9

A/B Testing#

Compare strategies by running the same test suite:

text

Config A: chunk_size=256, overlap=10%, no_rerank
Config B: chunk_size=512, overlap=20%, cohere_rerank
→ Compare MRR, NDCG, Answer Accuracy
→ Choose the config with better metrics

Error Analysis Framework#

Classify errors to know where to optimize:

Error TypeCauseSolutionRetrieval MissCorrect chunk not foundImprove chunking, add hypothetical QRanking ErrorCorrect chunk found but ranked lowAdd rerankingGeneration ErrorCorrect chunk but LLM answers wrongImprove prompt, add few-shotNo AnswerInformation not in DBExpand knowledge baseHallucinationLLM fabricates informationAdd citation enforcement

Production Monitoring#

Log each query:

python

log_entry = {
    "timestamp": "2024-01-15T10:30:00",
    "query": "...",
    "retrieved_chunks": [...],
    "reranked_chunks": [...],
    "answer": "...",
    "confidence": 0.85,
    "latency_ms": 450,
    "user_feedback": None  # thumbs up/down
}

Alerts:

• Continuous confidence < 0.5 → review chunking/retrieval
• Latency > 2s → optimize index or reduce top_k
• Negative feedback > 20% → audit error patterns

---

5. Safeguards#

Hallucination Prevention#

Mandatory system prompt:

text

Answer ONLY based on the information provided in the context.
If you cannot find the information, respond: "I could not find this
information in the available data."
NEVER fabricate information.

Citation Enforcement#

Require source citations:

text

Every answer must include [Source: file_name, section/clause].
If a specific source cannot be cited, mark it as "unverified".

Confidence Thresholds#

python

if max_relevance_score < 0.3:
    return "No relevant information found."
elif max_relevance_score < 0.6:
    return answer + "\n⚠️ Low confidence. Please verify."
else:
    return answer + f"\n📎 Source: {sources}"

Answer Verification#

Cross-check the answer with the DB:

1. Extract claims from the answer (using LLM)
2. Verify each claim against retrieved chunks
3. Flag claims without supporting evidence
4. Return only verified claims

---

6. Embedding Model Selection#

Detailed comparison: read references/embedding-models.md

Quick Decision#

ScenarioModelReasonProduction, budget OKCohere embed-v4Highest MTEB, input_type optimizationProduction, low costOpenAI text-embedding-3-small$0.02/1M tokens, good qualitySelf-host, multilingualBGE-M3 ⭐Hybrid dense+sparse, 100+ languages, freeSelf-host, VietnameseBGE-M3 or multilingual-e5-largeBest for Vietnamese RAGPOC / Prototypeall-MiniLM-L6-v290MB, runs on CPU

Key Principles#

• Dimension reduction: OpenAI embed-3 supports Matryoshka — reduce 3072→512 with only ~3% quality loss
• Normalize embeddings: Always normalize_embeddings=True when encoding for cosine similarity
• Batch processing: Encode in batches (256-2000 items) instead of one at a time
• Consistency: Use the SAME model for indexing and querying

---

7. Vector DB Comparison#

Detailed comparison + HNSW tuning: read references/vector-db-comparison.md

Quick Decision#

text

Already have PostgreSQL and <5M vectors? → pgvector
Just prototype/POC? → ChromaDB
Production, want zero-ops? → Pinecone
Need performance + HNSW control? → Qdrant
Need hybrid BM25+vector built-in? → Weaviate

HNSW Tuning Quick Reference#

ParamDefaultAccuracy-criticalSpeed-criticalM1648-648-16ef_construction200400-500100-200ef (search)100200-25650-100

Trade-off: Higher M and ef → better recall but more RAM and slower. Tune per SLA.

---

8. Advanced Techniques#

Detailed code examples: read references/advanced-rag.md

Late Chunking#

Embed the entire document first, then pool embeddings by chunk boundaries. Each chunk retains context from surrounding text.

text

Traditional: Doc → Chunk → Embed each (loses context)
Late Chunking: Doc → Embed full → Pool by boundaries (retains context)

Use when: Documents have many co-references ("it", "this", "the package"). Quality gain: +5-10%.

RAPTOR (Recursive Abstractive Processing)#

Build a multi-level summary tree: Level 0 (chunks) → Level 1 (summaries) → Level 2 (summary of summaries).

Use when: Need to answer both broad queries ("Compare all insurance packages") and narrow queries ("Clause X of Package Y"). Quality gain: +10-15%.

GraphRAG (Microsoft)#

Build a knowledge graph from documents → detect communities → summarize communities → query via map-reduce.

Use when: Multi-hop reasoning, synthesize across many documents. Quality gain: +15-25% for synthesis queries. High overhead (many LLM calls when building the graph).

Combining Techniques (Production Stack)#

text

1. Late Chunking → better embeddings
2. Hybrid Search (BM25 + vector) → high recall
3. Reranking (Cohere/Cross-encoder) → high precision
4. RAPTOR → multi-level retrieval (optional)
5. GraphRAG → synthesis queries (optional, high cost)

---

9. Performance Optimization#

Caching Layer#

python

# Cache embeddings (avoid re-computation)
import hashlib, json, redis

r = redis.Redis()

def cached_embed(text, model):
    key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    embedding = model.encode([text])[0].tolist()
    r.setex(key, 3600, json.dumps(embedding))  # TTL 1h
    return embedding

# Cache search results (avoid re-searching)
def cached_search(query, search_fn, ttl=300):
    key = f"search:{hashlib.md5(query.encode()).hexdigest()}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    results = search_fn(query)
    r.setex(key, ttl, json.dumps(results))
    return results

Async Retrieval#

python

import asyncio

async def parallel_retrieve(query, retrievers):
    """Run multiple retrievers in parallel."""
    tasks = [r.search(query) for r in retrievers]
    results = await asyncio.gather(*tasks)
    return merge_and_deduplicate(results)

HNSW Index Tuning#

See details in references/vector-db-comparison.md HNSW section. Key: tune ef (search) per latency SLA, tune M per recall target.

---

10. Vietnamese-Specific RAG#

Details: read references/vietnam-nlp.md

Key Challenges#

IssueSolutionDiacritics (with vs without)Dual indexing: index both versionsCompound words ("bảo hiểm")Word segmentation (underthesea)Abbreviations (BHXH, TTCK, BLLĐ)Abbreviation expansion dictionaryVietnamese proper namesNER with underthesea/PhoBERTDomain terms (finance, law, medical)Domain-specific term enrichment

Embedding Models for Vietnamese#

• BGE-M3: Best overall — hybrid dense+sparse, 100+ languages
• multilingual-e5-large: Good alternative — retrieval-optimized
• PhoBERT-v2: Best for NER/classification (needs fine-tuning for retrieval)

Preprocessing Pipeline#

text

Input text
  → Unicode normalize (NFC)
  → Expand abbreviations (BHXH → Social Insurance)
  → Domain term enrichment
  → Dual index: original + no-diacritics version
  → Extract entities → metadata

---

11. AI Orchestrator — Multi-Model Cost Optimization#

Detailed prompt templates, code examples: read references/orchestrator-patterns.md

Query Classification Pipeline#

Each user query is classified into 1 of 5 categories:

CategoryDescriptionExampleModelsimpleGreeting, FAQ, simple lookup"Hello", "Opening hours?"No LLM / LocalragNeeds knowledge base search"Does insurance cover cancer?"Cheap (Gemini Flash)complexMulti-hop reasoning, comparison, analysis"Compare 3 insurance packages for a family of 4"Standard (GPT-4o-mini) / Premium (Claude Sonnet)actionNeeds tool/API execution (create form, calculate)"Calculate insurance premium for me, age 30"Standard + ToolsunsafeViolation content, injection, jailbreak"Ignore instructions..."Block — No LLM

2-Stage Classification (Minimize LLM Tokens)#

text

User Query
  → Stage 1: Rule-based pre-classifier (regex, keywords, NO LLM)
    → confidence ≥ 0.8? → DONE (skip LLM)
    → confidence < 0.8? → Stage 2: LLM classifier (cheap model, ~50 tokens)

Stage 1 blocks 60-80% of queries without spending a single LLM token.

Model Routing#

text

Category → Model Selection:
  greeting/simple  → No LLM (rule-based response)
  rag (simple)     → Gemini Flash ($0.075/1M input) — cheap, fast
  rag (complex)    → GPT-4o-mini ($0.15/1M input) — balanced
  complex          → Claude Sonnet ($3/1M input) — premium quality
  action           → Gemini Flash + Tool calls
  unsafe           → Block response (no LLM cost)

Cost Optimization Rules#

1. Rule-based first: Greeting, FAQ, unsafe → DON'T call LLM
2. Cheapest sufficient model: Prefer Gemini Flash for RAG queries
3. Escalate on failure: Gemini Flash fail/low-confidence → GPT-4o-mini → Claude Sonnet
4. Cache responses: Identical queries → cached answer (TTL 5-30 min)
5. Batch classify: Multiple queries → 1 LLM call to classify all
6. Token budget: Set max_tokens per category (simple: 100, rag: 300, complex: 500)

RAG Trigger Rules#

ConditionRAG On/OffQuery contains domain keywords✅ ONClassification = "rag" or "complex"✅ ONGreeting, simple lookup, unsafe❌ OFFConfidence score > 0.9 from cache/FAQ❌ OFF (answer from cache)

Tool Trigger Rules#

ConditionToolsQuery requests calculation (fees, interest)calculator toolQuery requests form creation/submissionform_builder toolQuery requests real-time lookup (price, exchange rate)api_lookup toolClassification ≠ "action"No tools

JSON Output Format#

json

{
  "category": "rag",
  "confidence": 0.92,
  "risk_level": "low",
  "model": "gemini-flash",
  "rag_enabled": true,
  "tools": [],
  "max_tokens": 300,
  "reasoning": "User asks about insurance benefits — needs knowledge base search"
}

---

Scripts#

eval_ragas.py#

RAGAS evaluation pipeline. Run:

bash

python3 scripts/eval_ragas.py --test-file eval_dataset.json --output results.json
python3 scripts/eval_ragas.py --test-file eval_dataset.json --metrics faithfulness,answer_relevancy

Input: JSON file with test cases (question, answer, contexts, ground_truth). Output: metrics report + threshold checks. Requires: pip install ragas langchain-openai datasets

embedding_benchmark.py#

Benchmark embedding models on a Vietnamese dataset. Run:

bash

python3 scripts/embedding_benchmark.py --models bge-m3,multilingual-e5 --dataset vi_pairs.json
python3 scripts/embedding_benchmark.py --models all --quick  # Use built-in test pairs

Input: JSON file with query-positive-negative pairs. Output: accuracy + latency comparison. Requires: pip install sentence-transformers numpy torch

chunk_optimizer.py#

Evaluate chunk quality. Run:

bash

python3 scripts/chunk_optimizer.py --input chunks.jsonl --output report.json

Input: JSONL file, each line is {"text": "...", "metadata": {...}}. Output: quality report with scores.

accuracy_test.py#

Test framework for RAG accuracy. Run:

bash

python3 scripts/accuracy_test.py --test-file tests.json --results-dir ./results

Input: JSON file with test cases (question, expected_answer, expected_source). Output: metrics report.

---

References#

• references/chunking-patterns.md — Python code examples for chunking strategies
• references/retrieval-patterns.md — Code examples for hybrid search, reranking, multi-query
• references/embedding-models.md — Detailed embedding model comparison (OpenAI, Cohere, BGE-M3, PhoBERT...)
• references/vector-db-comparison.md — Vector DB comparison + HNSW tuning guide
• references/advanced-rag.md — Late Chunking, RAPTOR, GraphRAG with code examples
• references/testing-frameworks.md — RAGAS, LLM-as-Judge, Adversarial testing
• references/vietnam-nlp.md — Vietnamese NLP: diacritics, abbreviations, NER, domain terms
• references/orchestrator-patterns.md — Multi-model orchestrator: prompt templates, rule-based pre-classifier, cost comparison, fallback chain, monitoring