RAG PIPELINES WITH LLAMAINDEX AND VECTOR DATABASES
3 Days | Introductory to Intermediate
OVERVIEW
Retrieval is where most enterprise LLM projects actually succeed or fail. A demo that answers questions over ten PDFs rarely survives contact with ten million documents, permission boundaries, tables, stale content, and users who ask questions the corpus cannot answer.
This course builds RAG the way it has to be built for production: measure first, then improve. Students start with a naive pipeline and an evaluation set, then systematically address each weakness — chunking, embedding choice, hybrid search, metadata filtering, reranking, query transformation, and multi-document routing. Students work with multiple vector stores so they can reason about index types, recall/latency trade-offs, filtering support, and operational cost rather than picking by brand.
The course closes on the hard parts: ingestion pipelines that keep an index fresh, permission-aware retrieval, structured and tabular data, agentic retrieval, and the economics of an index at scale.
WHO SHOULD ATTEND
Data engineers, backend and ML engineers, and architects responsible for search or knowledge-access features.
PREREQUISITES
Python proficiency and basic SQL. Prior LLM familiarity is helpful but not required.
WHAT YOU WILL LEARN
• Build an end-to-end RAG pipeline with LlamaIndex and defend each design choice with measurements
• Select chunking, embedding, and index strategies that fit a given corpus
• Choose and operate a vector database, including index tuning and metadata filtering
• Implement hybrid search, reranking, and query rewriting, and quantify the gain from each
• Build incremental ingestion with change detection, deletion, and reindexing
• Enforce document-level permissions in retrieval
• Measure retrieval and answer quality, and detect regressions
• Estimate and control the cost of an index at scale
COURSE OUTLINE
DAY 1 — PIPELINE FUNDAMENTALS AND MEASUREMENT
Module 1 · Why Retrieval, and Why It Breaks
• Grounding vs. fine-tuning vs. long context: choosing by constraint, not fashion
• The RAG pipeline end to end: ingest, chunk, embed, index, retrieve, rerank, synthesize
• The failure taxonomy: missed retrieval, wrong chunk, lost context, contradiction, confident nonsense
• Setting up for honest evaluation before writing pipeline code
Module 2 · Documents, Parsing, and Chunking
• Loaders and readers; PDFs, HTML, Office documents, code, and transcripts
• Layout-aware parsing; tables, headers, and figures
• Chunking strategies: fixed, sentence, semantic, hierarchical, and document-structure-aware
• Metadata extraction and why metadata drives retrieval quality
Module 3 · Embeddings and Vector Stores
• Embedding model selection: dimension, domain fit, multilingual, cost, and self-hosted options
• Similarity metrics and normalization
• Index types: flat, IVF, HNSW, and quantization trade-offs
• Vector store landscape: pgvector, Qdrant, and a managed option — filtering, scale, and operations
DAY 2 — MAKING RETRIEVAL ACTUALLY GOOD
Module 4 · Hybrid Search and Reranking
• Where dense retrieval fails: identifiers, acronyms, rare terms, exact phrases
• BM25 and keyword search; fusion strategies
• Cross-encoder reranking and the latency it buys you
• Choosing top-k honestly
Module 5 · Query Understanding
• Query rewriting, expansion, and decomposition
• Metadata filter extraction from natural language
• Routing across multiple indexes and data sources
• Handling questions the corpus cannot answer, and refusing well
Module 6 · Response Synthesis and Citations
• Synthesis modes: compact, refine, tree summarize, and their cost profiles
• Citation and provenance that survives summarization
• Context ordering and the lost-in-the-middle effect
• Groundedness checks and hallucination mitigation
Module 7 · Evaluation
• Retrieval metrics and answer metrics
• LLM-as-judge design and calibration
• Regression gates and release criteria
DAY 3 — PRODUCTION CONCERNS
Module 8 · Ingestion at Scale
• Ingestion pipelines, caching, and idempotent document IDs
• Change detection, upserts, deletions, and index drift
• Backfills and zero-downtime reindexing
• Scheduling and orchestration options
Module 9 · Security and Multi-Tenancy
• Document-level and row-level permissions in retrieval
• Pre-filter vs. post-filter, and why post-filtering leaks
• Tenant isolation strategies: namespace, filter, or separate index
• PII handling, redaction, and audit trails
Module 10 · Beyond Text Chunks
• Structured data: text-to-SQL alongside vector retrieval
• Tabular and spreadsheet content
• Knowledge graph and property-graph indexes
• Agentic retrieval: retrieval as a tool rather than a fixed pipeline stage