AGENT EVALUATION AND OBSERVABILITY WITH LANGSMITH AND LANGFUSE
2 Days | Intermediate
OVERVIEW
Agents are hard to improve because they are hard to see. A failure might come from a prompt, a retrieval miss, a tool error, a routing decision, or a model change made by a vendor last night. Without tracing and evaluation, teams ship on vibes and regress silently.
This course gives students the instrumentation and measurement discipline that makes agent development an engineering activity. Students instrument an existing agent with OpenTelemetry-based tracing, work in both LangSmith and Langfuse, and build evaluation suites that run offline in CI and online against production traffic. The course covers dataset construction, LLM-as-judge design and its calibration traps, human review workflows, regression gating, and the cost and latency analysis that drives most real optimization decisions.
Students leave with a dashboard, an evaluation suite that blocks bad releases, and an online feedback loop that turns production failures into test cases.
WHO SHOULD ATTEND
ML and software engineers responsible for LLM application quality, QA and platform engineers, and technical leads who need defensible quality metrics.
PREREQUISITES
Python proficiency and a working LLM application or agent.
WHAT YOU WILL LEARN
• Instrument agents, chains, and retrieval with complete, low-overhead traces
• Operate LangSmith and Langfuse, and articulate where each fits
• Build evaluation datasets from production traffic, synthesis, and human curation
• Write deterministic, statistical, and LLM-as-judge evaluators, and validate judges against human labels
• Run offline evaluations in CI with regression gates and release criteria
• Run online evaluations, A/B tests, and canary releases on live traffic
• Analyze cost, latency, and token usage per step and act on it
• Build human review and annotation workflows that feed back into datasets
• Monitor drift, alert on quality regressions, and debug incidents from traces
COURSE OUTLINE
DAY 1 — OBSERVABILITY AND OFFLINE EVALUATION
Module 1 · What Makes Agents Hard to Evaluate
• Non-determinism, multi-step attribution, and the absence of a single correct answer
• Quality dimensions: correctness, groundedness, task completion, safety, tone, cost, latency
• Offline vs. online evaluation; what each can and cannot tell you
• Building the measurement loop before the optimization work
Module 2 · Tracing and Instrumentation
• Traces, spans, runs, and sessions; the agent trace model
• OpenTelemetry and GenAI semantic conventions
• Instrumenting LangChain/LangGraph, LlamaIndex, and plain SDK calls
• What to capture: inputs, outputs, tokens, tool calls, retrieved context, metadata, and user IDs
• Sampling, redaction of sensitive data, and overhead
Module 3 · Datasets
• Dataset design: coverage, difficulty distribution, and known-hard cases
• Sources: production traces, synthetic generation, SME curation, adversarial cases
• Golden answers, reference-free cases, and expected-tool-call traces
• Versioning datasets and avoiding leakage
Module 4 · Writing Evaluators
• Deterministic checks: exact match, schema validity, tool-call correctness, latency and cost budgets
• Retrieval metrics for RAG steps
• LLM-as-judge: rubric design, pairwise vs. pointwise, position and verbosity bias
• Calibrating judges against human labels; measuring the judge itself
• Trajectory evaluation for multi-step agents
DAY 2 — PRODUCTION QUALITY LOOPS
Module 5 · Evaluation in CI
• Running suites on every change; prompt, model, and retrieval changes as versioned artifacts
• Regression gates, thresholds, and handling noisy metrics
• Statistical significance with small datasets; repeat runs and variance
• Comparison experiments and release criteria
Module 6 · Online Evaluation and Feedback
• Production scorers and sampled online judging
• User feedback: explicit signals, implicit signals, and their biases
• Human annotation queues and review workflows
• Closing the loop: promoting production failures into the dataset
Module 7 · Experiments and Rollout
• Prompt and model A/B testing on live traffic
• Canary releases and automatic rollback on quality metrics
• Model migration: evaluating a new model version before it is forced on you
• Prompt and configuration versioning and management
Module 8 · Cost, Latency, and Operations
• Per-step cost and token attribution; finding the expensive node
• Latency breakdown and the retrieval/rerank/generation budget
• Caching impact and measuring it honestly
• Dashboards, alerting on quality and cost drift, and on-call debugging from traces
• Self-hosting Langfuse; data residency and retention