A hands-on, progressive introduction to evaluating LLM applications, built around a single example app: an HR Onboarding agent. You learn the app once, then spend the rest of the workshop on evaluation technique — from a first single-turn eval all the way to monitoring evals in production.
Stack: Python + LangChain (create_agent) + LangSmith.
The workshop runs in two sessions:
- Session 1 — Foundations (Modules 1–4): what an eval is, single-turn and agent evals, and gating a PR with an offline suite.
- Session 2 — Evals for Production (Modules 5–7): online evals on live traces, improving evaluators with annotation + few-shot, and monitoring production for quality drift.
Teams at different stages, in the same room: some who run evals daily, some who are just starting. Modules 1–2 are the shared vocabulary — quick if you know it, essential if you don't. Everything after that is where experienced teams spend their time. Basic Python and comfort with the command line is enough; no prior eval experience is assumed.
You don't have to read in order. Pick the row that sounds like your problem:
| If you… | Start with | Then |
|---|---|---|
| are new to evals | Module 1 | in order |
| have datasets whose ground truth goes stale because the underlying data keeps changing | Module 3 → stale data | Module 4 to gate on it |
| reach for an LLM judge by default | Choosing an evaluator | Module 6 to validate the ones you keep |
| have agents in production and want to see what they're doing | Module 5 → Trajectory view | Module 6 → Engine |
| evaluate inputs that aren't text: images, or several files per example | Module 2 → multimodal | Module 6 to validate the image judge |
| are wiring evals into CI | Module 4 | Module 7 for scheduled monitoring |
Start at the top and stop at the first row that answers your question:
| The question is… | Use | Because |
|---|---|---|
| a fact you can look up or compute — an ID exists, a date matches, a field is valid, an argument equals a tool's output | Deterministic check (code) | Free, exact, never flakes. Most "is it broken?" questions are this. |
| a narrow, typed call about meaning you can read straight off the text — yes/no, or one of a few labels — that you need to make at volume | System 1 model (e.g. Jev) | Cheap, fast, low variance, and it returns probabilities you can route on. It doesn't reason or explain, so nothing that has to be derived. |
| open-ended — tone, whether free text is grounded, "was that a reasonable approach" — or needs a written rationale | LLM judge | The only one that reasons and explains. It costs money and is noisy, so validate it. |
| high-stakes, or you aren't sure the judge is right | Human review | The ground truth the others are measured against. Send low-confidence answers here. |
Whichever you pick, measure it against labeled examples on a held-out slice before you trust it (Module 6). The System 1 row is the newest and least proven: Jev is early access, sends text to a third party, and the example here hasn't been run against the live API. The full ladder, worked examples, and that example are in Choosing an evaluator.
SESSION 1 — Foundations SESSION 2 — Evals for Production
Module 1 Module 2 Module 3 Module 4 │ Module 5 Module 6 Module 7
Fundamentals -> Single-turn evals -> Agent evals -> Evals in │ Online evals -> Improving evals -> Production
(the 4 parts) (judge the answer) (the process) CI (gate) │ (live traces) (annotate+few-shot) monitor (drift)
| Module | You learn to… | Evaluators introduced |
|---|---|---|
| 1 — Fundamentals | Name the 4 parts of any eval; run one end-to-end. | first deterministic check |
| 2 — Single-turn | Judge a single answer, and decide when a judge is warranted at all. | deterministic (facts, shape validation) + LLM-judge (correctness, groundedness, tone) + a decision ladder + a System 1 (Jev) judge vs labels example + multimodal (receipt images, several attachments per example, an image-aware judge) |
| 3 — Agent evals | Judge the trajectory and tool calls, not just the answer. Mock tool outputs to reach failure states. Replay recorded tool outputs when the data changes under your ground truth. | trajectory (exact / required / forbidden / efficiency), tool-args, LLM trajectory judge, failure handling, point-in-time replay + invariants |
| 4 — CI | Gate a build on eval results. | pytest per-example gate + aggregate threshold gate + GitHub Actions |
| Module | You learn to… | What's new |
|---|---|---|
| 5 — Online evals | Tell offline experiments from online evals; score live traces with no ground truth; read whole sessions. | reference-free evaluators, scoring traces + writing feedback, the data flywheel, threads + the Trajectory view |
| 6 — Improving evals | Align an LLM judge to your humans; promote traces into a dataset; see what LangSmith Engine automates of that loop. | annotation queues, "evaluate the evaluator", few-shot judge alignment, production→dataset curation, Engine |
| 7 — Production CI | Monitor production for quality drift. | scheduled monitor, baseline/drift alerting, scheduled GitHub Actions |
The same HR agent (hr_agent/) is the system-under-test in every module.
Lives in hr_agent/. A tool-calling agent that:
- answers HR policy/benefits questions (single-turn material), and
- performs onboarding actions — provisioning accounts, ordering equipment, scheduling orientation (multi-step trajectory material).
Tools are deterministic (they read static mock data), so the same input always produces the same tool output. That reproducibility is what makes evaluation meaningful — you measure the model's behavior, not flaky downstream systems.
The flip side: tools that always succeed can't test what happens when they
don't. hr_agent/mocking.py is middleware that swaps
tool outputs for canned ones — per test case — so Module 3 can evaluate the
agent against outages, unknown employees, and partial failures the real tools
could never return.
The opposite problem shows up in production: real tools return changing data, so
an answer key written today is wrong next month.
hr_agent/replay.py is middleware that replays the tool
outputs recorded in a trace — matched on tool name and arguments — while the agent
itself is re-run live. Same hook, different job: mocking invents a world, replay
freezes one that existed.
# 1. Python 3.10+ and a virtualenv
python -m venv .venv && source .venv/bin/activate
# 2. Install
pip install -r requirements.txt
# 3. Configure keys
cp .env.example .env # then edit: LANGSMITH_API_KEY + ANTHROPIC_API_KEYYou need:
- a LangSmith API key (smith.langchain.com → Settings → API Keys), and
- an Anthropic API key (or set
WORKSHOP_MODEL=openai:gpt-4o-miniand anOPENAI_API_KEY— seeconfig.py/.env.example).
The deterministic evaluators self-test as pure functions:
python module_2_single_turn/deterministic_evals.py
python module_3_agent_evals/trajectory_evals.py
python module_3_agent_evals/tool_evals.py
python module_3_agent_evals/replay_evals.py
python module_5_online_evals/reference_free_evals.py
python hr_agent/mocking.py
python hr_agent/replay.py
python module_2_single_turn/system1_client.py
python module_2_single_turn/system1_judge.py
python module_2_single_turn/system1_labels.py
python module_2_single_turn/system1_alignment.py --self-test
python module_2_single_turn/system1_alignment.py # phrase-list baseline vs labels# Session 1 — offline experiments
python module_1_fundamentals/01_first_eval.py
python module_2_single_turn/run_eval.py
python module_3_agent_evals/run_eval.py
python module_3_agent_evals/mocked_eval.py # mocked tool failures
python module_3_agent_evals/replay_eval.py # stale data: frozen vs drifted world
python module_4_ci/ci_gate.py --suite agent
# Session 2 — production evals
python module_5_online_evals/production_traffic.py # create live traces (threads)
python module_5_online_evals/score_traces.py # online-eval loop
python module_6_improving_evals/judge_alignment.py # zero-shot vs few-shot
python module_6_improving_evals/curate_dataset.py # traces -> dataset (flywheel)
python module_7_production_ci/monitor.py # drift vs baseline
# Snapshot a real trace so it can be replayed (dry run by default)
python module_3_agent_evals/snapshot_trace.py --run-id <root-run-uuid>Each prints a link/name to open the experiment (or project) in LangSmith.
A note on the GitHub Actions workflows. Both ship as reference material with
workflow_dispatchas their only trigger — they never run on a PR, a push, or a schedule out of the box. Eval runs make real, paid model calls and need API keys, so a workshop repo that fired them automatically would bill anyone who forked it and fail for anyone who hadn't configured secrets. Each file documents exactly which triggers to uncomment to arm it for real. See module 4 and module 7.
hr_agent/ # the system under test (agent + tools + mock data + mocking + replay)
# --- Session 1: Foundations ---
module_1_fundamentals/ # concepts + first eval
module_2_single_turn/ # deterministic + LLM-judge + System 1 evaluators; which-evaluator guide
module_3_agent_evals/ # trajectory + tool evaluators
module_4_ci/ # pytest gate, aggregate gate, GitHub Actions
# --- Session 2: Evals for Production ---
module_5_online_evals/ # reference-free evals + scoring live traces
module_6_improving_evals/ # annotation queues + few-shot judge alignment
module_7_production_ci/ # scheduled drift monitor + baseline
.github/workflows/evals.yml # offline gate (reference; manual-only)
.github/workflows/online-evals.yml # online monitor (reference; manual-only)
config.py # model + LangSmith config, experiment metadata
fixtures/cassettes/ # recorded model responses (VCR) for fast local tests
Worth copying into your own repo — they're what keeps an eval suite usable once it grows past a handful of datasets.
hr-onboarding/policy-qa/intro # Module 1
hr-onboarding/policy-qa/v1 # Module 2
hr-onboarding/tool-selection/v1 # Module 3
hr-onboarding/tool-failures/v1 # Module 3 (mocked)
hr-onboarding/policy-qa/from-production # Module 6 (curated)
Slash-separated names sort and filter cleanly in the UI. Prose titles don't.
Every example gets a split. The scratch slice is where you tune prompts and
few-shot judges — tune against your gate and the gate stops measuring anything.
On the names. Splits are free-form strings; nothing in LangSmith enforces
any particular vocabulary. test/train is the conventional pair and it is
borrowed from model training, which is misleading here — no model is being
trained, and a PM asked to pick a "train" split has no idea what that means.
gate/scratch says what the two slices are for. Only the holdout
discipline matters; pick names your team reads correctly.
On the direction — this part is not cosmetic. Filtering to the gated
split looks equivalent to filtering out the scratch one, and it fails
open: LangSmith files every example with no explicit split under the implicit
base split, so each example a PM adds through the web UI is silently dropped
from the gate. Nothing breaks, no one is told, and coverage quietly shrinks.
Exclude scratch instead and a forgotten split becomes a false failure — loud
and fixable — rather than a test that never ran.
client.create_examples(..., splits=[e["split"] for e in EXAMPLES])
# Fail-open — an example with no split assigned is invisible here:
# data = list(client.list_examples(dataset_name=NAME, splits=["gate"]))
# Fail-safe — unassigned examples get gated:
all_examples = list(client.list_examples(dataset_name=NAME))
held_out = {e.id for e in client.list_examples(dataset_name=NAME, splits=["scratch"])}
data = [e for e in all_examples if e.id not in held_out]module_4_ci/ci_gate.py does this and prints the accounting every run
(total / gated / held out / unassigned), so drift shows up in the CI log.
--require-splits turns "someone forgot" into a hard failure.
Renaming splits on a dataset that already exists.
ensure_dataset()is idempotent — it won't relabel a dataset it finds. A workspace created before this rename still hastest/train, which is whyci_gate.HELD_OUT_SPLITSkeepstrainas a legacy alias: dropping it would silently start gating someone's scratch examples. To actually migrate:old = list(client.list_examples(dataset_name=NAME, splits=["train"])) ids = [e.id for e in old] client.update_dataset_splits(dataset_name=NAME, split_name="scratch", example_ids=ids) client.update_dataset_splits(dataset_name=NAME, split_name="train", example_ids=ids, remove=True)
Datasets get {owner, capability, module, source}; examples get
{difficulty, policy_topic, source}; experiments get commit, branch, model,
and author via config.experiment_metadata().
Gotcha: the
LANGSMITH_EXPERIMENTenv var is read only by the langsmith pytest plugin. It does not tag experiments created byclient.evaluate— those needmetadata=passed explicitly. That's whatexperiment_metadata()is for.
module_4_ci/conftest.py points LANGSMITH_TEST_CACHE at fixtures/cassettes/
so local test runs record and replay model calls. Measured on the module 4
suite: ~45 s cold → ~1.4 s fully cached.
It's disabled when $CI is set: a gate that replays yesterday's responses
can't detect that today's model regressed.
This caches the model. It is not the fix for stale data — that's caching the
tools' outputs so the agent is re-run against a frozen world. Different thing,
different reason, and the opposite default: replaying tool outputs is exactly
what you do want in CI, because it holds the world still while the agent (the
thing under test) varies. See Module 3. Cassettes are gitignored — no
credentials reach disk, but their filenames are keyed on the LangSmith dataset
UUID, so they'd never replay in another workspace. WORKSHOP_NO_CACHE=1 forces
real calls locally. See module 4 for the cached_hosts trap.
The module-per-lesson layout here is pedagogical. In a production repo, organize by role instead:
evals/
├── datasets/
│ ├── from_production.py # sample traces -> dataset (module_6/curate_dataset.py)
│ ├── synthetic_generation.py # generate test cases
│ └── schemas/ # pydantic models for inputs/outputs
├── evaluators/
│ ├── __init__.py # export all evaluators
│ ├── correctness.py # (module_2/deterministic_evals.py)
│ ├── tool_selection.py # (module_3/trajectory_evals.py, tool_evals.py)
│ ├── safety.py
│ └── llm_judges/ # (module_2/llm_judge_evals.py)
├── fixtures/
│ ├── tool_mocks/ # (hr_agent/mocking.py + module_3/mock_datasets.py)
│ ├── tool_replays/ # (hr_agent/replay.py + module_3/snapshot_trace.py)
│ └── cassettes/ # VCR recordings
├── suites/
│ ├── regression.py # (module_4/ci_gate.py build_suite + THRESHOLDS)
│ ├── capability_specific.py
│ └── nightly.py # comprehensive scheduled runs
└── conftest.py # (module_4/conftest.py)
The rule behind it: evaluators, dataset-creation scripts, orchestration, and mocks live in code (versioned, reviewed, testable). Dataset storage, experiment comparison, and human feedback live in LangSmith.
That rule is often misread as "never author an evaluator in the LangSmith UI." It isn't. Where an evaluator is written and where it executes are separate choices, and you don't have to trade one for the other:
| Offline experiments | Online evals on live traces | |
|---|---|---|
| Evaluator authored in code | evaluate(evaluators=[...]) — this repo |
push it to a project rule (below) |
| Evaluator authored in the UI | attach to a dataset | Tracing project → Rules → + New Rule |
A code-authored evaluator can be registered server-side as a project rule
(POST /runs/rules with a code_evaluators payload; the langsmith-evaluator
skill ships an upload_evaluators.py that does this with --project and
--sample-rate). So a single reviewed definition in your repo can drive both
the pre-merge gate and sampled scoring of production traffic — you do not have
to maintain the same logic in two places to get online evals. See
module 5.
Two caveats worth knowing before you commit to either direction:
- Uploaded code evaluators are sandboxed. They ship as a self-contained function and cannot import your repo's modules, so anything with real internal dependencies has to be inlined or kept UI-side. This is the honest argument for authoring in the UI.
- UI-authored LLM judges are unversioned config. Edit a judge prompt and every historical score silently changes meaning, which quietly breaks experiment-over-experiment comparison — the thing you keep experiments for. If you author judges in the UI, version their prompts deliberately (Prompt Hub, or an export job). That risk, not "code vs UI", is what this convention is actually protecting against.
The teams that get burned are the ones who end up with drifted duplicates — the same judge in a repo and in the UI, edited independently. One definition, either home, is the thing that matters.