LLM evaluation training: evaluation that holds up
A hands-on workshop for engineering, data, and product teams on LLM evaluation: building a production-grade eval harness, putting error bars on LLM metrics, and eval gates that stop bad merges before they ship. Taught live online, on-site, or blended.
What it covers
The material comes from courses Bruno already teaches publicly, and from code we have published. A private session takes it and points it at your systems.
- Building a production-grade eval harness
- Putting error bars on LLM metrics
- Eval gates that stop bad merges before they ship
Taught in public
The public sessions on this topic. The private format is the same material, tailored to your team, your stack, and your data.
Upcoming
-
You Can't Ship What You Can't Measure — Building a Production-Grade LLM Eval Harness
ODSC AI NYC [In Person]
Design, build, and operate a real automated LLM evaluation harness end to end — from first principles to a regression suite you can drop into CI. Using MMLU as the system under test, layer deterministic answer-matching, LLM-as-judge rubrics, and human spot-checks, add statistical rigor with bootstrap CIs and paired significance tests, and build a CLI that compares two prompt/model configurations and flags significant regressions. Leave with working, reusable code.
-
Build a Production-Grade LLM Eval Harness
Maven Workshop, Live on Zoom, 4 hours, $300
In four hours, build a production eval harness from scratch — calibrated judges, error bars on every metric, paired tests that settle arguments, and a CI gate that blocks bad merges. You leave with a running Inspect-AI harness, the full repo, and a swap guide to run it on your own product data. Monday morning, it runs.
-
Jev vs Claude: A Measured Guide
Maven, Live cohort, 4 hours, $300
Learn Jev end to end, then prove task by task where it beats Claude or not. Build a paired bakeoff harness, run it against Claude Opus 5.5 on routing, scoring, judging and agent gating, and leave with the confidence intervals, the cost per million decisions, and an eleven-question decision guide.
Already taught
-
Maven, Free lightning lesson, 30 min, Zoom + recording
Pull 113 labeled yes/no answers, draw the reliability diagram for Jev and for Claude's self-reported confidence, compute ECE and Brier score in ten lines of numpy, and set auto / confirm / human bands from the cost of a false allow vs a false block.
-
Stop Bad Merges with an LLM Eval Gate
Maven, Live on Zoom, 30 min, Free
A bad prompt change reaches production the same way a good one does: nobody measured either. In this free lesson, learn to wire an eval suite into GitHub Actions so regressions die in the pull-request queue — not in front of users. The full setup fits in one YAML file. Vigilance does not scale. Policy does.
-
Put Error Bars on Your LLM Metrics
Maven, Live on Zoom, 30 min, Free
You ran the same eval twice and got 84.2, then 81.9. Which number goes in the report? Without error bars, every score is a coin flip dressed as a fact. In 30 minutes, learn to bootstrap a confidence interval on any metric — accuracy, cost, or latency — in 20 lines of Python. No distribution assumptions required.
-
Prove Your Prompt Change Actually Helped
Maven, Live on Zoom, 30 min, Free
You changed a prompt. The score went up. But did it actually improve — or did you get lucky? In this free 30-minute lesson, learn how to run a proper paired test on two prompt versions, how to tell a real improvement from noise, and how to pick a sample size that settles the argument. No more gut-feel comparisons.
Formats
-
Live online
Interactive remote sessions with exercises, live coding, and Q&A — the format Bruno has run for O’Reilly, Pearson, and Packt since 2019.
-
On-site or in-person
Deep dives tailored to your stack and your use-cases, working with your data where it can be used.
-
Blended
Pre-work, workshop, and follow-up office hours, so what was learned actually lands in your codebase.
The methods, written up
We publish the code behind the advice. The articles on this topic:
Prefer implementation help? See the Production Hardening engagement.
Questions about the training
- Who is it for?
- Engineering, data, and product teams.
- Is it online or in person?
- Either. Live online: interactive remote sessions with exercises, live coding, and Q&A. On-site: deep dives tailored to your stack and your use-cases. Blended: pre-work, workshop, and follow-up office hours.
- Can it use our stack and our data?
- Yes. Private sessions take the material Bruno teaches publicly and point it at your systems; on-site sessions work with your data where it can be used.
- What does the team do during the session?
- They build. No slideware tours: your team builds a RAG pipeline, an eval harness, or an agent during the session, on patterns they can carry straight into your codebase.
- How do we book?
- With a call: thirty minutes to pick the topics, the format, and the depth — and leave with a proposed outline.
Tell us what your team needs to ship.
Thirty minutes to pick the topics, the format, and the depth — and leave with a proposed outline.
Or email info@data4sci.com