Grade a reply yourselfTry it yourself
Take a seat on the panelGrade an AI reply to a patient message and see where the clinicians split. Or compare it with a nurse’s reply without knowing which one the AI wrote.
In one study, experts agreed with each
other ~75%73%72-75% of the time and with AI judges only as little as 65%64%64-68% of the time (Szymanski et al., 2024).
Two kinds of panel study
Grading see example to the rightsee example below
Experts score your AI's replies, and you see where they split.
Comparison
Experts judge AI and human replies without knowing which is which.
Theoretical Grading exampleInvented Grading exampleMade-up Grading exampleHypothetical Grading example: a chatbot answering patient messages after hours
For the past two weeks, my chest keeps feeling tight while going up the stairs, but it goes away after a few minutes. I'm only 52. Should I be worried?
A doctor should see you within 24 hours. I've booked a visit for tomorrow morning. If the tightness comes on at rest, call 911.
4 clinicians split 2-2 on this reply. You're the 5th.4 of the 5 clinicians split 2-2 on this reply. You take the urgent care seat.
Is a visit tomorrow the right level of care?
Split 2-2
You
Right level of care?
- Family yes
- Urgent care no
- TelehealthInternist yes
- ER no
- Cardiology no
Split 3-2
- Family medicineFamily “A visit tomorrow, with clear 911 triggers, is reasonable.”“A visit tomorrow, with clear 911 triggers, is reasonable for most patients like him.”
- Cardiology “He needs the ER now. Tomorrow is too late.”“New chest pain on exertion needs to be seen today.”
- Emergency medicineER “I’d send him in now. If a chatbot tells him to wait and something happens tonight, that’s on us.”
Invented Comparison example: the same patient message, answered by the AI and by a nurse
For the past two weeks, my chest keeps feeling tight while going up the stairs, but it goes away after a few minutes. I’m only 52. Should I be worried?
A doctor should see you within 24 hours. I’ve booked a visit for tomorrow morning. If the tightness comes on at rest, call 911.
Let’s get you seen by a doctor tomorrow morning, and I’ve booked the visit. If the tightness comes on while you’re resting, call 911 right away.
Chest tightness when you climb stairs needs to be checked today. Please go to the ER now. If it gets worse or comes on at rest, call 911.
Clinicians split 2-2 without knowing which reply was AI. You’re the 5th.
Which reply would you send?
- Family doctorFamily picked reply A
- TelehealthInternist picked reply A
- ER doctorER picked reply B
- CardiologistCardiology picked reply B
- You
Reply A was the AI. Reply B was a nurseReply A was a nurse. Reply B was the AI
How a panel study works
Grading
Experts score your AI's replies against the standards your field uses, and you see where they split.
Comparison
Experts judge AI and human replies side by side, without knowing which is which.
Pick the replies and the standards
Choose the AI replies to grade and the standards to consider.
Recruit the panel
5 experts from different specialties, like clinicians for a patient messaging assistant.
Experts grade every reply
Each expert grades alone and explains why before seeing the others’ grades. In a Comparison study, they don’t know which replies are AI.
Readout and decisions
Find where the panel agreed and split, and the decisions your team needs to make, each with an owner.
Two kinds of panel study
Grading
Experts score your AI's replies against the standards your field uses, and you see where they split.
Comparison
Experts judge AI and human replies side by side, without knowing which is which.
How a panel study works
Grading
Experts score your AI's replies against the standards your field uses, and you see where they split.
Comparison
Experts judge AI and human replies side by side, without knowing which is which.
Pick the replies and the standards
Real replies from after launch where possible, and the standards your field uses.
Recruit the panel
5 experts from different specialties.
Experts grade every reply
Each grades alone and says why.
Readout and decisions
Each decision your team needs to make gets an owner.
What the readout shows
When a buyer asks how you validated your AI, the readout is what you show them.
See one example page of a Grading readout: 5 clinicians, one AI reply, and the standards they agreed and split on.
Comparison readouts highlight where AI underperforms and outperforms humans while providing training to improve AI answersComparison readouts show, reply by reply and overall, where your AI does better or worse than a person, and which gaps to fix first
What is different about this method
- Human agreement comes first Before checking an AI judge against people,Before checking your AI evals against people, the panel checks whether the subject-matter experts agree.
- Every disagreement gets a cause and an owner Each split is traced to the rubric, a rater, or a disputed standard only your team can set.Each split is traced to the rubric, a rater, or a disputed standard only your team can set, like how much legal risk a reply should take on.
- Contested standards go to your team to decide They come back as named decisions and are never averaged away.
Builds on Fleisig et al. (2024) and CrowdTruth (Aroyo & Welty, 2015), who showed that rater disagreement is useful information.
Panel readoutSample page, invented example
The panel agreed on the wording and split on urgency
5 clinicians graded 30 AI replies. This page shows one.
- Right level of care2 of 5 yes
- Clear signs for when to call 9115 of 5 yes
- Plain enough for a patient to act on4 of 5 yes
Marks in order: family, urgent care, telehealthinternist, ER, cardiology
They split over a contested standard
- Visit within 24 hoursClinical judgment: within 24hrs2 of 5. A visit tomorrow, with clear 911 triggers, is enough.
- Emergency room nowSafest to defend: ER now3 of 5. New chest pain on exertion is unstable until ruled out.3 of 5. One cited clinical risk, and two cited liability if the patient waits.
Which standard should the assistant meet for chest symptoms on exertion?Should the assistant give the answer most clinicians would give, or the one that’s safest to defend?
Decision owner: medical directorDecision owners: medical director and head of legal
Panel studies find what to fix. They don't certify an AI as safe.
30 minutes, free
AI experience
- Evaluating conversational AI since 2019Starting at Memora Health
- Blinded ratings of human and AI writing doing the same tasksDesigned blinded study of human & AI writing on the same clinical tasksAt Kooth Digital Health
- Designed and evaluated a clinical case note summary LLMAt Kooth Digital Health
- Disagreement is the SignalQuant UX Con 2026 talk
- Member of the Global AI Ethics CommitteeAt Kooth Digital Health
- Tested a postpartum chatbot before it reached patientsBMJ Innovations paper
- Designed a clinical case note summary LLM and ran blinded human vs AI ratingsAt Kooth Digital Health
Why run a panel study
- Your grader passes every replyYour AI evals pass every replyBut no expert has checked the graderBut no expert has checked what they scoreGrading studyRubric test
- Experts disagree on the hard callsSo nobody knows what the grader should scoreSo nobody knows what your evals should scoreRubric test
- One expert signed offOnly one expert has signed offAnd their specialty decides what counts as safeBut another specialist might call the same replies unsafeGrading study
- Replacing human replies with AIReplacing human work with AINo one has compared the two blindBut nobody knows if the AI is as good as a personComparison study
Most teams check their AI with an automated grader or a single expert. Both assume experts agree on what a good answer is. Often they don’t, and the grader learns whichever standard it was given. A panel shows where your experts split, so your team decides which standard the AI should meet.
What the research shows
In one study, experts agreed with each other ~75%73%72-75% of the time and with AI judges only as little as 65%64%64-68% of the timeIn one study, human experts agreed with each other 73% of the time, and with AI evalsLLM judges only ~66% of the time (Szymanski et al., 2024).
Who takes the blame when your AI is wrong?Don’t make one expert the accountability sinkA reviewer who can only sign off can’t catch the error
Anthropic and OpenAI now require a qualified person to review AI advice in fields like medicine, law, housing, and employment. Under Anthropic’s policy, that person stays responsible for what the AI delivers.
If they sign off on hundreds of replies a day, with no time to read them and no authority to change them, they’re there to take the blame when the AI is wrong. And when the AI is usually right, people stop checking. Researchers call this automation bias.
A panel study shows your reviewer where experts agree and where they split, so the hard calls get decided on purpose, by a named owner, before launch.
An accountability sink is a person who takes the blame for a decision they had no real power to change.
- The labs now require a human reviewer Anthropic and OpenAI require a qualified person to review AI advice in fields like medicine, law, housing, and employment. Under Anthropic’s policy, that person stays responsible.
- Sign-off with no time or authority makes the reviewer the sink With hundreds of replies a day and no power to change them, they can’t catch the error or fix it after the fact.
- A reliable AI makes reviewers stop checking When the AI is usually right, people stop looking closely. Researchers call this automation bias.
- A panel study shows them where to look It shows which replies experts agree on and which calls they split on, so each split gets a written decision and a named owner before launch.
An accountability sink is a person who takes the blame for a decision they had no real power to change.
“The qualified person remains responsible for the accuracy and appropriateness of what is delivered or decided.”
OpenAI also requires human review for high-stakes use cases
- Sign-off with no time or authority makes the reviewer the sink With hundreds of replies a day and no power to change them, they can’t catch the error or fix it after the fact
- A reliable AI makes reviewers stop checking When the AI is usually right, people stop looking closely. Researchers call this automation bias
- A panel checks whether experts agree first Before your AI evals are checked against people, 5 experts show which calls they agree on and which they split on
- Each split goes to your team with an owner It’s traced to the rubric, a rater, or a standard only your team can set, and it’s never averaged away
Builds on Fleisig et al. (2024) and CrowdTruth (Aroyo & Welty, 2015), who showed that rater disagreement is useful information
- Sign-off with no time or authority makes the reviewer the sink With hundreds of replies a day and no power to change them, they can’t catch the error or fix it after the fact.
- A reliable AI makes reviewers stop checking thoroughly When the AI is usually right, people stop looking closely. Researchers call this automation bias.
- A panel study shows them where to look Find which replies experts agree on and which calls they split on, so each split gets a written decision and a named owner.
Start with a rubric test
Find out which criteria a grader can be built on, before you run a study.A rubric is the checklist your team or an AI grader uses to score replies.A rubric is the checklist your team or your AI evals use to score replies. Two experts from your field grade the same 25 replies with it, so you know which criteria are ready for a graderyour evals and which need a rewrite first. Already run AI evals? The experts grade the same replies, so you see where your evals can be trusted.Building an AI grader or LLM judge? It’s only as good as your rubric, the checklist it scores replies against. Two experts from your field grade the same 25 replies with it, and you see which criteria they agree on before you build.
Find out which criteria in your AI eval rubric are ready to use
A rubric is the checklist your team or your AI evals use to score replies. No rubric yet? Share a product spec to get started on a first draft.
What you keep after a rubric test
- Which criteria are ready, and rewrites of the rest
- Where your AI evals and the experts split
- Open questions for your team, each with an owner
- 2 experts from your field
- 25 of your AI’s replies
- About 2 weeks
- From $3,000, fixed, all infixed before we startfixed price
Book a Grading study within 60 days, and the full $3,000fee counts toward it.
30 minutes, free
You provide
Your rubric and 25 de-identified AI replies. No rubric yet? Send a product spec for a first draft. Already run AI evals? Send their scores on the same replies.
Two experts grade
Each grades every reply alone, in one pass, and flags any criterion they can’t answer with the rubric as written.
You get a rubric you can build a graderyour evals on
- Criteria that are ready to use
- Where your AI evals and the experts split
- Rewrites of the unclear ones
- Open questions for your team, each with an owner
- Failing replies from your AI
- All of it is yours to keep
You send
Your rubric and 25 de-identified replies
Two experts grade
Each alone, in one pass
You keep
- Which criteria are ready, and rewrites of the rest
- Where your AI evals and the experts split
- Open questions for your team, each with an owner
Know a team shipping AI in high-risk domains like healthcare, law, HR, & housing?
Send them this page
Evaluation FAQs
Why does it matter if experts disagree?
Your AI gives one answer to everyone who asks the same question. Where experts split, someone has to choose that answer. A panel shows where those choices are, so your team makes them on purpose before launch.
Why does it matter if experts disagree?
Your AI gives one answer to everyone who asks the same question. Where experts split, someone has to choose that answer. A panel shows where those choices are, so your team makes them on purpose before launch.
Isn't one expert enough?
For many products, yes. In medicine, law, housing, and HR, experts in the same role often disagree. A panel shows which criteria one expert can score alone, and which ones your team has to decide first.
Isn't one expert enough?
For many products, yes. Common eval advice is to name one expert as the judge and add more when the field calls for it. In medicine, law, housing, and HR, experts in the same role often disagree. A panel shows which criteria one expert can score on their own, and which standards your team has to settle first.
Why five experts and not one?
With one expert, your grader learns that person's threshold, and nobody on your team chose it. Five experts show which criteria one person can score on their own, and where your team has to pick a standard. After that, one expert can run the rubric.
Can you test how people react to our AI’s replies?
Yes. People who would use your AI try it during research like usability tests, so you see how they read its replies and what they do next.
Don’t the AI labs already require human review?
Anthropic and OpenAI both require a qualified person to review AI advice in fields like medicine, law, housing, and employment. Neither says how that reviewer should decide what a good answer is, and in these fields experts often disagree. A panel shows your reviewer which calls are contested.
How is this different from automated evals?
Automated graders score each reply against one standard. A panel shows where experts in your field disagree on that standard, which is where automated graders struggle most.
How is this different from automated evalsAI evals or marketplace graders?
AI evals score each reply against one standard. Marketplaces supply graders at scale, mostly to AI labs. A panel study matches experts to your field, has them grade against your rubric, and shows where they disagree and who on your team decides.
Don’t the AI labs already require human review?
Anthropic and OpenAI both require a qualified person to review AI advice in fields like medicine, law, housing, and employment. Neither says how that reviewer should decide what a good answer is, and in these fields experts often disagree. A panel shows your reviewer which calls are contested.
Who are the experts, and where do they come from?
Practitioners in your field, like clinicians from different specialties for a patient messaging assistant. They come from professional associations, research recruiting platforms, and expert marketplaces. Each expert’s license or role is checked before they grade, and the readout lists their roles.
How is this different from hiring graders from a marketplace?
Marketplaces supply graders at scale, mostly to labs training models. A panel study is built around one product. Experts are matched to where your AI’s replies land, they grade against your rubric, and the readout shows where they split and who on your team decides. Some panelists may come through a marketplace, and the study works the same way.
Won’t experts just pick the safest answer?
Sometimes. Each expert says why they chose an answer, so you can see whether they disagree about liability or about the facts. Your team, often with your head of legal, decides which standard your AI meets.
Which fields do you work in?
Legal, medical, finance, housing, HR, and education.Healthcare most often, since that’s where my AI work has been. Also law, HR, housing, finance, and education.Healthcare, from patient messaging to clinical notes and triage. Product & UX work covers any field.
Can we say Desire Path reviewed our AI?
You can say Desire Path ran a study of your AI and share what it found. It isn’t a certification or an endorsement. Any marketing, sales, or regulatory claim that names Desire Path gets agreed in writing first.You can say Desire Path ran a study of your AI and share what it found. A study finds what to fix and which standards your team still has to settle. It isn’t a certification or an endorsement, and any marketing, sales, or regulatory claim that names Desire Path gets agreed in writing first.
What can we tell a buyer who asks how we validated our AI?Can we mention these studies to buyers?
That outside experts in your field graded your AI’s replies, where they split, and what your team decided. You can share the readout. A study finds what to fix and which standards your team still has to settle. It isn’t a certification or an endorsement, and any marketing, sales, or regulatory claim that names Desire Path gets agreed in writing first.Yes, but a panel study isn’t a certification or an endorsement. You can say outside experts in your field graded your AI’s replies, and share what your team decided. Any marketing, sales, or regulatory claim that names Desire Path must be agreed to in writing first.
What can we tell a buyer who asks how we validated our AI?
That outside experts in your field graded your AI’s replies, where they split, and what your team decided. You can share the readout. A study finds what to fix and which standards your team still has to settle. It isn’t a certification or an endorsement, and any marketing, sales, or regulatory claim that names Desire Path gets agreed in writing first.
How do you keep our information secure?
Everything you share is confidential by default. I'm happy to sign your NDA, or a BAA for health data, and I only access what a project needs.
Does a panel study certify our AI as safe?
No. A panel study finds what to fix and which standards your team still has to settle. It doesn't certify an AI as safe.
Let's connect
- 30 minutes, free
- You tell me what your AI does and who reads its replies
- We pick the first step, a rubric test or a panel study
- Within 48 hours, you get a scope & approach with a start date
Or email karl@desirepathux.com
After the call, you get a written scope & approach with timelines, options, & prices within 48 hours.
- Intro call30 minutes, free
- Scope & approachTimelines, options, & prices within 48 hours
- KickoffQuestions, first impressions, & quick wins