How do you correctly use LLM as a judge?

Give it one narrow question and a written rule, and ask for pass or fail with a short reason. Use a different model from the one being judged. Check its marks against an expert's on real answers, and fix the rule until it catches the failures the expert found. Then run it on every change and re-check a fresh sample regularly.

What is the difference between LLM as a judge and human evaluation?

Human evaluation is a person marking answers. It is slow and costly, but it is the ground truth. An LLM judge is a model marking answers. It is fast and cheap, so it can mark every answer on every change. The two work together: people mark a sample, and that sample tells you how far to trust the judge.

How many answers do you need to calibrate an LLM judge?

Enough to include real failures, not just passes. If your sample has only a handful of wrong answers, you can't tell whether the judge catches them. Start with the real questions you already have. In our experience, real questions from real users are worth more than invented ones. Add more failing examples if your first sample has too few.

What does an LLM judge do?

It reads three things: the question, your system's answer and a written rule called a rubric. Then it returns a mark, usually pass or fail, with a short reason a person can check. When comparing two answers, it says which better follows the rubric. It doesn't decide anything on its own: its marks feed a test report or a review queue.

Can an LLM judge replace human review?

For volume, mostly yes. For the final say on costly decisions, no. A calibrated judge can mark every answer and sort the ones that need a look. A person should still mark a fresh sample regularly, decide the cases the judge is unsure about, and own any decision where one wrong answer is expensive.

Should the judge model be the same as the model being tested?

No. An example in Anthropic's evaluation guide notes it is generally best practice to evaluate with a different model from the one that produced the answer. Research names self-enhancement bias: a judge favouring answers from its own model. In our own tools, a judge model that wrote one of the answers being compared is not allowed to judge them.

LLM as a Judge: Calibration and Best Practices

Q: How good is an LLM as a judge?

Good enough to run on every change, once it is calibrated for your task. Research shows judges approximate human preferences. It also names their biases: towards an answer's position, towards longer answers, and towards their own model's answers. How good yours is depends on your rule and your data, so measure it against an expert's marks before you rely on it.

Suppose your AI test report says 94% of answers passed. It looks like good news. Then you notice who did the marking: another AI model.

Marking hundreds of answers by hand on every change is slow and costly. So teams let a second AI model do the marking. OpenAI's guide says that is cheaper and scales better than people marking by hand (human evaluation). But if nobody has checked the marker, you can't defend the number.

This post shows how to check it. You'll see what that AI marker, called an LLM judge, does well and where it slips. You'll get a step-by-step way to compare its marks with an expert's. And you'll see the guards we built into our own tools.

What is LLM as a judge?

LLM as a judge means using a second AI model (a large language model, or LLM) to mark another AI system's answers against a written rule. Usually it returns pass or fail with a short reason. It is fast, but it has biases. So you calibrate it against human evaluation before you trust it.

Put simply:

The judge marks the answers. A person marks the judge.

The written rule is called a rubric. "Pass if the answer gives the fee from the fund's fact sheet and links to it" is a rubric. The judge reads the question, the answer and the rubric, then decides.

Teams use judges for what code can't check: was the answer faithful to the source, was the tone right, did it refuse when it should have. Our guide to what AI evaluation is covers where a judge sits among the other kinds of checks. This post is about the step that is easiest to skip: calibration, which means proving the judge agrees with your expert before you rely on it.

How good is an LLM as a judge?

Good enough to be useful, and wrong in ways you can predict. An early paper on the method, by Zheng and colleagues, found it a "scalable and explainable way to approximate human preferences" (Zheng et al.). The same paper named the judge's biases.

OpenAI's evaluation guide lists the same problems. It calls them position bias and verbosity bias.

Bias	What it looks like	The guard
Position bias	Shown two answers, it prefers the first one (or the second)	Judge every pair twice, in both orders
Verbosity bias	It prefers the longer answer, even when it says no more	Say in the rubric that length isn't quality; pass or fail, not "which is better written"
Self-enhancement	It favours answers written by its own model	Never let a model judge its own model's work
Confident tone (a risk we design for)	A sure-sounding wrong answer beats a hedged right one	Ask it to check the answer against the source, not the tone

None of these make a judge useless. They mean the judge needs the same thing any new team member needs: someone checking its work before it works alone.

When should you use a code check, an LLM judge or human evaluation?

Pick the cheapest check that can actually tell right from wrong. Anthropic's guide to building evaluations ranks them the same way. Code checks are "fastest and most reliable". Human grading is "slow and expensive". LLM grading sits between: "Test to ensure reliability first, then scale."

The answer is exact. A fee, a date, a link that must be there, valid format. Use a code check. It is free and gives the same result every run.
The answer needs reading. Faithful to the source, right tone, refused when it should. Use an LLM judge, calibrated first.
A wrong answer is costly, or the case is new. Use a person. A judge can still sort the queue, but a person decides.

Question you're asking	Best check	Why
Does the figure match the database?	Code check	One right answer, so code is exact and free
Is there a source link?	Code check	A rule, not a judgement
Did it answer only from the document?	LLM judge	Needs reading, and there are too many answers for people
Is this reply to a customer's money question safe to send?	Human review; the judge sorts the queue	A miss is costly, so a person decides every one
Is this new kind of question handled well?	Person	Nobody has written the rubric yet

So LLM as a judge vs human evaluation is the wrong framing. You need both. People mark a sample. The sample tells you how far to trust the judge. The judge then marks everything else.

How do you calibrate an LLM judge against human evaluation?

Treat the judge like a new marker on probation. An expert marks a set of answers, the judge marks the same set, and you compare. OpenAI's guide puts it plainly: "Use human feedback to calibrate automated scoring."

1. An expert marks real answers

Take real questions and your system's real answers. The person who knows what right looks like marks each one pass or fail, with a reason. For a fund assistant, that is someone who knows the fund. In our experience, real questions beat invented ones. India Avenue's team, for example, supplied and marked 50 real questions for its fund assistant (as of 21 Oct 2026).

Make sure the set contains failures. A sample of nothing but passes can't tell you whether the judge catches anything.

2. Write the rubric from the expert's reasons

The expert's reasons are your rubric. If they failed three answers for quoting an old fee, the rubric says "fail if the fee differs from the current fact sheet". One question per judge works better than one judge checking five things.

3. The judge marks the same answers

Run the judge on the same answers. Don't show it the expert's marks. Keep some marked answers aside and don't tune the rubric on them. They are your final test.

4. Compare, and read every disagreement

Here is the trap. Say an expert marks 100 answers: 90 pass and 10 fail. A judge that passes everything agrees with the expert 90 times out of 100. That sounds fine. It caught none of the 10 failures.

Agreement is easy when most answers pass. Count the failures the judge caught.

So measure two things separately:

Failures caught. Of the answers the expert failed, how many did the judge fail? Engineers call this the true positive rate (with failure as the positive).
Passes kept. Of the answers the expert passed, how many did the judge pass? This is the true negative rate.

Then read each disagreement. Often the judge is wrong. Sometimes the expert's reason shows the rubric was vague. Sometimes the expert changes their mind.

5. Fix and run again, then keep checking

Fix the rubric, or fix your system's instructions if they were unclear. Run again. Write the bar down before you look, for example "catches at least 9 in 10 of the expert's failures". When the judge clears it, scale up. OpenAI's wording: once it "consistently agrees with human annotations, scale up".

Calibration isn't done once. Re-check a fresh sample regularly. Re-calibrate whenever you change the rubric or the judge model.

LLM as a judge best practices

These come from the vendor guides and from what we've built. Most guard against a bias in the table above. The rest keep the marks honest and repeatable.

Ask for pass or fail (or a choice between two), with a reason

OpenAI recommends "pairwise comparison or pass/fail for more reliability". A pass or fail also says plainly what went wrong, where a score of 6 out of 10 leaves you guessing. A written reason lets a person check the mark in seconds.

Use your own instructions as the rubric

When a judge checks an answer, ask whether it followed your instructions, not whether it is "good". Otherwise the judge swaps in its own taste.

Never let a model judge its own work

An example in Anthropic's guide notes that it's "Generally best practice to evaluate with a different model from the one used to generate the evaluated output". If you're comparing two models, the judge must be a third.

Judge pairs in both orders

When the judge picks the better of two answers, show them twice with the order swapped. If the verdict flips, that is position bias, not a finding.

Never cut the input to make it fit

If a long document or call is shortened to fit the judge, the judge marks an answer against text it never saw. No mark is better than a confident wrong one.

Decide on reasoning on purpose

Anthropic suggests a judge that reasons before it scores, because it judges better. We turn reasoning off in one tool, and set the randomness setting (temperature) to zero, so the marks repeat as closely as possible from run to run. Both are reasonable choices. Make yours deliberately, and calibrate with the setting you'll use.

How we run LLM judges in Avesta AgentOS and Metricsense

We use judges in two places. The first is testing an AI co-worker's answers when something changes. The second is comparing a new AI model with the current one.

Testing agents in Avesta AgentOS

In Avesta AgentOS, each test answer is marked one of two ways. A code check marks anything exact. Or an LLM judge marks it pass or fail on meaning, with a written reason. The judge can be a different model from the agent.

A run exports to a spreadsheet. The expert marks the same rows, next to a column for whether they agree with the judge. Our post on how to test a RAG chatbot (one that answers from your own documents) walks through that export.

Every agreeing row is evidence the judge can be trusted. Every disagreeing row is something to read.

Trying a new model in Metricsense

Metricsense is our call and agent analytics product. It reads calls and answers questions about them, such as whether a required statement was made. Metricsense can run a model trial: a new AI model is tested against the current one before anyone switches. The current model and the new one answer the same questions on the same calls, with the same instructions.

Where they agree, there is nothing to judge. Where they disagree, a judge gives a second opinion. We built these guards into it:

It is asked one narrow question. Which answer better follows the instructions for this call? Not which is better written. Longer isn't better, and more confident isn't better.
It never marks its own work. If the judge model wrote one of the answers in the trial, the run stops with a clear error.
It judges every disagreement twice, with the order swapped. Each time it also reports its confidence: high, medium or low. The three outcomes are in the diagram after this list.
It never reads a shortened call. If a transcript is too long for the judge, the judge gives no opinion.
It can say the instructions don't settle it. When both answers are fair readings, the problem is the instructions. That is a prompt to fix, not a model to pick.
It suggests. A person decides. Every disagreement is settled by a person. The judge's reason is always on screen, and its suggestion is drawn quieter than the person's decision. Once the person decides, the screen shows whether they agreed with the judge.

Cards diagram: one disagreement judged twice with the answers swapped, and three possible outcomes. 1 Same answer both times: a suggestion, with the lower of the two confidence levels. 2 Different answer each time: position bias, recorded as no opinion. 3 One order picks an answer, the other calls them equal: kept as a suggestion with low confidence. A person still decides.

Same answer both times. The judge's pick stands as a suggestion. It takes the lower of its two confidence levels, never the higher.
A different answer each time. That is position bias. It is recorded as no opinion, not as a finding.
One order picks an answer, the other calls them equal. The pick is kept, with low confidence.

That last guard is calibration built into the daily work. Every decision a person makes is also a mark on the judge.

The same caution applies to the two models' own agreement. If most of their agreements are the same easy answer, the screen warns that the match rate is weaker evidence than it looks. That is the 90-out-of-100 trap again.

One design choice that matters more than it looks

Our first design started the judge as soon as one call had answers from both models. That looks efficient. But it would have left the judge working on a partial set, and calls that finished later would never be judged. Nothing would fail. The judge's column would just be quietly incomplete.

So the judge starts only after both models have finished every call. And every disagreement it can't judge gets a row that says why: the transcript was too long, the limit on judged calls was reached, or the judge failed. A skipped mark should never look like a mark that was never asked for.

What is the best LLM for LLM as a judge?

There isn't one best judge model for every task. OpenAI's advice is to "use the most capable model to grade if you can". Then check whether it works for your rubric. Three rules matter more than the brand:

A different model from the one being judged. Research names this self-enhancement bias: judges favouring their own model's answers (Zheng et al.).
Recorded with every mark. In Metricsense, the judge model is stored on every judgement, so you can always see which model made a mark.
Re-calibrated when it changes. A new judge model shifts every mark. Run the calibration set again before you trust it.

The research is still moving. A survey of LLM-as-a-Judge research, last revised in October 2025, frames the open question as how to build judges that are reliable. Calibrating on your own data is how you answer it for your task.

When should you not use an LLM judge?

A judge isn't always the right tool. Skip it in these cases.

When code can check it. Exact numbers, dates, links and formats. A judge adds cost and a chance of error for no gain.
When nobody can say what right looks like. Without an expert's marks, you can't calibrate. An uncalibrated judge gives false confidence, which is worse than no test.
When each wrong answer is costly. Advice to a customer, a payment, a compliance call. Let the judge sort the queue, and let a person decide. For which actions need a person, see our guide to AI agent human approval.
When volume is low. If your system gives 30 answers a week, have a person read them.

And a judge is only one part of evaluation. Our post on why evaluation matters for AI agents covers the rest.

The bottom line

LLM as a judge is how you mark AI answers at the speed you ship changes. It is also a model with biases of its own. So treat it like a new marker. An expert checks its work on real answers. You count the failures it catches. You fix the rubric until it clears a bar you set in advance. Then let it mark the volume, and keep people on the decisions that matter.

For the wider picture of evaluation, read what AI evaluation is.

Who in your team checked the marker behind your last test report?

Want an evaluation setup your experts can trust, with judges calibrated against their own marks? We build testing, judges and human review into AI agents before they go live.

See how Safe AgentOps works

Get started: hello@avestalabs.ai