Your LLM Evals Are Measuring Style, Not Correctness

Q: How accurate is LLM-as-a-judge compared to human raters?

In the MT-Bench study that popularized the technique (Zheng et al., NeurIPS 2023, arXiv:2306.05685), GPT-4 pairwise judging agreed with human raters 85 percent of the time when tied votes were excluded. The number that gives it meaning is on the next line of the same table: humans agreed with each other 81 percent of the time on the same data. So a strong judge matches humans roughly as well as humans match each other. That is a comparative claim, not an absolute one. It does not mean the judge is 85 percent correct — it means the judge reproduces human preferences, including the parts of human preference that have nothing to do with whether the answer is right.

Q: Do LLM judges prefer longer answers?

Strongly, and it is the single largest style effect measured. In the Length-Controlled AlpacaEval work (Dubois et al., arXiv:2404.04475), changing only a verbosity instruction — same model, same underlying capability — moved a model's win rate from 22.9 percent to 64.3 percent. That is a 41-point swing purchased with nothing but word count. LMSYS ran a separate Bradley-Terry regression on Chatbot Arena votes and found length carried a coefficient of 0.249 while markdown lists, headers, and bold sat at 0.031, 0.024, and 0.019 respectively. Their conclusion was that length dominates and other formatting effects are second order.

Q: Do LLM judges favor their own outputs?

This is more contested than the folklore suggests. The paper usually cited for self-enhancement bias, Zheng et al., explicitly declines to claim it — the authors state that due to limited data and small differences, their study cannot determine whether models exhibit self-enhancement bias. Panickssery et al. (arXiv:2404.13076) did establish something sharper: a linear correlation between a model's ability to recognize its own text and the strength of its self-preference, demonstrated causally through fine-tuning. But a 2025 challenge (arXiv:2504.03846) argues much apparent self-preference is legitimate, because better generators tend to be better judges. The honest summary is that self-preference is partly earned quality and partly recognition-driven bias, and the harmful portion concentrates where the judge is already wrong.

Q: Is pairwise comparison better than absolute scoring for LLM evals?

It depends on what you are defending against, and the sources genuinely disagree because they measure different things. Zheng et al. found pairwise judging tracked human preference slightly better than single-answer grading. But a COLM 2025 paper (arXiv:2504.14716) found pairwise preferences flipped about 35 percent of the time under distractor manipulation versus 9 percent for absolute scores. Pairwise wins on fidelity to human preference on clean data; absolute scoring wins on robustness when something is actively gaming the judge. If your generator optimizes against your judge, the adversarial condition is your normal condition, and that changes the answer.

Q: What is the most effective fix for an unreliable LLM judge?

Giving it a reference answer, by a wide margin, in the one setting where it was cleanly measured. Zheng et al. found the default judge prompt failed on 14 of 20 math questions — 70 percent — and reference-guided grading cut that to 3 of 20, or 15 percent. The interesting part is why it failed: GPT-4 could solve those problems on its own but was misled once it saw the candidate answers. Judging is a different task from solving, and a judge with no ground truth to anchor on is doing free-form preference matching whether you intended that or not.

Q: Why is evaluating AI agents harder than evaluating chatbot responses?

Because the thing that fails is a trajectory, not a string, and you usually cannot tell where it broke. On the Who&When dataset (arXiv:2505.00212), built from failure logs of 127 LLM multi-agent systems, the best method identified which agent was responsible for a failure 53.5 percent of the time but pinpointed the failing step only 14.2 percent of the time. The authors note that even frontier reasoning models fail to reach practical usability at this. So you can often see that a run failed and roughly who owned it, while remaining nearly blind to when it went wrong — which is exactly the information a fix requires.

Q: Is agent tracing standardized with OpenTelemetry yet?

Not in any sense you should rely on. The OpenTelemetry GenAI semantic conventions moved to their own repository, and as of July 2026 the agent spans specification carries a Status: Development banner with no releases published and the schema URL section still marked TODO. Attributes like gen_ai.operation.name, gen_ai.agent.name, and gen_ai.request.model are all Development stability. The notable exception is error.type, which is Stable — but only because it is inherited from core OpenTelemetry rather than being GenAI-specific. Blog posts claiming these conventions are settled contradict the specification's own banner.

Q: Are agent benchmarks like SWE-bench trustworthy?

Treat them as a signal with a known contamination problem, not as ground truth. The SWE-Bench Illusion paper (arXiv:2506.12286) ran a memorization probe: give models only an issue description and a repo name — no repo structure, no code — and ask for the buggy file path. Models hit up to 76 percent accuracy on SWE-Bench Verified versus up to 53 percent on repositories outside the benchmark. Function reproduction showed up to 35 percent consecutive 5-gram overlap on SWE-Bench versus up to 18 percent elsewhere. The authors are appropriately careful that separating memorization from genuine reasoning is hard and they rely on cross-benchmark comparison as a proxy. But a model that can name the buggy file having never seen the code has told you something about the benchmark.

Your judge gives the new prompt an 8.2. The old one scored 7.4. You ship it. Two weeks later support is fielding complaints about answers that are confidently, fluently wrong — and when you pull those exact conversations back through your eval suite, the judge scores them 8-point-something too.

Nothing is broken. The judge is doing precisely what you built it to do. The problem is that you believed it was grading correctness, and it was grading something else.

This is not a story about a bad prompt. It is about a measurement instrument whose failure modes are well documented in the literature and almost never reflected in the eval suites people actually run. Below: what the evidence says LLM judges reward, which fixes have numbers behind them, which are folklore, and why agent evaluation is a harder problem than the single-turn version people are still struggling with.

The 85% everybody quotes

The technique got its credibility from one paper: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685, NeurIPS 2023). The number that escaped into the discourse is that GPT-4 agrees with human raters over 80% of the time.

That number is real. It is also almost always cited without the two conditions that determine what it means.

Condition one: ties were dropped. The paper reports two setups. With tied votes included, GPT-4 pairwise judging agreed with humans 66% of the time. Excluding ties, 85%. Same judge, same data, nineteen points apart. The random baseline moves too — 33% versus 50% — so neither number is wrong, but quoting 85% without “non-tie votes only” is quoting a different experiment than the one you think.

Condition two — and this is the one that matters:

Comparison (ties excluded)	Agreement
GPT-4 pairwise vs. Human	85%
GPT-4 single-answer vs. Human	84–85%
Human vs. Human	81–82%

Humans agree with each other 81% of the time. The judge is not scoring 85% on a test with a 100% ceiling. It is matching human preference about as well as humans match each other — which is the strongest honest version of the claim, and a completely different claim from “85% accurate.”

One correction worth internalizing, because it circulates: on Chatbot Arena data the paper reports GPT-4 vs. human at 87%, and GPT-4 vs. itself in a different configuration at 95%. That 95% is one LLM agreeing with another LLM. It shows up in summaries as human agreement. It is not.

So: judges track human preference faithfully. The uncomfortable question nobody asks next is what human preference actually contains.

What judges are actually rewarding

Here is the finding that should reorganize how you think about your eval suite.

The SOS-Bench work (arXiv:2409.15268) injected specific defects into responses and measured how much judges penalized each one:

Injected defect          Penalty
─────────────────────────────────
Sarcasm                    −96%
Conciseness                −63%
Wrong information          −13%
Blandness                   −8%

Read that column twice. Making an answer sarcastic is punished seven times harder than making it factually wrong. Making it shorter is punished nearly five times harder than making it incorrect. The authors’ summary is that judges are highly critical of unconventional stylistic changes and fairly lenient on major factual errors.

Now put that next to the agreement numbers and the contradiction dissolves. The judge matches human preference at near-human rates and barely penalizes false statements. Both are true, because human preference — as captured by fast pairwise votes on fluent text — is substantially about style. The judge did not invent the bias. It inherited it, faithfully, from the thing you asked it to imitate.

Length is the whole ballgame

If you fix one thing, fix this one.

Dubois et al. (arXiv:2404.04475) ran the cleanest experiment on this. Hold the model constant. Hold its actual capability constant. Change only an instruction telling it to be more verbose. Win rate moved from 22.9% to 64.3%.

A 41-point swing, bought with word count, on identical underlying ability. If a model in your leaderboard gained 41 points you would call it a breakthrough. Here it is a formatting instruction.

Their length-control correction compressed that same model’s rate from 41.9% to 51.6% and — the part that proves it was signal, not noise — pushed correlation with Chatbot Arena from 0.94 to 0.98 Spearman. Removing length bias made the metric more predictive of the thing it was trying to predict.

LMSYS reached the same place from a different direction (style control on Arena votes), fitting Bradley-Terry coefficients per style feature:

length          0.249   ████████████████████████
markdown list   0.031   ███
header          0.024   ██
bold            0.019   ██

Their conclusion, in their words: length was the dominant style factor and all other markdown effects are second order. When they controlled for style, rankings moved hard — GPT-4o-mini fell from 6th to 11th, Grok-2-mini from 6th to 18th, while Claude 3.5 Sonnet rose from 6th to 4th and Claude 3 Opus from 16th to 10th.

Two independent teams, different methods, same answer. This is as settled as anything in the field gets.

The engineering consequence: if your eval suite has ever rewarded a prompt change that made outputs longer, you cannot distinguish that from a real improvement without controlling for length. Not “probably should” — cannot. The effect size is larger than the improvements you are trying to detect.

Position bias fails exactly when you need it

Order matters. Swap answer A and answer B, and judges change their minds. Zheng et al. measured consistency under swap with the default prompt:

Judge	Consistency under swap
GPT-4	65.0%
GPT-3.5	46.2%
Claude-v1	23.8%

The best judge tested contradicts itself on a third of comparisons based purely on presentation order.

An independent replication (arXiv:2406.07791) found the mechanism, and it is nastier than the raw rate suggests: position consistency scales with the quality gap between the two answers. When one answer is clearly better, order barely matters. When the two are close, order dominates.

Which means position bias is worst precisely in the regime you built the eval for. Nobody needs a judge to tell them a great answer beats a broken one. You need it for “is variant B slightly better than variant A” — and that is the exact condition where the judge is closest to flipping a coin.

The self-preference story is not what you think

Ask an engineer why they use a different model as judge than as generator and you will hear about self-preference bias, stated as settled fact.

It is not settled, and the citation trail is a good lesson in how folklore forms.

The paper everyone points to is Zheng et al. It does report GPT-4 favoring itself by about 10 percentage points and Claude-v1 by about 25. It then says, verbatim:

Due to limited data and small differences, our study cannot determine whether the models exhibit a self-enhancement bias.

The most-cited source for self-enhancement bias explicitly declines to establish it.

The real evidence came later and is more specific. Panickssery et al. (arXiv:2404.13076) showed a linear correlation between a model’s ability to recognize its own text and the strength of its self-preference — and established it causally, by fine-tuning self-recognition and watching preference track it.

Then a 2025 paper (arXiv:2504.03846) pushed back from the other side: much self-preference is legitimate, because better generators are genuinely better judges. They measure task accuracy and judge accuracy correlating at r = 0.795–0.899 and split the phenomenon into legitimate self-preference versus harmful self-preference. For one model on MATH500: 96.57% of its self-preference was legitimate — but where it was wrong, 86% of the residue was harmful.

These findings are compatible, and the synthesis is more useful than either headline. Self-preference is partly earned quality and partly recognition-driven bias. The harmful part concentrates exactly where the judge is already mistaken — so it does not spread your error around, it amplifies error where error already lives.

Use a separate judge model. But use it because of the recognition mechanism, not because a 2023 paper proved something it specifically said it could not prove.

What actually works

Sorted by strength of evidence, not by how often you see it in blog posts.

Give the judge a reference answer

The strongest single mitigation with a clean number attached. Zheng et al. found the default prompt failed on 14 of 20 math questions (70%). Reference-guided grading cut it to 3 of 20 (15%).

The diagnosis matters more than the fix. GPT-4 could solve those problems standalone. It was misled once it saw candidate answers. Solving and judging are different tasks, and a judge with nothing to anchor on drifts into free-form preference matching — which, per the section above, means style.

If you have ground truth, put it in the prompt. This is cheap and most suites skip it.

Swap positions and treat disagreement as data

def judge_both_orders(judge, question, a, b):
    fwd = judge(question, first=a, second=b)
    rev = judge(question, first=b, second=a)
    if fwd == "first" and rev == "second":
        return "a"
    if fwd == "second" and rev == "first":
        return "b"
    return "inconsistent"   # do not silently coerce this to a tie

Note what this does and does not do. It detects order sensitivity; it does not remove it. The honest read of the 65% consistency figure is that a third of your comparisons return inconsistent — and that is the finding, not an inconvenience to average away. A suite that reports “38% of pairwise calls were order-inconsistent” is telling you something true. One that silently resolves them to ties is hiding its own error bar.

Correct the metric, not the judge

Both length results above work by fixing the aggregate, not by asking the judge nicely to ignore length. Regress out style features after the fact. Dubois’s Spearman going 0.94 → 0.98 is the receipt.

This generalizes into a principle worth stealing: post-hoc statistical correction of a biased instrument beats prompt-engineering the bias away. You cannot instruct a model out of a preference baked into its preference training. You can subtract it downstream.

Use a jury of cheap judges

Cohere’s Panel of LLM Evaluators (arXiv:2404.18796) replaced one large judge with three small ones from different families. Cohen’s κ against human labels:

Dataset	Panel (3 small)	Single large judge
Natural Questions	0.763	0.627
TriviaQA	0.906	0.841
HotpotQA	0.867	0.830
Chatbot Arena (Pearson)	0.917	0.817

Better agreement, roughly 7–8× cheaper. The mechanism is intuitive — judges from different families have different biases, and disagreeing biases partially cancel, while one judge’s bias is systematic and cancels with nothing.

Read the authors’ own limitation before adopting: they tested only three evaluator settings with a limited number of panel compositions, and they did not test math or reasoning. This is a strong result in a narrow frame.

Chain-of-thought, in perspective

G-Eval (arXiv:2303.16634, EMNLP 2023) is the standard citation for reasoning-then-scoring judges. It reports Spearman 0.514 with human judgments on summarization, which was state of the art.

Sit with that. 0.514 is a weak-to-moderate correlation, and it was the best available. Keep CoT — it helps and it makes judgments auditable. Just do not narrate it as a solved problem; the number says otherwise.

Where practitioner advice outruns evidence

Binary pass/fail over 1–5 Likert scales, expert critiques as few-shot anchors, decomposing rubrics into single-criterion judges — the most thoughtful case for these is Hamel Husain’s, drawn from 30+ implementations. His warning that raw agreement misleads under class imbalance (measure precision and recall separately) is correct and underrated.

But by its own framing this is experience-based assertion, not comparative study. And it is not uncontested: at least one 2026 preprint reports 0–5 grading maximizing human alignment, cutting directly against binary. Weak, single-source, unreplicated — which is my point. Label it opinion, run it as an experiment on your data, do not inherit it as fact.

My own read, clearly labeled as opinion: binary decisions are worth it less for judge accuracy than for organizational reasons — you cannot argue about whether 6 means the same thing to two annotators when the only options are pass and fail. That is a real benefit. It is just not the benefit usually claimed.

The ceiling nobody mentions

Every mitigation above lives under a formal constraint that deserves more attention than it gets.

Dorner, Nastl, and Hardt (arXiv:2410.13341, ICLR 2025):

when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half.

Empirically, they find real savings are more modest than even that bound.

The implication is sharp and it is where most eval strategies quietly break. If you are evaluating a frontier model, your judge is not more accurate than it — there is nothing more accurate available. In that regime no prompt trick, no jury, no rubric buys you out of collecting human labels. The best any debiasing method can do is halve them, and in practice less.

LLM judges are a way to stretch a ground-truth budget. They are not a way to eliminate one. Any eval roadmap whose end state is “no humans in the loop” is arguing with a proof.

Agents: the same problems, worse

Everything above concerns judging a string. Agents make it structurally harder, and the honest summary is that the tooling is not there yet.

You cannot find where it broke

This is the number that reframed the problem for me. The Who&When dataset (arXiv:2505.00212, ICML 2025 Spotlight) collected failure logs from 127 LLM multi-agent systems and asked methods to attribute failures. Results:

Identify which AGENT caused the failure    53.5%
Identify which STEP caused the failure     14.2%

The authors note that even frontier reasoning models fail to achieve practical usability here.

So: you can usually tell a run failed. You can coin-flip which agent owned it. You are effectively blind to when it went wrong — and “when” is the only thing a fix needs. Debugging agents today is closer to reading a crash with no stack trace than to reading a failing test.

The compounding math, done honestly

The p^n argument gets thrown around loosely, so here is the rigorous version — and the reason it is more interesting than the slogan.

Sinha et al., The Illusion of Diminishing Returns (arXiv:2509.09677), state their model as an explicit assumption rather than a finding:

we make two simplifying assumptions similar to LeCun (2023). First, we assume a model’s step accuracy remains constant over the task. Second, we assume a model does not self-correct, meaning any single error leads to task failure.

That gives P(success, H) = p^H, and their Proposition 1: the horizon at which success drops to 50% is H₀.₅(p) = −ln(2)/ln(p).

The consequence is genuinely counterintuitive: horizon length grows hyperbolically in step accuracy. Above a high threshold, tiny step-accuracy gains buy enormous horizon gains. Which is a decent argument that step-accuracy improvements looking boring on single-turn benchmarks can be the whole game for agents.

Now the part that makes this worth citing over any blog rendition: the same paper empirically breaks both of its own assumptions.

Self-conditioning. Models become more likely to make mistakes when the context contains their errors from prior turns. They demonstrate it by injecting errors into chat history at controlled rates — accuracy at turn 100 degrades further as injection rises. Their hypothesis is mechanical: a model trained to predict the likely next token, conditioned on its own error-riddled history, finds further errors likely.
Scale does not save you. Unlike long-context handling, they report scaling model size does not mitigate self-conditioning.

Take this seriously and the naive model is not merely “too pessimistic.” It is wrong in both directions at once. Self-conditioning means errors are positively correlated, so real degradation can be worse than i.i.d. p^n. Retries and self-correction push the other way, so it can be better. Which force wins is a property of your specific system, and p^n cannot tell you — it can only tell you the shape of the cliff.

The deployment-facing consequence shows up in τ-bench (arXiv:2406.12045), whose authors report state-of-the-art agents succeeding on under 50% of tasks and, more damning, being “quite inconsistent (pass^8 <25% in retail)” — pass^8 meaning all 8 independent trials succeed.

Note the caret. pass^k (all k succeed) is not pass@k (at least one succeeds). They get swapped constantly, including in otherwise careful summaries, and they mean opposite things. For anything customer-facing, pass^k is the metric you actually care about, because a workflow that works 3 times out of 8 is not a feature.

Trajectory vs. outcome

The survey literature (arXiv:2503.16416, ACL Findings) frames the choice cleanly:

Approach	What you get	What it costs
Final outcome	Cheap, automatable, unambiguous	”cannot assess intermediate decisions, execution efficiency, or failure causes”
Trajectory, reference-based	Precision, reproducibility	Requires a predefined expected path
Trajectory, reference-free	Flexibility (LLM judges the path)	“greater flexibility at the cost of reliability” — and now you have all the judge problems from the first half of this article, applied to sequences

The survey is blunt that static environments “fail to capture the cascading effects of errors: incorrect actions have no downstream consequences, missing compounding failures” — and that binary outcome metrics are insufficient to understand intermediate progress.

There is no free option here. Outcome eval is reliable and uninformative. Trajectory eval is informative and unreliable. Most teams should run outcome eval as the gate and trajectory eval as the debugger, and stop expecting one number to do both jobs.

Benchmark scores are contaminated

The SWE-Bench Illusion paper (arXiv:2506.12286) ran an elegant probe. Give a model only an issue description and a repo name — no repo structure, no code — and ask it to name the buggy file. It should be impossible.

Buggy file path identification (no code provided)
  SWE-Bench Verified          up to 76%
  Repos outside the benchmark up to 53%

Function reproduction (consecutive 5-gram overlap)
  SWE-Bench Verified/Full     up to 35%
  Other benchmarks            up to 18%

The authors are careful — they note directly measuring memorization versus genuine reasoning is challenging, and they lean on cross-benchmark comparison as a proxy. Fair. But a model naming buggy files it has never seen the code for is evidence about the benchmark, not the model.

Worth remembering that SWE-bench Verified is itself the fix for a previous validity problem: it exists because the original had overly specific tests, underspecified issues, and broken environments. Human annotation filtered out 68.3% of the original samples. Two-thirds.

The pattern generalizes. WebArena (arXiv:2307.13854) reported a best GPT-4 agent at 14.41% end-to-end success against 78.24% for humans. GAIA (arXiv:2311.12983) reported humans at 92% versus GPT-4 with plugins at 15%. Those gaps have narrowed since publication — but the survey notes GAIA had already “saturated the easier portion” of itself, prompting a successor, and that WebVoyager found “over-optimistic performance estimates” on web benchmarks, prompting Online-Mind2Web.

Benchmarks decay. Treat a public score as a prior, never as a measurement of your workload.

Tracing is not standardized

If your plan is “adopt the standard,” check the standard’s own banner first.

The OpenTelemetry GenAI semantic conventions have moved to a dedicated repository. As of July 2026 the agent spans spec reads Status: Development, the repo shows no releases published, and the schema URL section is still marked TODO.

Attribute	Requirement	Stability
`gen_ai.operation.name`	Required	Development
`gen_ai.provider.name`	Required (client spans)	Development
`gen_ai.agent.name`	Conditionally Required	Development
`gen_ai.request.model`	Recommended	Development
`gen_ai.input.messages`	Opt-In	Development
`error.type`	Conditionally Required	Stable

Defined span operations: create_agent, invoke_agent, invoke_workflow, plan.

The one Stable attribute is error.type — and it is stable only because it is inherited from core OpenTelemetry, not because anything GenAI-specific has settled. Instrument against these conventions anyway; the shape is right and it beats inventing your own. Just budget for churn, and disbelieve any post telling you agent tracing is a solved, standardized layer. The spec disagrees with them.

What I would actually build

Given all of the above, a defensible suite looks less like a leaderboard and more like a lab with error bars.

Control for length before you believe any win. Either regress it out or hold output length fixed across variants. Until you do, you cannot separate a real improvement from a verbosity effect, and the verbosity effect is bigger than your improvement.
Anchor on references wherever ground truth exists. 70% → 15% is the largest cleanly-measured win available and it costs a prompt field.
Run both orders. Report inconsistency as a metric. Order-flip rate is your judge’s error bar. Publish it next to the score; a score without it is a point estimate pretending to be a measurement.
Jury of small, diverse judges over one big one — with the caveat that the supporting study did not cover math or reasoning. Test it on your data before trusting it there.
Keep humans in the loop, permanently. Not as a phase-one crutch. Dorner et al. is a proof, not a tooling gap: at the frontier, debiasing halves your label budget at best.
For agents, gate on outcomes and debug on trajectories. And measure pass^k, not pass@k, for anything a customer touches. Consistency is the product.
Sample production traffic back into the set. Route on real signals — thumbs-down, a specific tool call, a customer segment — rather than uniformly. LangSmith’s online evaluation docs show the shape: filter runs where a user left negative feedback, then apply a sampling rate for cost control. Your failure set should be grown from actual failures, not imagination.
Never gate on a public benchmark score. Contamination is documented, saturation is documented, and neither measures your workload.

The uncomfortable through-line: none of this makes your eval correct. It makes it honest about its own error, which is a different and more achievable goal. An eval suite reporting “variant B wins by 3 points, order-inconsistency 31%, length uncontrolled” is more useful than one reporting 8.2 — because the first one tells you not to ship on that evidence, and the second one told you to ship.

If your evals have never told you “this result is not distinguishable from noise,” they are not measuring quality. They are producing agreement — with your judge’s preferences, and with your hopes.

FAQ

How accurate is LLM-as-a-judge compared to human raters? In the MT-Bench study that popularized the technique (Zheng et al., NeurIPS 2023, arXiv:2306.05685), GPT-4 pairwise judging agreed with human raters 85 percent of the time when tied votes were excluded. The number that gives it meaning is on the next line of the same table: humans agreed with each other 81 percent of the time on the same data. So a strong judge matches humans roughly as well as humans match each other. That is a comparative claim, not an absolute one. It does not mean the judge is 85 percent correct — it means the judge reproduces human preferences, including the parts of human preference that have nothing to do with whether the answer is right.

Do LLM judges prefer longer answers? Strongly, and it is the single largest style effect measured. In the Length-Controlled AlpacaEval work (Dubois et al., arXiv:2404.04475), changing only a verbosity instruction — same model, same underlying capability — moved a model’s win rate from 22.9 percent to 64.3 percent. That is a 41-point swing purchased with nothing but word count. LMSYS ran a separate Bradley-Terry regression on Chatbot Arena votes and found length carried a coefficient of 0.249 while markdown lists, headers, and bold sat at 0.031, 0.024, and 0.019 respectively. Their conclusion was that length dominates and other formatting effects are second order.

Do LLM judges favor their own outputs? This is more contested than the folklore suggests. The paper usually cited for self-enhancement bias, Zheng et al., explicitly declines to claim it — the authors state that due to limited data and small differences, their study cannot determine whether models exhibit self-enhancement bias. Panickssery et al. (arXiv:2404.13076) did establish something sharper: a linear correlation between a model’s ability to recognize its own text and the strength of its self-preference, demonstrated causally through fine-tuning. But a 2025 challenge (arXiv:2504.03846) argues much apparent self-preference is legitimate, because better generators tend to be better judges. The honest summary is that self-preference is partly earned quality and partly recognition-driven bias, and the harmful portion concentrates where the judge is already wrong.

Is pairwise comparison better than absolute scoring for LLM evals? It depends on what you are defending against, and the sources genuinely disagree because they measure different things. Zheng et al. found pairwise judging tracked human preference slightly better than single-answer grading. But a COLM 2025 paper (arXiv:2504.14716) found pairwise preferences flipped about 35 percent of the time under distractor manipulation versus 9 percent for absolute scores. Pairwise wins on fidelity to human preference on clean data; absolute scoring wins on robustness when something is actively gaming the judge. If your generator optimizes against your judge, the adversarial condition is your normal condition, and that changes the answer.

What is the most effective fix for an unreliable LLM judge? Giving it a reference answer, by a wide margin, in the one setting where it was cleanly measured. Zheng et al. found the default judge prompt failed on 14 of 20 math questions — 70 percent — and reference-guided grading cut that to 3 of 20, or 15 percent. The interesting part is why it failed: GPT-4 could solve those problems on its own but was misled once it saw the candidate answers. Judging is a different task from solving, and a judge with no ground truth to anchor on is doing free-form preference matching whether you intended that or not.

Why is evaluating AI agents harder than evaluating chatbot responses? Because the thing that fails is a trajectory, not a string, and you usually cannot tell where it broke. On the Who&When dataset (arXiv:2505.00212), built from failure logs of 127 LLM multi-agent systems, the best method identified which agent was responsible for a failure 53.5 percent of the time but pinpointed the failing step only 14.2 percent of the time. The authors note that even frontier reasoning models fail to reach practical usability at this. So you can often see that a run failed and roughly who owned it, while remaining nearly blind to when it went wrong — which is exactly the information a fix requires.

Is agent tracing standardized with OpenTelemetry yet? Not in any sense you should rely on. The OpenTelemetry GenAI semantic conventions moved to their own repository, and as of July 2026 the agent spans specification carries a Status: Development banner with no releases published and the schema URL section still marked TODO. Attributes like gen_ai.operation.name, gen_ai.agent.name, and gen_ai.request.model are all Development stability. The notable exception is error.type, which is Stable — but only because it is inherited from core OpenTelemetry rather than being GenAI-specific. Blog posts claiming these conventions are settled contradict the specification’s own banner.

Are agent benchmarks like SWE-bench trustworthy? Treat them as a signal with a known contamination problem, not as ground truth. The SWE-Bench Illusion paper (arXiv:2506.12286) ran a memorization probe: give models only an issue description and a repo name — no repo structure, no code — and ask for the buggy file path. Models hit up to 76 percent accuracy on SWE-Bench Verified versus up to 53 percent on repositories outside the benchmark. Function reproduction showed up to 35 percent consecutive 5-gram overlap on SWE-Bench versus up to 18 percent elsewhere. The authors are appropriately careful that separating memorization from genuine reasoning is hard and they rely on cross-benchmark comparison as a proxy. But a model that can name the buggy file having never seen the code has told you something about the benchmark.