LLM Bias in Automated Evaluation

The Problem with LLMs as Judges

You can't trust Large Language Models as judges. That's what Bhaskarjit Sarmah said at a workshop at DHS 2026, and it's a point that stuck. LLMs are biased, and that's a problem when we're using them to evaluate everything from student code to research papers.

And it's not just that LLMs are biased - it's that we're relying on them more and more to make decisions. They're fast, they're cheap, and they scale. But speed and cost shouldn't come at the expense of fairness.

How LLMs Work

So, how do LLMs actually work? They're trained on vast amounts of data, which they use to generate text or make decisions. But the data they're trained on is often biased, and that bias gets perpetuated in their decisions.

For example, if an LLM is trained on a dataset that's predominantly written by men, it's going to be better at understanding and evaluating text written by men. And that's a problem when we're using it to judge research papers or student code.

The Consequences of LLM Bias

So, what are the consequences of LLM bias? It can perpetuate existing inequalities, for one thing. If an LLM is biased towards evaluating text written by men, that means that women are going to be at a disadvantage.

And it's not just about gender - LLM bias can affect anyone who doesn't fit the mold of the data they were trained on. It's a problem of representation, and it's one that we need to solve if we're going to use LLMs to make decisions.

What We Can Do

So, what can we do to mitigate LLM bias? We need to diversify the data they're trained on, for one thing. That means including more women, more people of color, and more people from different backgrounds.

We also need to test LLMs for bias, and be transparent about the results. If an LLM is biased, we need to know about it, and we need to take steps to fix it.

The Verdict

LLMs are not perfect judges, and we need to be careful about how we use them. They're fast and cheap, but they're not fair, and that's a problem. We need to take steps to mitigate LLM bias, and we need to be transparent about the results. Only then can we trust LLMs to make decisions that affect our lives.