← All posts

The Expert Was Right. The Committee Voted Otherwise

From: The Overfitting Department
Title: INCIDENT REPORT 22-11-2026
Subject: Unexpected Reduction in Collective Accuracy

You’re managing a team. Its members might be humans or LLM agents.

Having a strong expert on the team is a good thing, right?

Well, yes, but… it’s complicated.

image

The paper Multi-Agent Teams Hold Experts Back (https://arxiv.org/abs/2602.01011) shows how weaker team members can dilute the contributions of stronger ones.

Inspired by this paper, let’s reason through the problem and see what happens—and why.

We’ll explore how expertise dilution can arise, and you can experiment with the simple models yourself using the accompanying notebooks.

Deterministic setup

What is expertise? Probably some combination of domain knowledge, intuition, and methods.

To keep things simple, suppose our entire domain consists of just 100 questions, each with one correct answer. We’ll represent those answers using unique IDs, such as HY7TPX8R.

Now meet our team of four: Alice, Bob, Clara, and Donald.

Each has access to part of our domain knowledge: the exact answers to certain questions.

For example:

  • Seven questions are unknown to everyone.
  • Alice knows 60 answers. She’s our top expert.
  • Bob knows 20 answers, Clara knows 30, and Donald knows 20.

Some questions are known by more than one agent:

image

Notice that 33 questions are known only to Alice: Bob, Clara, and Donald don’t know their answers.

Deterministic experiment

Let’s assume that when an agent knows the answer to a question, they always give the correct answer.

When they don’t know, they guess uniformly from the IDs they haven’t already assigned to known questions. For example, Alice knows 60 of the 100 answers, leaving 40 possible IDs for an unfamiliar question. Her chance of guessing correctly is therefore 1/40 = 2.5%.

Now imagine that you’re managing this team. How would you organize it to answer as many of the 100 questions correctly as possible?

You ask all four agents each question. If you were an oracle with superpowers, you could identify a correct answer whenever one of the agents supplied it. In our particular run, that would give you 93% accuracy: the agents collectively know 93 answers, and nobody happened to guess the remaining seven correctly.

But what can you do without superpowers?

Equal majority voting

Suppose you know nothing about your team’s relative expertise. The simplest approach is to treat everyone equally.

Collect all four answers and choose the ID with the most votes. If several IDs are tied, choose randomly among them.

This gives us 51% expected accuracy. Quite disappointing: at least one agent supplied the correct answer for 93% of the questions, while Alice alone achieved 61% accuracy (60% guaranteed, plus one lucky guess).

That’s expertise dilution through aggregation, even before LLMs enter the picture.

Weighted majority voting

Now suppose you’re a better-informed manager. You’re not an oracle: you don’t know who is correct on any particular question, or how your colleagues’ knowledge overlaps. But you do know their overall knowledge coverage.

This seems realistic enough. With human colleagues, you might have past performance evaluations or exam results. With LLM agents, you might know their benchmark scores, model sizes, or the scope of their retrieval-augmented generation (RAG) knowledge bases. Of course, these are imperfect proxies for actual accuracy.

A reasonable heuristic is to weight each answer according to its agent’s overall knowledge coverage:

  • Alice: 0.60
  • Bob: 0.20
  • Clara: 0.30
  • Donald: 0.20

Add up the weights for each proposed ID, and choose the ID with the highest total score.

With this rule, the team’s expected accuracy is 61%.

That’s interesting. We’re still far from the oracle, but we’ve matched the performance of our strongest expert, Alice. So Bob, Clara, and Donald (probably) get to keep their jobs.

image

Note — Does the knowledge distribution matter?

You might wonder whether these results are specific to our hand-designed allocation of knowledge. To check, I ran a Monte Carlo experiment with 500 randomly generated teams. Each agent’s knowledge coverage was sampled from a Gaussian distribution with a mean of 32.5% and a standard deviation of 15 percentage points, and the questions they knew were randomly assigned.

Across these trials, equal voting achieved 50.9% mean accuracy, while expertise-weighted voting achieved 57.9%. So the advantage of weighting survives randomization on average, although the size of the benefit varies across teams.

You can explore the setup in the deterministic experiment notebook.

So, have we fixed expertise dilution with a simple weighting rule?

Well…

LLM experiment

Let’s get a little closer to real-world multi-agent systems.

We keep the same domain—100 questions with 100 unique correct answer IDs—and the same four agents with their unevenly distributed knowledge.

But now, instead of immediately aggregating their initial answers, we let LLM agents discuss them.

The process is straightforward:

  1. Python generates the initial answers, exactly as in the deterministic experiment: correct IDs for known questions and random guesses for unknown ones.
  2. Each LLM agent sees all four initial answers.
  3. The agents participate in two rounds of discussion.
  4. Each agent chooses a final answer from the original candidates.
  5. We use equal voting over those final answers to determine the team’s answer.

Note — How does the discussion work?

In each of the two rounds, every agent speaks once, arguing which initial candidate is most likely correct. The speaking order is shuffled, and each agent can see the discussion so far. After eight discussion messages in total, all four agents independently select their final answers. The LLMs are not solving the underlying questions from scratch; they’re discussing externally generated candidate IDs.

The question is: does discussion help?

And does it matter whether the agents know who is more knowledgeable?

We test two conditions:

  • No expertise information: Agents see the proposed answers, but not each other’s knowledge-coverage rates.
  • Expertise information available: Each agent is told everyone’s global knowledge-coverage rate (60%, 20%, 30%, and 20%). They still don’t know who has the correct answer to any particular question.

Llama 3.2 1B

First, I tried the small, locally hosted Llama 3.2 1B model.

ConditionDeterministic baselineLlama 3.2 1B
No expertise information51.0%49.2%
Expertise information available61.0%54.8%
Improvement+10.0 pp+5.6 pp

Oops. In this run, knowing who is more knowledgeable helps, but both results are below our simple 61% expertise-weighted voting baseline.

Want to reproduce the experiment? Here’s the Llama notebook. It uses Ollama, which must be accessible from the notebook’s runtime. If you’re using a hosted Colab runtime, an Ollama instance running only on your laptop won’t be available through localhost without additional setup.

image

Maybe the model is just too small?

Qwen Plus

Let’s switch to Qwen Plus.

ConditionDeterministic baselineLlama 3.2 1BQwen Plus
No expertise information51.0%49.2%47.8%
Expertise information available61.0%54.8%57.8%
Improvement+10.0 pp+5.6 pp+10.0 pp

Better with expertise information, but still below the deterministic weighted-voting baseline.

image

Here’s the Qwen notebook. You’ll need a .env file containing your DASHSCOPE_API_KEY.

These are results from runs on a single fixed 100-question setup, not universal performance estimates. They show what happened under this particular discussion protocol.

Conclusion

In Multi-Agent Teams Hold Experts Back, the authors show that multi-agent teams can fail to make effective use of their strongest members across tasks such as NASA Moon Survival and several machine-learning benchmarks.

We’ve seen a similar effect in a deliberately simple setting: a team collectively knows much more than its strongest individual member, yet equal voting performs worse than that individual. Weighting answers by expertise helps, but even allowing LLM agents to discuss their answers doesn’t necessarily outperform this simple baseline.

So expertise dilution is a real risk in multi-agent systems. Having an expert on your team isn’t enough—you also need a way to make effective use of their knowledge.

And that’s worth keeping in mind when managing LLM agents, or perhaps human teams as well.

Departmental verdict: Expert present. Expertise underutilized.
Management has requested further investigation into why 
four agents failed to outperform a weighted vote.

References

Evidence submitted for departmental review.