Taming LLM Variance: Ensemble Scoring with Outlier Rejection

Score the same CV twice and you get two different numbers. We found that out early at Navero, where an LLM scores every candidate CV against the skills a role requires — and where a hiring decision rides on the result.

The Problem

Hiring decisions ride on these scores, so run-to-run variance is not a cosmetic issue — it is a fairness issue. A candidate's ranking should not depend on which random sample of the model's distribution they happened to draw.

Lowering temperature does not eliminate the variance, and it degrades the quality of the reasoning. One-shot scoring was simply not trustworthy enough to ship.

The Solution: Ensemble, Reject, Retry

We replaced single-shot scoring with a parallel ensemble strategy:

1. Fan out — run N independent scoring attempts for the same CV in parallel
2. Aggregate — average the per-skill scores across attempts
3. Reject outliers — discard attempts whose scores deviate beyond a statistical threshold from the ensemble
4. Guard the ensemble — if fewer than 60% of attempts survive outlier rejection, the whole ensemble is considered unreliable and re-runs, up to 3 retries

The interesting question is the one most teams skip: what should N be? Every additional attempt costs real money at scale.

We answered it with an eval instead of a debate:

For max_attempts in {1, 3, 5, 7, 9}:
    run the full scoring pipeline 5 times per CV
    compute the coefficient of variation (CV%) of final scores
Plot CV% against attempts; pick the smallest ensemble
whose variance falls under a 5% threshold line.

The output is a chart, not an opinion. It shows variance dropping steeply from one attempt to three, then flattening — the point where extra attempts buy almost no additional consistency. That knee is the ensemble size worth paying for.

Key Lessons

Variance is a product bug, not a model quirk. If two runs disagree, at least one of them is wrong. Treat consistency as a requirement with a number attached, not a nice-to-have.

Reject outliers before averaging. A single degenerate attempt — a misparse, a hallucinated skill — can drag a plain average badly off course. Rejection plus a minimum-survivor ratio keeps one bad sample from poisoning the result.

Size the ensemble with an experiment. "More attempts is better" is true but unbounded. The coefficient-of-variation sweep converts an argument about cost versus accuracy into a chart with a threshold line on it.

Retry the ensemble, not the attempt. When most attempts disagree with each other, the problem is upstream — the input, the prompt, the parse. Re-running the whole ensemble handles that; retrying one attempt just resamples the noise.

Results

• Score consistency across repeated runs is measured and held under an explicit variance threshold
• Ensemble size is chosen from data — the smallest N meeting the threshold — instead of a guess
• Degenerate scoring runs are caught by the survivor-ratio guard and retried automatically instead of reaching a recruiter

Related Reading

• End-to-End Testing LLM Features Without Calling an LLM — keeping model variance out of CI entirely
• Prompt Engineering Without Guesswork — the eval loop that sits upstream of ensemble scoring
• Production RAG: Three Retrieval Strategies and When Each One Wins — where this same variance shows up the moment retrieval goes agentic