Beyond the Best Guess: Why LLMs Need More Than One Right Answer

New research from Cognizant AI Lab shows that Evolution Strategies can improve LLM accuracy while preserving a broader range of possible solutions, making it a promising approach for problems where finding the right answer may require more than one attempt.

LLMs are increasingly being used for problems where finding the right answer can take more than one attempt. In math, science, coding, and other discovery tasks, a model’s first or most likely response may not be its best one. One way to improve performance is to give the model more opportunities to explore, generating multiple possible solutions and using additional test-time compute to find the right one.

The challenge is that the way we typically post-train LLMs can actually make that search less effective. Reinforcement learning (RL) improves a model by making high-reward answers more likely, which can increase the accuracy of its first response. But it can also concentrate the model around a narrower set of answers. So while the model may become better at its first guess, it has fewer alternatives to explore when given more attempts. In some cases, this effect is strong enough that an untuned base model can outperform its own RL-trained version at pass@k, which measures whether a correct answer appears among multiple sampled responses.

In our latest research, Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies, we explore whether we can get the accuracy gains of post-training without losing that range of possible solutions. We use Evolution Strategies (ES), a gradient-free approach that optimizes directly in model weight space rather than following gradients through sampled responses. Where RL can improve a model’s first response at the expense of the range of answers it produces, ES allows us to improve that first response while preserving more of the solutions the model may need to find the right answer over multiple attempts. Across models from 1.5B to 32B parameters and multiple math benchmarks, ES improves pass@1, or single-attempt accuracy, while maintaining broader solution coverage as more responses are generated.

This becomes increasingly important as more AI systems use additional inference compute to search for better answers. Repeated sampling, self-consistency, tree search, and agentic systems can only explore solutions the underlying model is still capable of generating. If post-training has narrowed that space too much, spending more compute cannot bring those solutions back. For discovery problems, where the useful answer may be an unlikely one, our research suggests ES can make the model’s first answer better without closing off as many of the other paths it may need to find the right one.

Reinforcement Learning Improves the Best Guess at the Expense of Solution Coverage

To understand why this tradeoff emerges, it helps to look at what RL is actually optimizing. Methods such as Group Relative Policy Optimization (GRPO) reinforce high-reward responses by increasing their probability. This is highly effective when the goal is to improve the answer a model is most likely to produce. But repeatedly concentrating probability on successful responses can also narrow the model’s output distribution, pushing lower-probability alternatives further out of reach.

The consequences become clearer as the model is given more attempts. At pass@1, concentrating probability around high-reward answers can work well. As k increases, however, a broader distribution becomes more valuable because each additional sample is another opportunity to surface a different solution. If post-training has removed too many of those alternatives, sampling more eventually stops buying much.

Our results show this tradeoff directly. Relative to the base model, RL increases pass@1 but loses solution coverage as more responses are sampled. ES follows a different pattern, improving pass@1 while also preserving broader coverage. Across the measured range up to k=128, ES remains above the base model, while RL can eventually fall below it.

Figure 1: RL sharpens accuracy while ES broadens solution coverage. Relative to the base model, RL improves pass@1 but loses coverage as k increases. ES improves both.

A model can therefore become better at giving one answer while becoming less capable of finding other answers that were previously within reach. That is a poor tradeoff if the model will ultimately be used as part of a larger search process. The question, then, is whether we can improve accuracy without applying the same pressure to narrow the solution space.

Evolution Strategies Takes a Fundamentally Different Route to the Same Goal

Rather than optimizing the probabilities of sampled token sequences, ES operates directly on the model’s weights. It applies random perturbations to those weights, evaluates a population of perturbed models, and uses the perturbations associated with stronger performance to determine how the underlying model should be updated.

That changes where the optimization pressure is applied. RL directly increases the probability of high-reward outputs, progressively sharpening the model’s output distribution around a smaller set of solutions. ES instead optimizes expected performance over a distribution of weight perturbations, favoring regions of parameter space that continue to perform well when the weights are perturbed.

This gives ES a useful property for discovery. Broader solution support is preserved through the optimization process itself rather than through an additional diversity penalty or regularizer. The model can improve without concentrating as much probability around a narrow set of responses, leaving more viable solutions available when it is later given additional opportunities to search.

The mechanism is promising, but preserving coverage is only useful if it translates into better performance. To answer that at a scale where the result would mean something, we tested ES across model families, model sizes, and increasingly difficult mathematical reasoning benchmarks.

ES Wins on Pass@k Across Model Families, and the Gap Widens as Sampling Increases

We evaluated Qwen2.5 and Qwen3 models ranging from 1.5B to 32B parameters across GSM8K, MATH500, Olympiad Bench, and Minerva. Rather than looking only at whether each model could get the answer right once, we tracked what happened as the number of sampled responses increased.

On GSM8K, ES and RL typically cross around k=2, after which ES pulls ahead. The difference grows as more responses are generated: RL tends to plateau sooner, while ES continues improving. For the Qwen2.5-Instruct models, the effect is strong enough that the untuned base model eventually overtakes its RL-trained counterpart at larger values of k. The same collapse is not observed with ES. 

The pattern also persists at larger scales. At 7B, 14B, and 32B parameters, the comparisons highlighted in the paper show ES outperforming state-of-the-art RL checkpoints on pass@k across MATH500, Olympiad Bench, and Minerva, respectively. The broader experiments reinforce the same solution-coverage advantage, although some individual model and benchmark combinations are competitive rather than strict ES wins at every value of k.

Figure 2: ES compared with state-of-the-art RL checkpoints at 7B, 14B, and 32B parameters. The pass@k advantage of ES persists as models scale, with broader solution coverage allowing performance to continue improving as more responses are sampled.

This changes the return we can expect from additional inference compute. Sampling a model more often only helps if there are useful possibilities left to uncover. Once post-training has narrowed the distribution around a small set of answers, spending more compute increasingly returns variations of what the model was already likely to produce. With ES, more of the search space remains intact, so giving the model more chances can continue to pay off.

But higher pass@k still leaves an important question unanswered. Is ES simply preserving the base model by changing it less, or is it actually learning new solutions while retaining old ones?

ES Gains New Solutions While Losing Fewer Existing Ones

To separate those possibilities, we looked at what happens to individual problems during post-training. We measure progressions as problems the base model could not solve but the post-trained model can, and regressions as problems the base model could solve but the post-trained model can no longer solve.

The results show that ES is not preserving coverage by learning less. On MATH500, ES produces 6 progressions and 3 regressions, compared with 3 progressions and 17 regressions for RL. On Olympiad Bench, ES produces 56 progressions and 15 regressions compared with 33 and 49 for RL. On Minerva, ES produces 22 progressions and 6 regressions compared with 16 and 40 for RL. Across these benchmarks, RL produces roughly three to seven times as many regressions.

Figure 3: Progressions and regressions for Qwen2.5-Math-7B across MATH500, Olympiad Bench, and Minerva. ES gains more newly solvable problems while losing substantially fewer problems the base model could already solve.

In other words, ES is not choosing preservation over learning. It gains more new correct solutions while retaining more of the solutions the base model could already produce. That combination helps explain why its pass@k curves continue to climb while RL saturates earlier. 

There is another revealing difference in what happens when the models get things wrong. Across MATH500, Olympiad Bench, and Minerva, ES has 24 total regressions compared with 106 for RL. ES failures also retain higher answer entropy, meaning the model continues to produce a broader range of possibilities. More than 25% of RL regressions have entropy close to zero, indicating that the model has instead converged around a small number of incorrect responses. 

That gives us a fuller picture of what distributional collapse looks like in practice. It is not only that RL loses some correct solutions. When it loses them, it can become highly confident in a narrow set of wrong alternatives. ES leaves more uncertainty intact, and with it, more room for another attempt to take the model somewhere different.

Why This Matters

These results become more consequential when we look at how LLMs are starting to be used. Increasingly, a model is not expected to answer once and stop. Repeated sampling and self-consistency generate multiple responses directly, while approaches such as tree search and agentic systems can use additional inference to explore and evaluate different paths.

Those techniques may organize the search differently, but they share an important dependency: the search can only work with possibilities the underlying model is capable of generating. A sophisticated inference strategy cannot recover a useful solution that post-training has already pushed out of reach.

Our self-consistency experiments provide a concrete example. Here, the model generates multiple independent responses and selects a final answer through plurality voting. As the number of samples increases, ES overtakes RL across MATH500, Olympiad Bench, and Minerva. The broader distribution is therefore not simply producing diversity for diversity’s sake. It translates into better final answers when additional inference compute is actually put to work. 

For discovery work in science, math, and code, this changes what we should expect from post-training. The answer worth finding may be one the model initially considers unlikely. That is the reason to search in the first place. If training improves the most likely answer by removing too many of the alternatives, we make the first guess better while weakening the model’s ability to go beyond it.

ES shows that this tradeoff is not inevitable. It improves single-attempt accuracy while preserving more of the solution space, giving additional inference more useful possibilities to explore. As models shift from answering questions to searching for answers, solution coverage becomes an increasingly important part of what makes the model capable.

What Comes Next

That shift also raises a natural next question: if we care about how well a model performs across an entire search process, why not train directly for that?

Because ES is gradient-free, the objective it optimizes does not need to be differentiable. Future work could therefore optimize pass@k directly during training instead of optimizing individual responses and hoping solution coverage survives as a side effect.

The same principle could potentially extend to self-consistency voting accuracy or multi-step agentic success rates. These are future research directions rather than results demonstrated in the current experiments, but they point toward a closer connection between what we optimize during post-training and what we ultimately ask models to accomplish at test time.

As LLMs move from answering questions toward searching for solutions, post-training needs to support that shift. Beyond the Best Guess shows that improving the first answer does not have to mean narrowing the rest of the solution space. By improving accuracy while preserving coverage, ES offers a promising foundation for models that can make better use of every additional opportunity to find the right answer.

Explore the code on Github and models on HuggingFace.

Read the paper.