LLMs stumble on counterintuitive probability

A benchmark shows LLMs handle standard probability well but break down on counterintuitive cases.

• Research org: Unspecified in arXiv abstract
• Core data: 0.96 average accuracy on standard problems
• Breakthrough: Benchmarked standard and counterintuitive discrete probability datasets

This paper is useful because it tests something developers often assume models already do well: reasoning about uncertainty. The authors show that a model can look strong on familiar probability questions while still getting tripped up by wording, misleading cues, and non-canonical forms of the same underlying problem.

For anyone building assistants, tutors, or decision-support tools, that matters. If a model is going to explain risk, compare odds, or help users reason about dice-like outcomes, you need to know whether it is actually reasoning probabilistically or just pattern-matching common templates.

What problem this paper is trying to fix

The paper looks at a narrow but important question: how reliable are large language models when the right answer depends on probabilistic reasoning rather than memorized patterns? The authors argue that performance on advanced math benchmarks does not necessarily mean a model can reason correctly about discrete probability in messy, adversarial, or unfamiliar formulations.

To probe that gap, they built two datasets. One contains standard exercises, which should be relatively straightforward for a model trained on lots of textbook-like material. The other contains counterintuitive exercises designed to trigger heuristic reasoning, where a model may be tempted to follow surface cues instead of working through the probability structure.

This is a practical distinction. In real applications, users do not always ask questions in the cleanest possible form. They rephrase, omit context, add distracting details, or accidentally introduce misleading hints. A system that only works when the question looks canonical is not robust enough for many production settings.

How the method works in plain English

The setup is a controlled benchmarking study on discrete probability problems. The authors evaluated eight state-of-the-art models, and each model was tested both with and without Chain-of-Thought prompting.

That matters because Chain-of-Thought is often used as a way to improve reasoning performance. Here, the paper uses it as a stress test: does encouraging step-by-step reasoning actually help with probability, or does the model still collapse when the problem is framed in a way that invites the wrong shortcut?

The study also checks two robustness issues that developers should care about. First is token bias: whether the model performs differently when the same problem is rewritten in a disguised variant instead of the canonical form. Second is prompt contamination: whether adding misleading suggestions into the prompt changes the answer quality.

In other words, this is not just a “can the model solve the problem?” benchmark. It is a “how stable is the model when the same logic is wrapped in different language?” benchmark. That is a much more realistic test for deployed systems.

What the paper actually shows

The headline result is a sharp split between easy-looking and counterintuitive probability questions. Across the evaluated models, average accuracy is 0.96 on standard problems, but only 0.59 on counterintuitive ones.

The abstract also reports that performance drops by over 20% when canonical formulations are replaced by disguised variants. That is a strong sign that phrasing alone can change how well the model handles a problem, even when the underlying probability task is the same.

Misleading suggestions in the prompt are even more damaging: they reduce performance by up to 34%, and no model is immune. The abstract does not break down those drops by individual model, so we do not know which systems were most resilient or whether Chain-of-Thought helped in specific cases.

What the paper does not provide in the abstract is just as important. There are no per-model benchmark tables, no dataset sizes, and no confidence intervals. So while the direction of the findings is clear, the abstract alone does not let us judge statistical strength or compare model families in detail.

• Eight state-of-the-art models were tested
• Each model was evaluated with and without Chain-of-Thought prompting
• Two datasets were used: standard exercises and counterintuitive exercises

Why developers should care

If you are building an LLM feature that touches uncertainty, this paper is a warning against overtrust. A model that answers textbook-style probability questions correctly may still fail when the same logic is phrased in a less familiar way.

That has direct implications for product design. If your app relies on an LLM to explain odds, handle risk analysis, or teach probability, you should assume that wording matters. You may need stronger validation, better prompt normalization, or an external reasoning layer instead of relying on raw model output.

The token-bias result is especially relevant for eval design. If a model’s score changes materially when a problem is reworded, then a single benchmark phrasing is not enough to establish reliability. You need multiple formulations that test whether the model understands the structure of the problem, not just the surface pattern.

The misleading-suggestion result also maps cleanly to real-world usage. Users often include hints, assumptions, or half-formed reasoning in their prompts. This paper suggests those additions can steer models badly, even when the underlying task is simple. For developers, that means prompt hygiene is not a minor detail; it is part of correctness.

What this means in practice

The broader takeaway is not that LLMs are useless at probability. The paper shows they can do very well on standard exercises. The problem is robustness. Once the task becomes counterintuitive, disguised, or contaminated by misleading cues, performance drops enough to matter.

That makes the paper a useful reminder that “reasoning” benchmarks are not interchangeable. Strong results on advanced math do not automatically transfer to probabilistic reasoning, especially when the answer depends on resisting a tempting shortcut.

For engineers, the safest interpretation is simple: treat LLMs as brittle on probabilistic edge cases until they are tested on varied formulations. If the output matters, verify it with deterministic logic, a calculator, or a domain-specific checker rather than assuming the model has internalized the probability rule.

And for eval teams, this paper points to a better testing pattern: include canonical and disguised variants, include misleading prompt noise, and measure whether performance stays stable. That kind of robustness testing is closer to how real users stress a system in production.

In short, the paper argues that current LLMs are not yet genuine probabilistic reasoners, even if they look impressive on other math tasks. The gap is not just accuracy; it is reliability under rewording, distraction, and counterintuitive structure.