Self-Distillation Can Shrink Model Diversity

Self-distillation can boost pass@1 while quietly reducing rollout diversity and hurting out-of-distribution robustness.

• Research org: Unspecified in arXiv abstract
• Core data: No benchmark numbers in abstract
• Breakthrough: Analyzes sampled-demonstration self-distillation as a biased policy update

On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity is a warning shot for anyone using self-distillation to improve model quality: the method can make a model look better on average while quietly narrowing the set of answers it can produce. In other words, you may get stronger top-1 performance and still lose the diversity that helps with harder or shifted inputs.

The practical issue is simple. If a model is trained to imitate itself using demonstrations sampled from its own outputs, it can start reinforcing the same high-probability paths over and over. That matters for developers because systems that seem better on standard evaluation can still become brittle when a task needs multiple valid strategies, not just one dominant answer pattern.

What problem this paper is trying to fix

The paper looks at on-policy self-distillation, a setup where one model acts as both teacher and student. The teacher is conditioned on a correct demonstration and gives dense token-level feedback to the student. This is appealing because it can improve pass@1 accuracy without requiring a separate teacher model.

But the authors argue that this setup has a hidden failure mode: rollout diversity drops, and pass@k curves flatten. That means generating more rollouts stops helping as much as you would expect, because the model keeps producing similar solutions instead of exploring different ones.

For engineers, that is a meaningful tradeoff. A model that scores well on the first answer can still be a poor choice for tasks where breadth matters, such as search, reasoning, synthesis, or any workflow where you want multiple distinct candidate solutions.

How the method works in plain English

The key design choice is the use of sampled correct demonstrations. The teacher evaluates each student rollout while being conditioned on a sampled correct rollout, and that feedback is then fed back through the model’s own learned biases.

The paper’s theoretical analysis says the optimal self-distillation policy tilts the base distribution using a pointwise conditional mutual information score between the student’s rollout and the correct rollout used as context. That is a formal way of saying the training signal does not just reward correctness; it also nudges probability mass toward answers that already fit the model’s preferred modes.

By contrast, the ideal optimal on-policy reinforcement learning setup preserves probability ratios among equally correct rollouts. That distinction is important: RL can reward correctness without necessarily collapsing the spread of valid solutions, while self-distillation can amplify existing probability gaps and concentrate mass on already-dominant modes.

In practical terms, the model is not merely learning “what works.” It is also learning “what it already tends to do,” which can make the policy more peaked and less exploratory.

What the paper actually shows

The authors study the effect in both theory and experiments. They show that self-distillation with sampled demonstrations can reduce rollout diversity and flatten pass@k curves, which is the empirical sign that extra samples bring less and less benefit.

They test the idea on a controlled graph path-finding task and on science question-answering benchmarks. The abstract does not give benchmark numbers, so there are no reported scores to quote here. What it does say is that self-distilled models match or exceed RL on average performance, while showing substantially lower functional and semantic diversity.

That combination is the real takeaway. If you only look at average accuracy, self-distillation can appear competitive or better. If you also care about the diversity of outputs, the picture changes sharply.

The paper also reports that the self-distilled models fail on out-of-distribution settings that require diverse strategies. That is exactly where a narrow policy becomes a liability: the model can overcommit to one family of solutions and miss alternatives that would have worked on shifted inputs.

Why developers should care

If you are building agents, reasoning systems, or any pipeline that samples multiple candidates, diversity is not a cosmetic metric. It affects whether beam search, reranking, self-consistency, or multi-sample selection actually buys you anything.

This paper suggests that self-distillation can undermine that benefit. A model may still look good on pass@1, but if its pass@k curve flattens, your extra inference budget is buying less than you think. That has direct implications for evaluation design, training choices, and how much trust you place in single-number leaderboards.

It also points to a broader engineering lesson: optimization objectives can hide distributional collapse. If training pushes too hard on the answers the model already likes, you may end up with a system that is more confident, less varied, and less robust outside the training distribution.

Limitations and open questions

The abstract is clear about the failure mode, but it does not provide benchmark numbers, ablation details, or implementation specifics. So while the direction of the effect is well stated, the exact size of the tradeoff is not visible from the raw summary alone.

It is also not a blanket rejection of self-distillation. The paper says the method can match or exceed RL on average performance, which means it may still be useful when top-line accuracy matters more than diversity. The open question is how to keep the gains while avoiding the collapse in output variety.

For practitioners, that means self-distillation should be evaluated with more than one metric. If your use case depends on diverse reasoning paths, robust sampling, or out-of-distribution resilience, you should check whether a training method is improving the answer you see first at the expense of the answers you do not see yet.

• Self-distillation can improve average accuracy while narrowing the model’s output space.
• Sampled demonstrations may reinforce existing probability biases instead of preserving diverse correct solutions.
• For multi-sample or out-of-distribution tasks, diversity metrics matter as much as pass@1.