Tag
distillation
Distillation transfers a larger model’s behavior—ranking preferences, generation patterns, or reasoning signals—into a smaller student model. It matters because teams use it to cut inference cost and latency while keeping SLMs useful for reranking, generation, and cross-architecture alignment.
12 articles
Test-Time Harnesses Transfer Skills Without Retraining
Strong models can build test-time harnesses that nearly double weaker models’ performance without updating parameters.
Seed’s anti-distillation rule turns open models into policy
A practical breakdown of Seed’s anti-distillation stance and what it means for teams shipping open-weight models.
Huang’s open-letter playbook for open-weight AI
Jensen Huang’s first X post argues open-weight AI needs fewer locks and more shared infrastructure.
Trump adviser accuses Moonshot AI of stealing Anthropic
A White House adviser says Moonshot AI copied Anthropic models and used restricted Nvidia servers as US-China AI tensions rise.
Direct-OPD reuses weak-model RL gains for stronger models
Direct-OPD lifts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 by distilling RL gains from a weaker model.
OPD lets you distill skills without brute-force RL
I break down On-Policy Distillation and turn the idea into a copy-ready post-training template.
DanceOPD distills image-editing skills into one model
DanceOPD trains flow-matching image models to combine text-to-image and editing skills without them fighting each other.
UNIEGO unifies egocentric video with proxy teachers
UNIEGO uses proxy models to distill nine teachers into one egocentric encoder.
Apple’s Gemini deal turns cloud AI into local AI
Apple is using Google Gemini distillation and Nvidia confidential compute to push Siri toward local-first AI with cloud backup.
CARV cuts diffusion-teacher gradient variance
CARV reduces Monte Carlo variance in diffusion-teacher pipelines by reusing expensive upstream work and smarter noise sampling.
Select-to-Think: Let SLMs Re-rank Themselves
A new method lets small language models re-rank their own candidates instead of calling an LLM at inference time.
TIDE distills diffusion LLMs across architectures
TIDE distills diffusion LLMs across architectures, adding noise-aware weighting and tokenizer-aware objectives to improve a 0.6B student.