The document analyzes how knowledge distillation transfers different teacher properties, showing benchmark knowledge passes through distillation while refusal backdoors largely do not; logit-based distillation can transfer backdoor refusals in zero-shot settings, indicating selectivity based on property structure rather than data exposure.
The document explains how distillation trains a student model to reproduce selected behavior from a teacher, using answers or token probabilities as training targets. It distinguishes distillation from speculative decoding and notes that a smaller student may gain task-specific skill while losing some of the teacher’s breadth.
Team Knowledge Distillation aggregates a whole team's transcripts into editable, persistent documents that capture cross-agent collaboration, handoffs, and working patterns, beyond what single-session context compaction can retain. It uses a three-pass pipeline to extract, cross-reference, and structure knowledge into OKF documents with YAML frontmatter for restart-time injection.
The document surveys quantum knowledge distillation, comparing quantum and classical approaches, and highlights the limited practical gains due to hardware constraints, varied protocols, and incomplete reporting; it calls for standardized benchmarking and careful ablations to assess true quantum advantages.
The paper introduces Localized Context Distillation (LoCD) to address exposure bias and uninformative states in context distillation. By localizing and regenerating only the informative spans from a student's trajectory, LoCD concentrates learning where it matters, leading to stronger performance and more stable, sparser training across several task domains.
Open-R1 is a collaborative project to reproduce and extend DeepSeek-R1’s training methods. Its goals include building distilled models, recreating the reinforcement-learning pipeline, and sharing code, datasets, and results.
Translated title. Original: AI用語の基礎知識【第29回】「モデル蒸留」とは――巨大AIの“賢さのエッセンス”を小さなAIに引き継ぐ技術
Distillation transfers the essence of a large AI model to a smaller one to reduce cost and enable on-device use, by letting the student learn from the teacher’s outputs and reasoning rather than copying the teacher’s internal parameters; it is distinct from quantization and can be combined with other methods like fine-tuning.
Translated title. Original: Sebastian Raschkaが11時間かけて見せる「推論モデルの作り方」 - PaPoo
The piece notes a Xiaomi mimo-v2.6 RL page that publicly displays live training metrics for two models, mimo-v2.6-pro and mimo-v2.6-flash, emphasizing development transparency over product announcements.
LM2LM uses reinforcement learning to train LLMs to propose improvements to language models, guided by a frozen world model that predicts gains from context. Real executions validate the approach, showing cost savings and superior performance over a strong baseline, with evidence of generalization to unseen models.
Translated title. Original: Китайские ИИ-разработчики обучали новые модели на ответах GPT-6 Astra
AI distillation is being used across firms, with OpenAI and Anthropic tightening defenses while Chinese firms defend distillation as global practice; claimed performance gains from distillation are seen in Moonshot AI and DeepSeek models amid US-China tech competition.