The paper analyzes generalization gaps in multi-teacher on-policy distillation (MOPD) versus on-policy distillation (OPD), showing strong in-distribution gains but persistent out-of-distribution deficits. It identifies teacher capability and data coverage as bottlenecks and proposes geometry-based patching and base-preserving replay to improve OOD retention while preserving INDistribution performance.
The document argues that a statement’s truth depends on whether it remains invariant across relevant changes in the environment, not on how likely it is under one distribution. It connects this view to out-of-distribution generalization, language-model testing, and continual learning.
IDM: Intrinsic Distribution Matching adds a moment-based loss to generator EMA features for distribution matching, achieving strong results at 2NFE and evaluating multiple model variants with and without tuning.
The document argues that grokking—delayed generalization—occurs broadly in practical DNN training, not just in contrived setups, and introduces delayed robustness as networks become adversarially robust after a phase shift in how linear regions organize in the input space. It provides an analytical view based on local input-output complexity and region migration, supported by examples in CNNs and ResNets.
A framework uses language models to generate behavioral profiles from existing experiments and tests their predictive power across varied experimental designs; profiles with few features predict outcomes better than generic personas, but adding features offers diminishing returns and cross-domain transfer can be unreliable.
A diagnostic framework, VLA-GAUGE, assesses zero-shot generalization and robustness of vision-language-action models using paired in-/out-of-distribution tasks, perturbation analysis, and targeted fine-tuning; results show limited zero-shot performance but diagnosis-driven refinement yields notable improvements.
The paper introduces AnyGoal, a benchmark for goal-shifted tasks in world-model environments to test how well learned dynamics generalize to new state-action combinations; it finds current models struggle with recombinations and analyzes where generalization fails to guide progress in planning tasks.
MaSS introduces a decoupled structural pathway and a scene-structure discrimination objective to make self-supervised depth learning robust across diverse, mixed-domain data, showing strong generalization to unseen datasets and out-of-distribution scenes.
TEM introduces Transformer Energy-Based Models, claiming they scale beyond diffusion models without guidance and offer scalable, contrastive EBMs with improved generalization, test-time performance, and OOD detection.
The study introduces a benchmark for comparing tabular models on generation, imputation, and prediction, and tests how training objectives affect performance across these tasks. Generalist models can match or beat specialists on utility, but specialists retain an edge in distributional fidelity.