Predictable Scaling (PS) forecasts full-scale model performance from small-scale proxies using scaling functions and curved-fit projections with uncertainty. It compares three strategies, showing increasing expressiveness, with Gaussian Processes delivering notable MAE improvements, and demonstrates substantial compute reductions while ranking hypotheses across two model types.
The work presents a data-efficient method to discover scaling laws for heterogeneous ranking models. By introducing a scaling layer and Axia, it predicts architecture behavior from low-dimensional hyper-hyperparameters, uses Bayesian-like selection to form Pareto-optimal configurations, and evaluates with a fleet protocol, achieving modest offline entropy gains and online lift across Meta's ads system.
The article describes a study on Mistral 7B where removing layers showed that larger models gain from depth but face rising costs and diminishing returns; some errors persist due to irreducible factors, and a committee-like averaging of floor edits helps retain performance, illustrating scaling laws and practical limits for upgrading models.
A study introduces a rewriting scaling law for synthetic data in model pretraining, linking loss to model size, data tokens, rewrite fraction, and compute spent on rewrites. It shows, across compute- and data-limited setups, that higher compute supports larger rewrite fractions and enables strong performance with rewrites using substantially less rewriter compute.
The document explains overfitting and underfitting in statistical modeling, illustrating how model complexity, training optimization, and data fit affect generalization, and lists techniques to reduce overfitting such as cross-validation and regularization.
The author argues that scaling laws predict improvements in next-token loss but do not guarantee steady gains in reasoning, accuracy, safety, or other useful abilities. Some claims of emergent abilities may reflect evaluation methods, and the limits of scaling vary by capability.
The paper analyzes how to optimize compute for large language model training by balancing model size and training data. It finds current models are larger than optimal for their compute budgets, suggesting a smaller model trained on more data could be more efficient.
The paper presents a method that uses feature-invariant scaling laws and statistical inference to estimate the full-scale effects of recommendation-system changes from small experiments. Tests across three model families and more than 40 changes found that the method can identify useful changes while reducing compute needs.
Harness Scaling introduces a new way to scale agentic reinforcement learning by using a diverse set of execution configurations. It shows that structured, diverse harness training improves generalization to unseen white-box and black-box harnesses more than single-harness training.
LLMs struggle with adaptive information gathering in experimental design. The work trains reasoning for action that optimizes information gain, using an independent generative model, and shows improved performance and transfer across tasks with lower cost than baselines.