Search

[2611.08379] Module-Aware Optimization for Graph Neural Networks

The paper introduces GraphMuon, an optimizer that adjusts Muon’s matrix updates to account for graph-propagated inputs in GNNs. Across three large-scale OGB benchmarks, it reaches target validation loss 1.32 to 2.42 times faster than AdamW and has lower time-to-target in 13 of 15 tested settings.

[2610.22650] The Ephemeral Cost of Switching Optimizers Across LLM Training Stages

The study shows switching optimizers between pretraining and post-training stages has limited impact on final performance; outcomes are mainly driven by the individual effects of each training stage's optimizer, with Muon often competitive to AdamW during post-training, and final benchmarks not significantly improved by matching pretraining optimizers.

PreCap: A Bounded, Self-Normalizing Activation Function that Matches SELU in Deep Normalization-Free MLPs | Zenodo

PreCap introduces a bounded, smooth activation; with SPreCap scaling it matches SELU in deep MLPs under fair tuning, while the e03 variant slightly outperforms SELU on MNIST at depth 100. The work also provides CUDA and Apple MLX kernels and an installable package, but notes negative results on robustness to learning rate.

Markov Chain Definition and Properties Explained | Daniel Lee posted on the topic | LinkedIn

The text argues for making neural optimization more than local gradient updates by introducing adaptive perturbations and evaluative judges. It outlines a layered system (model, proposal policy, update rule, judge) with continuous, double-loop optimization, archival learning, and tests to guard honesty, while emphasizing that the design of the action space and initial conditions gates all future improvement.

Rikiya Takehi on X: "We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: front… / X

A discussion of editing another language model’s weights to improve math performance without training, using a small weight adjustment and Show more forward passes; claims substantial gains in reasoning and instruction following, framed as Weight-Space Programming.

Adam Karvonen on X: "It looks like Astra didn't natively read Qwen's weights? As I understand it, Qwen would get stuck in a loop and not answer the question. Astra used code to perform some standard interp techniques, found a looping direction, and ablated it from the weights." / X

Astra modifies a model's weights to fix reasoning loops and improve math without training, by applying targeted weight-space edits and claiming significant gains in math and instruction following.

akhaliq/Fibo-2.0-edit-turbo-ComfyUI · Hugging Face

Fibo 2.0 Edit Turbo for ComfyUI provides an fp8 checkpoint that halves download and VRAM versus the bf16 version. It converts weight matrices to fp8 while keeping norms/biases in bf16, requires no training, and runs with native ComfyUI nodes. The fp8 path is experimental with validation pending; users should report results and may revert to bf16 if needed.

An AI has made another AI better with an unexpected method: cutting away what hindered it.

Translated title. Original: Una IA ha hecho mejor a otra IA con un método inesperado: recortarle lo que le estorbaba

MIT CSAIL researchers demonstrate Weight-Space Programming, a method that improves a smaller model by editing a tiny fraction of another model's weights, achieving a substantial jump in math performance without training. The approach is early, not yet peer-reviewed, and its practicality and safety for broader use remain uncertain.

JonathanColetti/Qwen3.5-0.8B-Astra-WeightEdit · Hugging Face

A modified Qwen3.5-0.8B model edits two tensors to fix looping behavior, without training, and reports preliminary benchmark results; authors credit MIT CSAIL researchers and provide instructions and caveats for reproducing the edit.

Hiccup1234/Krea2-Kroma-v0.3.1-turbo-opd-LoRA · Hugging Face

The document describes a delta-LoRA for converting Krea2 into Kroma v0.3.1 turbo opd, using per-layer rank-384 SVD to capture a substantial portion of the delta energy; it is an approximation, not an exact replica, and can be blended with other LoRAs to modify the base model.