Training large language models (LLMs) is a computationally and memory-intensive process. As these models grow larger, fitting them onto available hardware for training and fine-tuning becomes increasingly challenging. But what if there were a way to significantly reduce memory needs without sacrificing performance? New research into a technique called AdaRankGrad suggests exactly that. Traditional approaches like low-rank adaptation (LoRA) attempt to address the memory bottleneck by introducing smaller, parallel trainable matrices alongside the main model weights. While effective to a degree, these methods can compromise performance compared to full-rank training. The problem is that forcing model updates into a lower-rank space can disrupt the natural learning dynamics and requires careful initial training to mitigate the impact. AdaRankGrad takes a different approach. It leverages a newly discovered phenomenon: as LLM training progresses, the rank of the calculated gradient updates naturally decreases, asymptotically approaching rank one. In simpler terms, the essential information needed to update the model becomes concentrated within a smaller and smaller subspace. AdaRankGrad capitalizes on this by adaptively reducing the rank of the gradient calculations throughout the training process using efficient, online-updated low-rank projections. This essentially allows the model to fine-tune its parameters using a dynamically shrinking set of update directions, achieving significant memory savings without the artificial constraints of fixed low-rank methods. The researchers behind AdaRankGrad also introduce a randomized Singular Value Decomposition (SVD) scheme to further speed up the process of finding the right projection matrix. Experimental results are promising. When fine-tuning a RoBERTa-base model on the GLUE benchmark, AdaRankGrad demonstrated accuracy improvements while using significantly less memory than LoRA and GaLore (another memory-efficient training method). Similar memory savings and performance gains were observed when pre-training LLaMA models on the massive C4 dataset. AdaRankGrad offers a compelling new strategy for training ever-larger LLMs. By working *with* the natural dynamics of gradient descent, it promises to unlock greater efficiency and scalability, paving the way for more powerful and accessible language models in the future. Further research will likely explore its application with different optimizers (beyond Adam) and investigate alternative subspace projection algorithms. Additionally, analyzing its effectiveness in knowledge editing tasks represents an exciting direction for future development.