Training large language models (LLMs) with reinforcement learning from human feedback (RLHF) is a crucial but computationally expensive process. Imagine having to constantly pause training while waiting for the model to generate text samples for feedback. That's the traditional, synchronous approach to RLHF, and it's a bottleneck. New research explores a more efficient method called *asynchronous* RLHF, which allows training and text generation to happen simultaneously, like a well-oiled machine. This method cleverly separates these two tasks onto different computing units, allowing the model to learn from older samples while new ones are being generated. The key innovation lies in overcoming the challenge of *off-policy* learning, where the model learns from data generated by a slightly older version of itself. The research reveals that a specific RLHF algorithm, online DPO, is surprisingly robust to this off-policy data, particularly with larger models. The researchers experimented with further optimizations, demonstrating speedups of up to 40% when training a large-scale chatbot while maintaining performance. Although some technical challenges remain, such as communication bottlenecks between the training and generation processes, asynchronous RLHF offers a promising path to making LLM training faster, more efficient, and ultimately, more accessible.