Large Language Models (LLMs) have taken the world by storm, writing stories, poems, and even code. But how far can their abilities be pushed? Can they truly *learn* to perform complex tasks like a human programmer or mathematician? This research explores using Reinforcement Learning (RL) to teach LLMs formal languages—the structured languages of math and code—and unveils some intriguing insights into their learning process. Traditional LLM training focuses on predicting the next word in a sequence, a bit like sophisticated autocomplete. This works well for mimicking human language, but it falls short when true understanding is required. Imagine an LLM trying to write a program: it might generate code that *looks* right, but fails to actually run. That's where RL comes in. Reinforcement Learning is like training a dog with treats: you reward the model for correct outputs and penalize it for mistakes. In this research, the authors applied RL to three tasks: sentiment analysis (generating positive movie reviews), arithmetic, and game synthesis (creating board game rules in a formal language). Sentiment analysis acted as a baseline, proving the RL setup worked as expected. However, things got more interesting with arithmetic. Initially, the LLM struggled, even converging on a naive solution by simply outputting the average answer for all problems! It turned out the model needed help exploring different solutions, so the researchers introduced a novel “batch-entropy regularization” technique. This encouraged the LLM to try out diverse approaches within a batch of problems, rather than sticking to a single strategy. This boost helped the model escape the naive local optimum and start generating better solutions. The final frontier was game synthesis. Here, the complexity ramped up significantly. Despite pre-training the LLM on existing game descriptions, it couldn't consistently generate valid new games. This suggests a fundamental challenge for LLMs: RL might be great for fine-tuning existing abilities, like generating positive sentiment, but struggles to teach entirely new skills from scratch. This research offers a glimpse into the ongoing quest to enhance LLM capabilities. It highlights the limitations of current RL methods for complex tasks and underscores the need for further research. Perhaps future innovations will unlock the full potential of LLMs, enabling them to truly learn and reason like human programmers and mathematicians, pushing the boundaries of AI even further.