Large language models (LLMs) have become incredibly powerful, but are we underestimating their true capabilities? New research suggests that the benchmarks we use to evaluate LLMs may be riddled with errors, painting a misleading picture of their performance. A study focusing on the TRUE benchmark, which measures factual consistency across various NLP tasks like summarization and dialogue, discovered a surprising amount of mislabeled data. Researchers used an ensemble of LLMs, including GPT-4 and PaLM2, to re-annotate examples and flag potential errors. When these flagged examples were reviewed by human experts, it turned out that the LLMs were often right, revealing an error rate of up to 21% in the original datasets. This means that what we sometimes perceive as LLM mistakes might actually be human annotation errors. Furthermore, the study showed that the LLMs' confidence in their predictions correlated with their accuracy in spotting errors. The higher their confidence in a label different from the original, the more likely it was a genuine mistake in the dataset. This discovery has significant implications. First, it suggests that LLMs might be performing better than current benchmarks indicate. Second, training on mislabeled data could be hindering their true potential. The researchers explored techniques to mitigate this, like filtering out or correcting potentially mislabeled examples during training, which led to performance improvements. This study raises important questions about how we evaluate AI and underscores the need for higher-quality datasets. As LLMs become increasingly integral to the annotation process itself, they offer a powerful tool to identify and correct existing errors, paving the way for more accurate assessments of AI's capabilities and potentially unlocking even greater performance gains in the future.