Explainable AI (XAI) is crucial for building trust in artificial intelligence, especially in critical sectors like healthcare. But traditional methods for evaluating XAI tools, like user studies, are expensive and time-consuming. What if we could use AI itself to evaluate AI? New research explores the fascinating possibility of using Large Language Models (LLMs) to simulate human participants in XAI evaluations. Researchers recreated a user study comparing different types of explanations, replacing human volunteers with seven leading LLMs, including Llama 3, Qwen 2, Mistral 7B, and GPT-4o Mini. The results are intriguing: LLMs could often replicate the overall conclusions of the original study. However, different LLMs showed varying degrees of alignment with human preferences, suggesting the choice of model matters significantly. Furthermore, factors like the LLM's memory and the way responses are aggregated played a crucial role in how closely the LLM's judgments mirrored human ones. While LLMs won't replace human feedback entirely, this research suggests they offer a powerful new tool for streamlining XAI evaluation, potentially leading to more transparent and trustworthy AI systems in the future. The speed and scalability of using LLMs for this purpose could unlock faster progress in the XAI field, paving the way for more user-friendly and reliable AI tools in healthcare and beyond.