In recent years, the release of each new AI model has been accompanied by a flurry of claims regarding its advancements and superiority over previous technologies. Companies often advertise their latest models as outperforming competitors in a series of benchmark tests. For instance, GPT-4, a model from a well-known AI research lab, has been highlighted for surpassing other existing models in various assessments. However, experts argue that these benchmarks might not be as reliable or indicative of real-world performance as they appear. The benchmarks typically used to measure AI progress may be inadequate for several reasons. Firstly, benchmarks often consist of standardized tests that could oversimplify complex tasks AI is meant to perform. This raises the question of how well these models perform outside controlled environments. Secondly, the rapid advancements in AI mean that current benchmarks might not adequately capture the nuances of newly developed skills or applications. This begs the question: are we truly measuring progress? Or are we simply measuring a model’s ability to excel in a narrow set of criteria? A broader perspective suggests that instead of relying solely on benchmarks, the AI community needs to adopt more holistic evaluation metrics. New measures could take into account AI’s effectiveness in diverse and dynamic real-world situations, the ethical implications of its use, and its interaction with human users. It’s essential to develop metrics that reflect AI’s societal impact, balancing innovation with responsibility. As the AI field continues its explosive growth, we must ensure our evaluation processes keep pace, providing transparent and accurate assessments to guide future development.