{"id":173264,"date":"2024-11-12T13:33:12","date_gmt":"2024-11-12T13:33:12","guid":{"rendered":"https:\/\/news.talkwithrattan.com\/index.php\/2024\/11\/12\/gpt-4-and-gemini-scored-less-than-2-percent-on-this-new-ai-benchmark\/"},"modified":"2024-11-12T13:33:12","modified_gmt":"2024-11-12T13:33:12","slug":"gpt-4-and-gemini-scored-less-than-2-percent-on-this-new-ai-benchmark","status":"publish","type":"post","link":"https:\/\/news.talkwithrattan.com\/index.php\/2024\/11\/12\/gpt-4-and-gemini-scored-less-than-2-percent-on-this-new-ai-benchmark\/","title":{"rendered":"GPT-4 and Gemini Scored Less Than 2 Percent on This New AI Benchmark"},"content":{"rendered":"
\"GPT-4<\/div>


\n<\/p>\n

\n
\n

Epoch AI, a California-based research institute launched a new artificial intelligence<\/a> (AI) benchmark last week. Dubbed FrontierMath, the new AI benchmark tests large language models (LLMs) on their capability of reseasoning and mathematical problem-solving. The AI firm claims that existing math benchmarks are not very useful due to factors like data contamination and AI models scoring very high scores on them. Epoch AI claims that even the leading LLMs have scored less than two percent on the new benchmark.<\/p>\n

Epoch AI Launches FrontierMath Benchmark<\/h2>\n

In a post<\/a> on X (formerly known as Twitter), the AI firm explained that it collaborated with more than 60 mathematicians to create hundreds of origins and unpublished math problems. Epoch AI claims that these questions would take even mathematicians hours to solve. The reason behind developing the new benchmark was cited as the limitations with existing benchmarks such as GSM8K and MATH, where AI models generally score a high point.<\/p>\n

The company claimed that the high scores achieved by LLMs are largely due to data contamination. This means the questions somehow were already fed into the AI models, resulting in them easily solving the questions.<\/p>\n

FrontierMath solves the problem by including new problems that are unique and have not been published anywhere, mitigating the risks associated with data contamination. Further, the benchmark includes a wide range of questions including computationally intensive problems in number theory, real analysis, and algebraic geometry, as well as topics such as Zermelo\u2013Fraenkel set theory. The AI firm says all the questions are \u201cguess proof\u201d, meaning they cannot be solved accidentally without strong reasoning.<\/p>\n

Epoch AI highlighted that to measure AI’s aptitude, benchmarks should be created on creative problem-solving where the AI has to maintain reasoning over multiple steps. Notably, many industry veterans believe that the existing benchmarks are not sufficient to correctly measure how advanced an AI model is.<\/p>\n

Responding to the new benchmark in a post<\/a>, Noam Brown, an OpenAI researcher who was behind the company’s o1 model welcomed the new benchmark and said, \u201cI love seeing a new eval with such low pass rates for frontier models.\u201d<\/p>\n<\/div>\n

\n For the latest tech news<\/a> and reviews<\/a>, follow Gadgets 360 on X<\/a>, Facebook<\/a>, WhatsApp<\/a>, Threads<\/a> and Google News<\/a>. For the latest videos on gadgets and tech, subscribe to our YouTube channel<\/a>. If you want to know everything about top influencers, follow our in-house Who’sThat360<\/a> on Instagram<\/a> and YouTube<\/a>. <\/p>\n

\n
\n
\n <\/p>\n
<\/div>\n

Poco X7 Pro Could Be the First Smartphone to Ship With Xiaomi’s HyperOS 2 in India<\/span>
\n <\/a>\n <\/div>\n

\n
\n
\n iQOO 13 Colour Options Revealed Ahead of Launch in India on December 3<\/span><\/p>\n
\n \"\"\n <\/div>\n

<\/a>\n <\/div>\n<\/div><\/div>\n