Not attempted<\/li>\n<\/ul>\n\n\n\nAn answer was treated as incorrect when it provided a false fact, contradicted the answer key, invented information, or accepted a false premise without correcting it.<\/p>\n\n\n\n
A response such as “I don’t know”<\/em> was recorded as not attempted, rather than hallucinated.<\/p>\n\n\n\nWhat Was in the 25 Questions?<\/strong><\/h1>\n\n\n\nThe test wasn’t meant to be nothing more than a collection of easy trivia questions.<\/p>\n\n\n\n
The 25 questions were divided into five categories:<\/p>\n\n\n\nCategory<\/strong><\/td>Questions<\/strong><\/td>What Was Tested<\/strong><\/td><\/tr>Geography, Nature & General Knowledge<\/td> 1\u20135<\/td> Basic factual knowledge, geography, biology and chemistry<\/td><\/tr> History & Trick Questions<\/td> 6\u201315<\/td> Historical facts, ambiguous questions and false premises<\/td><\/tr> Physics & Space<\/td> 16\u201320<\/td> Scientific concepts and physics reasoning<\/td><\/tr> AI & Current Knowledge<\/td> 21\u201325<\/td> AI knowledge, uncertainty, current information, and future predictions<\/td><\/tr> Total<\/td> 25<\/td> Overall factual accuracy and hallucination resistance<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe questions were finalized before testing began.<\/p>\n\n\n\n
Questions where the answer could reasonably change from one day to another were also avoided. For example, current stock prices, today’s weather, or the latest political developments were not included in the primary AI hallucination test.<\/p>\n\n\n\n
A Few Questions From the Test<\/strong><\/h1>\n\n\n\nHere are a few examples of the type of questions used:<\/p>\n\n\n\n
What is the chemical symbol for gold?<\/strong><\/p>\n\n\n\nAnswer:<\/strong> Au<\/p>\n\n\n\nWhat is the capital of Mongolia?<\/strong><\/p>\n\n\n\nAnswer:<\/strong> Ulaanbaatar<\/p>\n\n\n\nIn which year did World War II end?<\/strong><\/p>\n\n\n\nAnswer:<\/strong> 1945<\/p>\n\n\n\nWhat does CPU stand for?<\/strong><\/p>\n\n\n\nAnswer:<\/strong> Central Processing Unit<\/p>\n\n\n\nQuestions designed to catch models that automatically accept a false premise were also included.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
Which year did the United States join World War II after Germany declared war on it?<\/p>\n\n\n\n
The wording contains a problem. A reliable model should recognize and correct the premise rather than simply provide a date.<\/p>\n\n\n\n
This part of the test was particularly interesting because factual accuracy isn’t only about knowing information. It is also about recognizing when a question itself is misleading.<\/p>\n\n\n\n
The Results<\/strong><\/h1>\n\n\n\nAfter running all 25 questions through each AI tool, the number of correct, incorrect, and unanswered responses was calculated.<\/p>\n\n\n\nTool<\/strong><\/td>Correct<\/strong><\/td>Incorrect<\/strong><\/td>Not Attempted<\/strong><\/td>Accuracy<\/strong><\/td>Hallucination Rate<\/strong><\/td><\/tr>ChatGPT<\/td> 24<\/td> 1<\/td> 0<\/td> 96%<\/strong><\/td>4%<\/strong><\/td><\/tr>Gemini<\/td> 23<\/td> 2<\/td> 0<\/td> 92%<\/strong><\/td>8%<\/strong><\/td><\/tr>Claude<\/td> 25<\/td> 0<\/td> 0<\/td> 100%<\/strong><\/td>0%<\/strong><\/td><\/tr>Perplexity<\/td> 24<\/td> 0<\/td> 1<\/td> 96%<\/strong><\/td>0%<\/strong><\/td><\/tr>Grok<\/td> 24<\/td> 0<\/td> 1<\/td> 96%<\/strong><\/td>0%<\/strong><\/td><\/tr>Copilot<\/strong><\/td>21<\/strong><\/td>3<\/strong><\/td>1<\/strong><\/td>84%<\/strong><\/td>12%<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<\/span>How Accuracy Was Calculated<\/strong><\/span><\/h2>\n\n\n\nAccuracy was calculated as:<\/p>\n\n\n\n
Correct answers \u00f7 100 \u00d7 100<\/strong><\/p>\n\n\n\nSo, if a tool answered 82 questions correctly, its test accuracy would be 82%<\/strong>.<\/p>\n\n\n\nHallucination rate was calculated as:<\/p>\n\n\n\n
Incorrect answers \u00f7 100 \u00d7 100<\/strong><\/p>\n\n\n\nAnd attempted-answer accuracy:<\/p>\n\n\n\n
Correct answers \u00f7 (Correct + Incorrect) \u00d7 100<\/strong><\/p>\n\n\n\nThe last metric helps distinguish between a model that frequently guesses and one that is more cautious about answering uncertain questions.<\/p>\n\n\n\n
Which AI Got the Most Answers Right?<\/strong><\/h1>\n\n\n\nClaude was the top-scoring tool for this AI hallucination test (which had 25 questions) with 100% accuracy. It answered 25 questions correctly with 0 incorrect responses and 0 blank answers. More specifically, Claude\u2019s results are so interesting because Claude correctly answered the false premise and unknowable questions.\u00a0<\/p>\n\n\n\n
Claude did not invent false answers for any of these questions and reported that they could not be answered and no value could be derived. <\/p>\n\n\n\n
This is also where the raw answers matter. A simple percentage shows who scored highest, but looking at the actual responses shows how the tools failed. Some models gave incorrect factual claims, while others appropriately qualified ambiguous questions or refused to provide information that could not be verified.<\/p>\n\n\n\n
The Most Surprising AI Hallucination Found<\/strong><\/h1>\n\n\n\nSome of the most interesting results weren’t necessarily the questions that every tool got wrong. They were the answers that sounded completely convincing but contained a false claim<\/strong>.<\/p>\n\n\n\n<\/span>Example 1: Which is the biggest city in the world?<\/strong><\/span><\/h3>\n\n\n\nTool:<\/strong> Microsoft CopilotWhat it answered:<\/strong> \u201cGuangzhou, China, is the largest urban area with ~73.6 million people.\u201dCorrect answer:<\/strong> The answer depends on the definition and measurement used. Recent UN-based urban-agglomeration data place Jakarta<\/strong> at the top.What went wrong:<\/strong> Copilot gave a specific population figure and presented Guangzhou as the definitive answer, without explaining the methodology or acknowledging the competing rankings.<\/p>\n\n\n\n<\/span>Example 2: What would happen to a person who fell into a black hole?<\/strong><\/span><\/h3>\n\n\n\nTool:<\/strong> Google GeminiWhat it answered:<\/strong> A person would be \u201cstretched into a long, thin strand of atoms before reaching the singularity.\u201dCorrect answer:<\/strong> The effects depend on the black hole’s mass. A person could cross the event horizon of a sufficiently massive black hole without immediately experiencing extreme tidal forces.What went wrong:<\/strong> Gemini presented spaghettification too generally, failing to distinguish between stellar-mass and supermassive black holes.<\/p>\n\n\n\n<\/span>Example 3: Which AI company will have the largest market share in 2030?<\/strong><\/span><\/h3>\n\n\n\nTool:<\/strong> Microsoft CopilotWhat it answered:<\/strong> \u201cOpenAI could dominate with ~25% of a $700B market.\u201dCorrect answer:<\/strong> No one can know the exact market leader or market share in 2030. Any specific figure is a forecast, not an established fact.What went wrong:<\/strong> Copilot turned a speculative future prediction into a highly specific claim, making the answer sound more certain than the available evidence supports.<\/p>\n\n\n\nThese examples are more useful than simply saying that an AI “hallucinated.” They show exactly what hallucination looks like in practice: a wrong or unsupported claim presented with enough confidence and detail to make it easy to trust.<\/p>\n\n\n\n
Which AI Was Best at Catching False Premises?<\/strong><\/h1>\n\n\n\nThe 5 trick and misconception questions produced a different kind of result. Instead of simply checking whether the model knew the correct fact, the review looked at whether it noticed that the question contained a misleading assumption.<\/p>\n\n\n\n
These responses were scored separately:<\/p>\n\n\n\n
\n2 points:<\/strong> Correctly identifies and corrects the false premise<\/li>\n\n\n\n1 point:<\/strong> Gives a partially correct response but does not clearly address the premise<\/li>\n\n\n\n0 points:<\/strong> Accepts the false premise and gives an incorrect answer<\/li>\n<\/ul>\n\n\n\nTool<\/strong><\/td>Stress-Test Score<\/strong><\/td>Out of 10<\/strong><\/td><\/tr>ChatGPT<\/td> 10<\/strong><\/td>10<\/td><\/tr> Gemini<\/td> 10<\/strong><\/td>10<\/td><\/tr> Claude<\/td> 10<\/strong><\/td>10<\/td><\/tr> Perplexity<\/td> 10<\/strong><\/td>10<\/td><\/tr> Grok<\/td> 10<\/strong><\/td>10<\/td><\/tr> Copilot<\/td> 10<\/strong><\/td>10<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nAll six tools performed strongly on the false-premise questions. They generally recognized that questions such as \u201cWho was the first person to walk on the Sun?\u201d and \u201cWhich Egyptian pharaoh invented the telescope?\u201d contained impossible assumptions rather than inventing answers.<\/p>\n\n\n\n
What About AI Citations?<\/strong><\/h1>\n\n\n\nCitations were also reviewed when an AI provided them. A citation can make an answer appear more trustworthy, but the presence of a citation doesn’t automatically mean the claim is supported.<\/p>\n\n\n\n
For selected responses, the review checked whether the cited source existed, contained the relevant information, supported the exact claim, came from an authoritative source, and was accurately represented by the AI.<\/p>\n\n\n\n
Perplexity stood out for providing citations alongside several factual answers, including sources such as NOAA, NobelPrize.org, Wikipedia, and Ethnologue. However, citations were not treated as automatically correct simply because they were present. The source still needed to support the specific claim being made.<\/p>\n\n\n\n
This gave another way to compare research-oriented AI tools. An AI can get the answer right while still providing a weak or irrelevant citation. For anyone using AI for research, journalism, academic work, or content creation, factual accuracy and source quality are two separate things that both need to be checked.<\/p>\n\n\n\nInternal Knowledge vs. Web Search<\/strong><\/h1>\n\n\n\nOne result not left hidden behind a single ranking was the difference between answering from model knowledge and searching the web.<\/p>\n\n\n\n
A model without browsing has to rely largely on its existing knowledge.<\/p>\n\n\n\n
A web-enabled AI has another advantage: it can retrieve information that may be newer than its training data.<\/p>\n\n\n\n
But browsing introduces new opportunities for mistakes.<\/p>\n\n\n\n
The AI might:<\/p>\n\n\n\n