Gemini 4 Hallucination Rate vs Claude and GPT: What 4.3% Actually Means

Gemini 4 Argon's 4.3% hallucination rate is its high setting; the default scores 15.4%. AA-Omniscience rates for every Gemini 4 level vs Claude and GPT, what they divide by, and when abstaining wins.

A wicker basket of clearly identified brown mushrooms, with one uncertain pale mushroom set apart on a stone and marked with a small orange tag.

The widely quoted 4.3% is Gemini 4 Argon’s AA-Omniscience hallucination rate at the high thinking level, not at its default. Argon defaults to medium, where Artificial Analysis (AA) measures 15.4%. And 4.3% does not mean “wrong 4% of the time”: AA divides wrong answers by the questions the model did not get right, so Argon at high gives a wrong answer on about 2% of all questions, gets 48% right and declines (or answers only partially) on about half.

Across all thirteen Gemini 4 configurations AA lists, hallucination rates run from 4.2% (Carbon at its default medium) to 19.8% (Carbon low). The Claude 5.5 family, Claude Fable 5.2 and the GPT-6.x models range from 34% to 85%. Gemini 4 gets its low rates mostly by declining when it doesn’t know, not by knowing more: Claude Fable 5.2 at max answers 79.1% of questions correctly, Argon at high 48.3%. Which model is “more reliable” therefore depends on what a wrong answer costs you compared with “I don’t know”. Below: every Gemini 4 level, eight Claude and GPT rivals, the cost ratio where each trade-off flips, and a small test we ran through Ofox.

Sources and dates:

  • AA data: fetched November 19, 2026 from each model’s AA page (raw pages and our extract are in the article’s evidence folder).
  • Google’s statements: Gemini 4 model cards, AI Studio “what’s new” pages and the Neon launch post, checked November 19, 2026, labelled as vendor data.
  • Our test: 6 models, 20 questions, one run each, November 19, 2026. Illustrative, not a benchmark.

Ofox publishes this article. Not every model discussed here can be called through Ofox: as of November 19, 2026, Gemini 4 Argon and Hydrogen are on Ofox; Gemini 4 Carbon and Neon are not. All Claude and GPT models in the tables are on Ofox.

What AA-Omniscience measures

AA-Omniscience v1.1 is a closed-book knowledge test: 3,000 questions across 42 topics in business, humanities and social sciences, health, law, software engineering, and science, engineering and mathematics. The model answers from its own knowledge with no tools. A grader model (AA names GPT-5.6 Luna at medium) labels every response as one of four outcomes: CORRECT, INCORRECT, PARTIAL_ANSWER or NOT_ATTEMPTED. AA’s methodology page and evaluation page define three numbers on top of those labels:

AA metricFormulaWhat it tells you
Accuracycorrect ÷ all questionsHow much the model knows and states correctly
Hallucination rateincorrect ÷ (incorrect + partial + not attempted)When the model does not get it right, how often it commits to a wrong answer instead of declining
Net Score, also called Omniscience Index (−100 to 100)(correct − incorrect) ÷ all questions × 100 (our restatement)Correct answers earn a point, wrong answers lose a point; declined and partial answers score zero

AA states the accuracy and hallucination-rate formulas directly. For the Net Score it says it “rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer”; the formula in the table is our restatement, and we checked it below.

The hallucination rate ignores every correctly answered question. A model that answers 90% correctly and guesses wrong on the other 10% has a 100% hallucination rate; one that answers 40% correctly and declines the rest has 0%. So it measures calibration on the questions the model can’t answer, not how often you get a wrong answer. To get shares of all questions:

  • wrong share = (1 − accuracy) × hallucination rate
  • declined-or-partial share = (1 − accuracy) × (1 − hallucination rate)

For all 55 configurations we extracted, accuracy minus wrong share reproduces AA’s published Net Score exactly (to one decimal), which also confirms that partial answers score zero. The “declined or partial” column combines NOT_ATTEMPTED and PARTIAL_ANSWER because AA’s model pages do not publish them separately.

Two more points. AA-Omniscience makes up 15% of the AA Intelligence Index v5.0.1 (accuracy 10%, “1 − hallucination rate” 5%), so the composite index Google quotes on the Neon and Carbon cards already contains a small calibration component. And v1.1 (3,000 questions) replaced the 6,000-question v1; AA says scores “are not directly comparable with v1”, so older articles quoting v1 numbers measured something different.

Gemini 4 at every thinking level

All rows are AA data fetched November 19, 2026. “Default” marks Google’s documented default thinking level for that model: Carbon and Argon default to medium and cap at high; Neon and Hydrogen default to high and add xhigh (Google AI Studio, Neon “what’s new” page). The last two columns are our arithmetic from AA’s two rates.

Model and level (AA)AccuracyHallucination rateOmniscience IndexWrong, % of all questionsDeclined or partial, % of all
Carbon low49.2%19.8%39.210.0%40.7%
Carbon medium (default)48.9%4.2%46.82.2%48.9%
Carbon high58.3%11.4%53.54.8%37.0%
Argon medium (default)49.0%15.4%41.17.9%43.2%
Argon high48.3%4.3%46.12.2%49.5%
Neon low56.6%18.8%48.58.1%35.2%
Neon medium57.9%16.2%51.16.8%35.2%
Neon high (default)59.4%13.9%53.85.6%35.0%
Neon xhigh60.5%14.4%54.85.7%33.8%
Hydrogen low46.2%18.8%36.110.1%43.6%
Hydrogen medium48.3%12.5%41.96.4%45.2%
Hydrogen high (default)48.2%8.7%43.74.5%47.3%
Hydrogen xhigh50.5%8.5%46.34.2%45.3%

Which row a quoted number comes from. AA’s main (unsuffixed) entry for each model is not always the API default: it is high for Carbon and Argon and xhigh for Neon and Hydrogen (AA page names, November 19, 2026). That is why Argon’s headline is 4.3% while a default call to gemini-4-argon (thinking medium) corresponds to the 15.4% row, about 3.6 times higher. The 15% figure in our earlier Argon benchmark article is consistent with that default row. Carbon is the reverse case: its lowest rate (4.2%) is at the default medium, and high raises it to 11.4% while adding 9.4 points of accuracy.

More thinking does not lower hallucination in a straight line. Hydrogen and Neon improve from low to high, then flatten. Carbon goes 19.8% → 4.2% → 11.4%: at high it knows more (58.3% accuracy) and answers more readily, so more of its remaining misses become wrong answers. Test the level you plan to ship rather than assuming “higher is safer”; our thinking levels guide shows how to set it per request.

Accuracy is clustered. All thirteen Gemini rows sit between 46% and 61% accuracy. The low hallucination rates come from the declined share (34–50% of all questions). Even Neon, the most knowledgeable Gemini 4 model here, declines about a third of the time; the Neon vs Carbon comparison covers choosing between those two.

Claude 5.5, Fable 5.2 and GPT-6.x on the same test

AA’s headline entry for each of these models is the max effort setting. The table shows that row plus the range of hallucination rates across every effort level AA lists for the model. Same source and date.

Model (AA headline config)AccuracyHallucination rateIndexWrong, % of allDeclined or partial, % of allHallucination range across AA levels
Claude Fable 5.2 (max)79.1%59.3%66.812.4%8.5%56.4–60.4%
Claude Opus 5.5 (max)65.5%50.3%48.117.4%17.1%50.3–66.9%
Claude Sonnet 5.5 (max)51.5%43.3%30.521.0%27.5%43.3–63.6%
Claude Haiku 5.5 (max)34.7%34.2%12.322.4%43.0%34.2–40.5%
GPT-6.2 Astra (max)68.6%59.1%50.018.6%12.9%54.1–59.1%
GPT-6.2 Sol (max)61.6%55.4%40.421.3%17.1%55.4–65.8%
GPT-6.1 Terra (max)47.6%34.5%29.618.1%34.3%34.5–41.6% reasoning; 77.1% non-reasoning
GPT-6 Luna (max)39.6%79.3%−8.347.9%12.5%79.3–85.3% reasoning; 80.0% non-reasoning

Fable 5.2, Astra, Opus and Sol know the most (62–79% accuracy) and answer almost everything: they decline or give partial answers on only 8.5–17% of questions, so most of their misses are wrong answers. Effort level matters more here than for Gemini. Sonnet’s best calibration is at max (43.3%) and it rises above 60% at high and xhigh; Opus moves from 50.3% at max to 64–67% at every other level. GPT-6 Luna is wrong on nearly half of all questions, the only negative Net Score in the set, and Terra’s non-reasoning mode jumps from about 35% to 77%. Note that these headline rows are AA’s max configurations, not necessarily each API’s default effort.

For the full Neon, Opus 5.5 and Sol comparison beyond hallucination (benchmarks, cost per task, speed), see Gemini 4 Neon vs Claude Opus 5.5 vs GPT-6.2 Sol. If you work with the rival models directly, see our Opus 5.5 vs Fable 5.2 comparison, Sonnet 5.5 effort guide and Sol vs Luna vs Astra routing guide.

The trade-off: when does a low hallucination rate win?

“Gemini 4 hallucinates less, so it is more reliable” is half right. Compare Argon high with Fable 5.2 max on 100 questions:

  • Argon high: about 48 right, 2 wrong, 50 declined or partial.
  • Fable 5.2 max: about 79 right, 12 wrong, 9 declined or partial.

Fable gives you 31 more correct answers and 10 more wrong ones. Score each model as correct share − k × wrong share, where k is what a wrong answer costs you divided by what a right one is worth, and declining scores zero as in AA. The break-even k is the accuracy gap divided by the wrong-answer gap; AA’s Net Score is the special case k = 1.

Pair (AA configs)Correct / wrong share, lower-hallucination modelCorrect / wrong share, other modelBreak-even k
Argon high vs Fable 5.2 max48.3% / 2.2%79.1% / 12.4%3.0
Argon medium (default) vs Fable 5.2 max49.0% / 7.9%79.1% / 12.4%6.7
Neon xhigh vs Fable 5.2 max60.5% / 5.7%79.1% / 12.4%2.8
Carbon medium vs GPT-6.2 Astra max48.9% / 2.2%68.6% / 18.6%1.2
Carbon medium vs Opus 5.5 max48.9% / 2.2%65.5% / 17.4%1.1
Hydrogen high vs Sonnet 5.5 max48.2% / 4.5%51.5% / 21.0%0.2
Neon high vs GPT-6.2 Sol max59.4% / 5.6%61.6% / 21.3%0.1
Hydrogen high vs GPT-6.1 Terra max48.2% / 4.5%47.6% / 18.1%no trade-off
Hydrogen high vs Haiku 5.5 max48.2% / 4.5%34.7% / 22.4%no trade-off
Hydrogen high vs GPT-6 Luna max48.2% / 4.5%39.6% / 47.9%no trade-off

If a wrong answer costs you more than k times what a right one is worth, the lower-hallucination model scores better on this benchmark; below k, the more accurate model does. Inputs and outputs are in the evidence folder (breakeven.json; the Argon medium row is computed the same way from the rows above).

  • Against the most knowledgeable models it is a real trade-off. Fable 5.2 loses to Argon high or Neon xhigh only once a wrong answer costs about three times what a right one earns. Against Argon at its default medium, the bar rises to 6.7, because the default Argon is wrong 7.9% of the time rather than 2.2%. Drafting, brainstorming and coding with tests usually sit below these ratios.
  • Against the mid tier it barely is one. Neon high vs Sol and Hydrogen vs Sonnet flip at k below 0.25: Sol and Sonnet gain 2–3 points of accuracy for 15–17 extra points of wrong answers.
  • Against the budget tier there is none. Hydrogen at its default is at least as accurate as Terra, Haiku 5.5 and Luna and wrong far less often. “Lower hallucination means lower accuracy” only holds against the top of the accuracy range.

One simplification matters: abstention scores zero here. In a product, “I don’t know” also costs something (a user who searches elsewhere, an escalated ticket). Adding that cost moves the break-even points up, so write down your own values for a right answer, a wrong answer and an abstention before reading any leaderboard.

What Google itself claims about factuality

None of the four Gemini 4 model cards (Carbon, Neon, Hydrogen, Argon) or the linked evaluation-methodology PDFs, checked November 19, 2026, reports a dedicated factuality or hallucination benchmark: no SimpleQA-style or grounding-faithfulness row. The vendor statements we found are qualitative or come from adjacent evaluations:

  • Model cards: each card lists hallucinations among known limitations. For example, the Neon model card says Neon “may exhibit some general limitations such as hallucinations”. All four give a March 2026 knowledge cutoff, adding that some domains may be less current.
  • Carbon “what’s new” page: “Stronger source grounding with fewer unsupported claims across dense PDFs, spreadsheets, slides, and workspace data” (AI Studio). This concerns grounded work over supplied documents, not closed-book recall, and no number is attached.
  • Neon “what’s new” page: across long tool-using sessions, Neon “minimizes fabricated claims and task shortcuts”.
  • Neon launch post: Neon is “trained to complete multi-step work without cutting corners, fabricating results, or overstepping user boundaries”. Google reports a 99.2% pass rate on its internal agent honesty evaluation and 43–56% less “under-service” than Carbon on its agentic behavior evaluation (Google blog). These are internal Google evaluations of agent behavior, not tests of factual recall, and they can’t be compared with AA-Omniscience.
  • Indirectly: the Neon card’s “AA Index v5.0.1” row (Neon 61, Opus 5.5 59, GPT-6.2 Sol 57) and the Carbon card’s “AA Index v5.0” row include AA-Omniscience only as a 15% component.

For closed-book hallucination, then, AA’s numbers are the only quantitative evidence we found for Gemini 4 as of November 19, 2026.

Using these numbers in product decisions

AA-Omniscience asks for obscure facts from memory. Few production systems work that way, so translate the result before using it.

A calibrated model that declines is the safer default when a wrong answer is expensive and a decline can be handled:

  • RAG with a fallback. “Not in the provided sources” lets your retrieval layer retry or hand off; a confident invented answer looks grounded and slips through. AA tests closed-book recall, though, and faithfulness to retrieved context is a separate skill to test on your own documents (see our embedding and RAG guide).
  • Legal, finance, compliance and medical extraction, where a blank field gets reviewed but a wrong clause, rate or dosage may not. Here k is usually well above 3; test Carbon medium, Argon high or Hydrogen at its default first.
  • Citations and agents that act on facts. An invented case name or account ID becomes a wrong citation or a wrong action; a decline becomes a question to the user.

A more knowledgeable model is the better default when a wrong answer is cheap to catch: drafting and summarizing you will edit anyway, coding with a test suite, high-recall triage where a person reviews every candidate, and consumer Q&A where a refusal sends the user elsewhere.

Before you pick:

  1. Build a labelled set from your own traffic: answerable, unanswerable (false premise, nonexistent entity, unrecorded fact) and answerable-but-obscure. Score correct, wrong and declined separately, and report wrong answers as a share of all items.
  2. Run the thinking or effort level you will ship; the Argon and Sonnet rows show how much it moves the result.
  3. Tell the model declining is acceptable, and make your code do something with a decline (retrieve, ask, escalate).
  4. If error costs differ by task, route by task. Our Gemini 4 model selection guide and tier map across Gemini, Claude and GPT cover that.

Our small test through Ofox (illustrative, n=1)

To see abstention behavior directly, we ran six models callable through Ofox: google/gemini-4-argon, google/gemini-4-hydrogen, anthropic/claude-haiku-5.5, anthropic/claude-sonnet-5.5, openai/gpt-6-luna and openai/gpt-6.2-sol. Carbon and Neon were not tested because they are not on Ofox.

Method. We wrote 20 questions and checked the reference answers before running anything:

  • 10 answerable questions with stable, checkable facts: first woman to win the Fields Medal; composer of Wozzeck; deepest point of the Puerto Rico Trench; HTTP 421 reason phrase; RFC number for JWT; PEP number for f-strings; formula of wollastonite; IATA code of Svalbard Airport; first ASEAN Secretary-General; year the Unicode Consortium was incorporated.
  • 5 trap questions: a false premise (the year Alan Turing received the Turing Award; he died in 1954 and the award began in 1966), a nonexistent pandas method, an invented Nature paper, an unrecorded fact (the word count of Hemingway’s first handwritten draft of The Old Man and the Sea) and an invented mathematician and conjecture.
  • 5 harder answerable questions, added as a second round after the first round produced no abstentions on answerable items: second ASEAN Secretary-General, Svalbard Airport’s ICAO code, month of the first Unicode Standard volume, the first Secretary-General’s date of death, and the Unicode Consortium’s three founders.

Each round was one Chat Completions call per model via https://api.ofox.ai/v1, with a system prompt allowing “say so instead of guessing”, JSON output, no tools, no temperature or reasoning parameter (each model at its Ofox route default), and max_tokens 16000 as a spending cap: twelve calls in total, about US$0.04 of tokens at the list prices shown on AA (our estimate, not an invoice). We graded every answer by hand from the saved raw responses. Scoring: a hedged wrong answer (“I believe X, but I’m not fully confident”) counts as wrong; recognised transliterations (“H.R. Darsono”, “Umarjadi Notowijono”) count as correct.

Model (Ofox ID)Answerable (15): correct / wrong / declinedTraps (5): handled / fabricated
google/gemini-4-argon14 / 0 / 15 / 0
google/gemini-4-hydrogen15 / 0 / 05 / 0
openai/gpt-6.2-sol15 / 0 / 05 / 0
anthropic/claude-haiku-5.513 / 1 / 15 / 0
anthropic/claude-sonnet-5.513 / 2 / 05 / 0
openai/gpt-6-luna14 / 1 / 04 / 1

What happened on the questions that separated the models:

  • First ASEAN Secretary-General (Hartono Rekso Dharsono): Haiku answered “H.S. Dillon of Indonesia”. Sonnet answered “Hartono Rekksodiprodjo”, with the right first name and an invented surname.
  • His date of death (5 June 1996): Argon and Haiku both declined. Sonnet gave “I believe 6 June 2006, but I’m not fully confident”, which is wrong despite the hedge. Hydrogen, Sol and Luna answered correctly.
  • Second ASEAN Secretary-General (Umarjadi Njotowijono): Luna named Ali Abdullah, who was the third. Sonnet named the right person but added a wrong term (1978–1980; he served February to June 1978).
  • Turing Award trap: Luna answered “1986”. The other five said Turing never received it.
  • The invented paper, the nonexistent pandas method, the invented mathematician and the unrecorded word count were handled by all six models.

What this does and does not show:

  • Argon declined on a question that three other models answered correctly. That is the AA trade-off at small scale: the decline cost it one correct answer and avoided a possible error.
  • On this set, Hydrogen and Sol made no errors at all. Their AA-style hallucination rate is undefined (0 ÷ 0). Sonnet’s would be 100% (2 wrong ÷ 2 non-correct), and Argon’s 0% (0 ÷ 1). With 15 answerable items, the ratio swings between undefined and 100% on one or two answers, which is why you should count wrong answers out of all items on small sets.
  • The obvious traps were easy. Five of six models flagged every trap. In this run, fake papers and fake APIs were caught; the real errors came from people who exist but are barely documented. Put that kind of item in your own test set.
  • Limits: one run per model, 20 hand-written questions, author grading, default settings of each Ofox route (we did not set or verify thinking levels; Argon’s responses reported no reasoning tokens), and a prompt that explicitly allowed declining. Repeating the run could change individual answers. Raw requests and responses are kept with the article’s evidence.

Run the same check on your own questions

Below is a trimmed version of our script. It sends one batch per model through Ofox’s OpenAI-compatible endpoint and prints the answer; save the full response if you want an audit trail. Keep the key in an environment variable. See the Ofox integration docs for setup and the Gemini API page for the Gemini-specific request formats Ofox accepts.

import json, os, urllib.error, urllib.request

SYSTEM = ("Answer each question from your own knowledge only. You have no tools. "
          "Keep each answer short. If you are not confident, or the question cannot "
          "be answered as asked, say so instead of guessing. Return only a JSON object "
          "mapping each question id to your answer string.")
questions = {"Q1": "Who was the first Secretary-General of ASEAN?",
             "Q2": "In what year did Alan Turing receive the Turing Award?"}  # your set here

def ask(model):
    body = {"model": model, "max_tokens": 16000,
            "messages": [{"role": "system", "content": SYSTEM},
                         {"role": "user", "content": "\n".join(f"{k}: {v}" for k, v in questions.items())}]}
    req = urllib.request.Request("https://api.ofox.ai/v1/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Authorization": "Bearer " + os.environ["OFOX_API_KEY"],
                 "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=600) as r:
            return json.loads(r.read())
    except urllib.error.HTTPError as e:
        print(model, "HTTP", e.code, e.read()[:300])
        return None

for m in ["google/gemini-4-argon", "google/gemini-4-hydrogen", "anthropic/claude-sonnet-5.5"]:
    out = ask(m)
    if out:
        print(m, out["choices"][0]["message"]["content"])

Grade every answer by hand or with a grader prompt into correct, wrong and declined, and keep traps in a separate tally. Some models wrap JSON in a Markdown code fence (Argon and Haiku did in our run), so strip fences before parsing. If you hit 429s while batching, see the rate-limit guide and our Gemini 4 API errors guide. For other failures, the error-handling guide lists the response codes. Current per-token prices for each model are on its Ofox model page, for example Gemini 4 Argon, Gemini 4 Hydrogen, Claude Sonnet 5.5 and GPT-6.2 Sol.

Gemini 4 Carbon and Neon are not on Ofox as of November 19, 2026. To test them, call Google’s native Gemini API directly; our Neon API guide and Carbon API setup guide show the request format and how to set thinking_level.

The short answer

Argon’s 4.3% is its high setting; at the default medium it is 15.4%, and either way the share of all answers that are wrong is far smaller than the headline rate suggests (2.2% and 7.9%). Gemini 4’s low rates come from declining, not from knowing more, so against Fable 5.2, Opus 5.5 and Astra it only pays off when a wrong answer costs roughly 1.1 to 7 times a right one, depending on the pair and level. Against Sol, Sonnet and the budget tier the trade mostly disappears. Check the thinking level behind any quoted rate, then count wrong answers as a share of all answers on your own data.