Cheapest LLM That Still Gets It Right: 86 Billed API Calls

On the everyday tasks we tested, the cheapest model matched every answer the most expensive one did, at 1/75 of the billed cost on meeting notes and 1/45 on a bug fix. On harder work, a low price per token stopped predicting a low price per result: DeepSeek V4.1 Flash and Qwen 3.8 Flash sometimes spent their whole output budget thinking, returned nothing, and were billed for it. So compare what a correct result costs, and check it before you trust it.

This is the written companion to our video walkthrough, presented by Marie, an AI-generated presenter. The ofox team designed and ran every test between October 20 and 21, 2026, reconciled each call against the ofox bill, and published the scripts and raw results below.

Watch on YouTube: How to Cut AI API Costs Without Losing Accuracy (7 Models Tested) (5 min).

Who this is for: developers and small teams calling LLM APIs who want to spend less without guessing. It does not measure subscription quota in Codex, ChatGPT or Claude apps, because those plans do not publish how usage converts into quota.

Why the price per token is the wrong number

Three things sit between a model’s rate card and your bill.

1. Task difficulty decides whether a cheap model is enough. If it fails, you pay again for a stronger model.
2. Reasoning models bill thinking as output. The reasoning_tokens inside completion_tokens are charged even when you never see them.
3. Failures are billed too. An empty or wrong answer costs the same tokens as a good one.

So the comparable number is cost per correct result: the billed cost of the runs it took to get an answer that passed your check. We used the same idea for video in usable-clip cost, and our Luna-to-Sol routing guide sets out the decision rules with authored examples. This article adds measured, billed numbers.

How we tested

The 86 completed calls are the 83 in the main test plus three extra runs of GPT-6 Sol on low reasoning effort, which matched all seven items on the hard notes each time at $0.003582 per run. The 83 include 17 unscored GPT-6 Luna calls from a long-conversation experiment we left out of this article because its two arms were not equivalent tasks, and 6 GPT-6 Sol calls comparing low and high reasoning effort on the bug fix (both passed; high cost about 18% more). Two test records hit rate limits (HTTP 429). Claude models were reachable through the OpenAI-compatible endpoint on October 20. Since October 21, ofox serves them through the Anthropic-compatible endpoint (/anthropic/v1) instead.

Result 1: everyday tasks

Meeting notes to action items. The notes contain eight dated action items and five traps: ideas nobody owns, a decision without a date, a maintenance notice. Every model found all eight owner + due date pairs with nothing extra, in both runs.

Opus cost 75.2 times as much as Luna for the same matched result. Price did not predict speed either: Luna answered in about 5 seconds, while DeepSeek V4.1 Flash took 264.5 seconds on one run and 13.2 on the other.

Fixing a function with four bugs. The function merges overlapping room bookings and has to sort, merge touching ranges, keep the later end time and drop invalid bookings. Every model passed all eight hidden tests in both runs. Opus averaged $0.010086 per run and Luna $0.000225, a 44.8 times difference.

Result 2: a harder task, and the first real failures

Easy tasks do not separate models, so we wrote notes where people change their minds mid-meeting, dated Tuesday, October 20, 2026. One task is reassigned from Priya to Omar, a deadline moves from the 27th to the 31st, one task is scrapped, and dates are relative: “tomorrow”, “two business days from today”, “end of next week”. There are seven correct items. Output was capped at 4,000 tokens.

The DeepSeek and Qwen failures were not wrong answers. They were empty answers. In each one, completion_tokens and reasoning_tokens were both 4,000, the full cap, and content was an empty string.

Empty replies are billed

We checked the bill. The ofox Activity page lists the request from 3:07:37 PM on October 20: 4K output tokens, 4K of them reasoning, charged $0.00480307.

The screenshot is the real dashboard in English, filtered before recording to our own test requests. Team, balance and other members’ data are not shown. The two empty DeepSeek replies in this test were billed $0.004803072 and $0.0049536.

Fixing the reasoning-model trap

We reran DeepSeek V4.1 Flash and Qwen 3.8 Flash on the same hard notes with more room, or with lower reasoning effort. Three runs each.

With room to finish, DeepSeek got the task right every time, but each correct run cost $0.0059. GPT-6 Sol did the same task for $0.0040 with about 305 output tokens. On the ofox price list DeepSeek V4.1 Flash is $0.30 input and $1.20 output per million tokens, against $2.00 and $10.00 for GPT-6 Sol: about 1/7 of the input price and 1/8 of the output price. It still cost more per correct result, because about 96.7% of its output was thinking.

Lowering effort helped Qwen most here: all three runs matched, at $0.000682 each and a 17.5-second median. It did not fix DeepSeek, which still returned one empty reply on low effort. In these three-run tests, raising the cap was the setting that made DeepSeek pass every time.

Two practical rules follow:

• Set an output cap and a reasoning effort on purpose, and treat an empty content with finish_reason: "length" as a failure, not an answer.
• Compare billed cost per correct result, never price per million tokens alone. Our routing break-even calculator shows how retries and fallbacks change that number.

Verify, then move up

The usual advice is to start with a cheap model and switch to a better one when it fails. In practice “when it fails” needs a check you can run. For code that check is tests. The script below runs models cheapest first, runs the tests on each reply, and moves up only when they fail.

To show the safety net working, we needed a cheap model that fails. GPT-6 Luna never failed: it passed all 7 of its scored runs in the main test and 15 of 15 across five coding tasks we wrote for this (business days, duration parsing, bill splitting, cron schedules and daylight-saving times). GPT-6 Sol on low effort missed three cases in one of its 18 coding runs (duration parsing, 16 of 19). So the demo starts on DeepSeek V4.1 Flash, which returned empty replies on one of those tasks, a cron “next run time” function with twelve tests, in all three of its runs at a 4,000-token cap.

import json, os, re
from openai import OpenAI

client = OpenAI(base_url="https://api.ofox.ai/v1",
                api_key=os.environ["OFOX_API_KEY"])

# Cheapest first. $ per 1M tokens (in, out), ofox, Oct 2026
TIERS = [
    ("deepseek/deepseek-v4.1-flash", 0.30, 1.20, {}),
    ("openai/gpt-6-sol", 2.00, 10.00,
     {"reasoning_effort": "low"}),
]

def passes(reply):
    m = re.search(r"```(?:python)?\n(.*?)```", reply, re.S)
    if not m:
        return 0
    ns = {}
    exec(m.group(1), ns)
    tests = json.load(open("tests.json"))
    return sum(ns["next_run"](*a) == want for a, want in tests)

task, total = open("task.txt").read(), 0.0
for model, p_in, p_out, extra in TIERS:
    r = client.chat.completions.create(
        model=model, max_tokens=4000,
        messages=[{"role": "user", "content": task}], **extra)
    u = r.usage
    cost = (u.prompt_tokens * p_in + u.completion_tokens * p_out) / 1e6
    total += cost
    ok = passes(r.choices[0].message.content or "")
    print(f"{model}\n  {u.completion_tokens} out tokens  ${cost:.4f}  tests {ok}/12")
    if ok == 12:
        print(f"PASS  total ${total:.4f}")
        break
    print("  check failed -> moving up")

The real run on October 21, 2026:

deepseek/deepseek-v4.1-flash
  4000 out tokens  $0.0049  tests 0/12
  check failed -> moving up
openai/gpt-6-sol
  971 out tokens  $0.0101  tests 12/12
PASS  total $0.0149

The costs printed here are the token usage the API returned multiplied by the ofox list price. Look at the total. The failed DeepSeek try added about half a cent, so this run cost more than starting on Sol would have.

A check is not there to make you try the cheap model every time. It tells you where each kind of task should start. DeepSeek V4.1 Flash was the wrong first tier for this task. GPT-6 Luna, which is cheaper per token than either, passed the same cron task in all three of its runs (with a 6,000-token cap in that check), and it also matched every item on the meeting notes. From our runs, extraction like the meeting-notes task and these five coding tasks can start on Luna. Start higher only for tasks where your own check shows the cheap model failing, and keep the escalation path for those cases.

Note that exec runs model-written code in your process. That is acceptable for a demo on code you are about to review anyway. For anything automated, run the tests in a sandbox or subprocess, as our full test runner does.

How to apply this to your own work

You do not have an answer key in real work, so build a small one.

1. Pick three tasks you run every day and save five to ten real inputs for each.
2. Write the check before you choose a model. For code, a handful of tests. For extraction, the fields you expect, plus spot checks of a few items against the source. For summaries, a short list of facts that must appear.
3. Run the cheapest plausible model two or three times per input and record pass rate, output tokens and billed cost. Check the bill, not your estimate. Reasoning-heavy models can cost far more than their rate suggests.
4. Set each task’s starting model to the cheapest one that passes reliably in your runs. Use escalation for the failures, not as the default path.
5. Re-run the check when prices or model versions change.

Limits of this test

• The inputs are synthetic and short. The meeting notes are about a thousand tokens.
• Each setup ran two or three times. Treat pass rates as what happened in these runs, not stable success rates.
• Notes scoring checks owner + due date pairs and extra items. It does not grade how each task is worded.
• Amounts are what ofox billed on October 20–21, 2026, except the escalation demo, which uses the API’s reported usage multiplied by list price. The Claude responses we received through the OpenAI-compatible endpoint did not include a prompt-cache breakdown.
• Prices and model versions change. Check the current ofox model list before reusing these numbers.
• We did not test prompt caching strategies, batch APIs or semantic caching. Our cost-reduction overview covers those.
• Ofox published this article and ran the tests on its own platform.

Downloads

All files are in /blog/_assets/evidence/cost-per-correct-result-20261022/:

• results-billed.csv: all 86 calls with tokens, score and billed amount
• runner.py: the test runner and scorers
• escalate.py, cron-task.txt, cron-tests.json: the escalation demo
• coding-tasks/results.csv: the five coding-task checks
• Inputs and answer keys: meeting_notes.md, meeting_notes_hard.md, ground_truth.json, ground_truth_hard.json, buggy_code.py, hidden_tests.py

Set OFOX_API_KEY in your environment to run them. Before running escalate.py, save cron-task.txt as task.txt and cron-tests.json as tests.json in the same folder. The coding-task results also include GLM 5.3 Flash and Qwen 3.8 Flash runs at a 6,000-token cap; most of those returned empty replies and are not discussed above. If you want to measure cost against limits rather than per call, our guides on Codex weekly limits and Claude Code usage limits cover the subscription side.