How do you compare and benchmark Frontier AI models? Building fair and meaningful tests for AI turns out to be surprisingly hard with the evolving capabilities of models.

This blog post outlines the AI evaluation benchmark landscape, what each benchmark measures, how fast it is saturating, and where the frontier AI stands today.

What is a Benchmark?

A benchmark is a standardized test used to measure how well a model performs on a specific task. Just as students take exams to assess knowledge, AI models are run through benchmarks to measure their capabilities.

“Frontier models” refers to the most capable AI systems currently available e.g. GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, Grok 4, etc. Evaluating these models requires progressively harder tests. When every model aces a test, that test no longer tells you anything useful i.e. it has saturated. The field then moves to a harder benchmark, and the cycle repeats.

What Benchmarks Actually Measure

Not all benchmarks test the same thing. Before looking at specific evaluations, it helps to understand the capability domains they cover, because a model that excels at one may fall short at another.

Knowledge & Factual Reasoning

The most natural place to start evaluating a model is: does it know things? The first generation of knowledge benchmarks tested broad coverage. As models mastered those, the tests had to get deeper and more expert.

MMLU

MMLU (Massive Multitask Language Understanding) contains 16k+ multiple-choice questions covering 57 academic subjects like humanities (law, history, philosophy, etc.), social sciences (politics, geography, etc.), STEM (science, math, physics, etc), medicine, etc. GPT-3 model scored around 43.9% in 2020 compared to human expert’s 89.8% on MMLU. Today every frontier model exceeds 88%. A 2-point gap between models falls within measurement noise. MMLU is now a floor check, useful only for confirming a model isn’t broken.

# Example Question:
One of the reasons that the government discourages and regulates monopolies is that
(A) producer surplus is lost and consumer surplus is gained.
(B) monopoly prices ensure productive efficiency but cost society allocative efficiency.
(C) monopoly firms do not engage in significant research and development.
(D) consumer surplus is lost with higher prices and lower levels of output.
## Answer: D

MMLU-Pro

MMLU-Pro is an upgrade over MMLU with 12k+ graduate-level reasoning-based questions with ten answer choices instead of four, making guessing much harder. As of early 2026, the leading score has already reached ~90%, and MMLU-Pro is itself approaching saturation.

# Example Question:
Ms. Chen purchased a used car, worth $1650, on the installment plan, paying $50 down and $1,840 in monthly installment payments over a period of two years. What annual interest rate did she pay?
Options:
A. 10% B. 17.5% C. 15.2% D. 13.3% E. 20% F. 19.8% G. 18% H. 16% I. 12% J. 14.4%
## Answer: J

GPQA Diamond

Graduate-Level Google-Proof Q&A contains much complex questions. Its 198 questions in biology, physics, and chemistry were written by PhD-level domain experts and designed to be unsolvable by searching the web.

# Example Question:
Methylcyclopentadiene was allowed to react with methyl isoamyl ketone and a catalytic amount of pyrrolidine. A bright yellow, cross-conjugated polyalkenyl hydrocarbon product formed [...] How many chemically distinct isomers make up the final product (not counting stereoisomers)?
(a) 2 (b) 16 (c) 8 (d) 4
## Answer: b
Explanation Methylcyclopentadiene exists as an interconverting mixture
of 3 isomers [...] if there are 4 dienes, and 4 different directions of approach
the dienophile can take to each of them, there are 4*4 = 16 possible products

• Skilled non-experts with unrestricted internet access: 34%
• PhD experts in the relevant field: ~65%
• GPT-5.4 (April 2026): 92%
• Claude Fable 5 and Gemini 3.1 Pro (June 2026): 94%

Frontier models have surpassed PhD experts on subject matter. GPT-6 Astra (96.0%), Gemini 3.8 Flash (95.3%), and Claude Fable 5.1 (93.7%) currently lead the leaderboard. GPQA Diamond is approaching saturation at the very top but still separates models in the 60–90% range.

Note that human expert scores vary across benchmarks: ~65% on GPQA Diamond (narrow, field-specific questions) versus ~90% on HLE (broader questions across many fields). Both use domain experts, but GPQA tests depth within a subfield while HLE tests breadth across disciplines.

Humanity’s Last Exam (HLE)

HLE is the current ceiling for knowledge evaluation. It comprises 2,500 questions created by domain experts across math, humanities, and natural sciences, all written from scratch, making it nearly impossible for a model to have “seen” the answers during training. The current status as of September 2026 looks like this:

The “with tools” means models can run code or search the web during the test. That gap tells you how much a model depends on external tools versus internal reasoning. At the current pace, HLE may saturate within a year or two, following the same arc as every benchmark before it.

LiveBench

LiveBench takes a different approach to keeping knowledge benchmarks fresh: it releases new questions monthly, drawn from recent math competitions, research papers, etc. Because questions are always fresh, models cannot have memorized the answers during training. It contains 18 tasks in 6 categories: math, coding, reasoning, language, instruction following, and data analysis.

# Example Question:
There are 3 people standing in a line numbered 1 through 3 in a left to right order.
Each person has a set of attributes: Food, Nationality, Hobby.
The attributes have the following possible values:
- Food: nectarine, garlic, cucumber
- Nationality: chinese, japanese, thai
- Hobby: magic-tricks, filmmaking, puzzles
and exactly one person in the line has a given value for an attribute.
Given the following premises about the line of people:
- the person that likes garlic is on the far left
- the person who is thai is somewhere to the right of the person who likes magic-tricks
- the person who is chinese is somewhere between the person that likes cucumber and the person
who likes puzzles
Answer the following question: What is the hobby of the person who is thai? Return your
answer as a single word, in the following format: **X**, where X is the answer.

Mathematics & Logical Reasoning

Mathematical reasoning tests the ability to chain together logical steps to solve math and logic questions.

GSM8K and HellaSwag

GSM8K tests grade-school math word problems. HellaSwag tests commonsense reasoning i.e. whether a model can pick the most sensible ending to a story. Both are now saturated with models scoring above 95% and 92% respectively. They are only useful as quick sanity checks to make sure a model isn’t broken.

# Example question from GSM8k:
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
## Answer
Natalia sold 48/2 = <<48/2=24>>24 clips in May.
Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.
Answer: 72

# Example question from HellaSwag:
A bearded man is seen speaking to the camera and making several faces. the man
a) then switches off and shows himself via the washer and dryer rolling down a towel and scrubbing the floor. (0.0%)
b) then rubs and wipes down an individual’s face and leads into another man playing another person’s flute. (0.0%)
c) is then seen eating food on a ladder while still speaking. (0.0%)
d) then holds up a razor and begins shaving his face. (100.0%)

AIME

AIME (American Invitational Mathematics Examination) consists of 15 difficult competition math problems with integer answers. It became a useful evaluation once the easier tests saturated. Top frontier models in 2026 approach ceiling performance on AIME 2025, thus pushing the field toward harder contests like the USAMO and PutnamBench.

FrontierMath

It consists of hundreds of unpublished challenging math problems. These are divided into 4 difficulty tiers with level 4 for research-level math. As of September 2026, the FrontierMath Tier 4 v2 scores show dramatic progress:

# Example Question:
Construct a degree 19 polynomial p(x) ∈ C[x] such that X := {p(x) = p(y)} ⊂ P1 × P1 has at least 3 (but not all linear) irreducible components over C. Choose p(x) to be odd, monic, have real coefficients and linear coefficient -19 and calculate p(19).
## Answer: 1876572071974094803391179

ARC-AGI-2 and ARC-AGI-3

ARC-AGI-2 tests abstract reasoning and fluid intelligence, the ability to solve novel visual puzzles from just a few examples, with no relevant training data to draw on. The puzzles are grids of colored squares; the model must infer the underlying rule and complete the pattern.

ARC-AGI-3, released in 2026, pushes the frontier further with interactive reasoning environments that challenge AI agents to explore novel environments, acquire goals on the fly, and learn continuously. Unlike its predecessors which tested passive puzzle-solving, ARC-AGI-3 measures skill-acquisition efficiency and adaptation over time.

Note that GPT-6 Astra scores 99.95% with the Provider Adapter harness (which preserves reasoning state across requests) and 62.7% with the Standard harness.

Coding & Software Engineering

These benchmarks measure whether a model can write, debug, and navigate real codebases, more than just producing plausible-looking code in isolation.

HumanEval

One of the first coding benchmarks, it asked models to complete Python functions from docstrings. It saturated above 93% and has been retired in favor of harder successors.

# Example Question 
def words_string(s):
    """You will be given a string of words separated by
    commas or spaces. Your task is to split the string 
    into words and return an array of the words.
    For example:
    words_string("Hi, my name is John") == ["Hi", "my",
        "name", "is", "John"]
    words_string("One, two, three, four, five, six") ==
        ["One", "two", "three", "four", "five", "six"]
    """

SWE-Bench, SWE-Bench Verified, SWE-Bench Pro

The SWE-Bench measures: Given a real GitHub codebase and a bug report, can the model produce a correct patch? The SWE-Bench Verified is a subset of 500 problems from SWE-Bench verified by researchers. The SWE-Bench Pro is harder version with problems from spanning complex codebases e.g. consumer applications, B2B services, and developer tools. It contains 1865 problems from 41 actively managed repositories.

• 2023: 4.4% of issues solved
• 2024: 71.7% solved (+67 pp in one year)
• Mid-2026: Claude Fable 5 reached 80.3% on SWE-bench Pro
• September 2026: GPT-6 Astra and Claude Fable 5.1 push the frontier further

FrontierCode and DeepSWE

FrontierCode (by Cognition) evaluates whether the model can write production-quality code that would eventually get merged by benchmarking it against PR rubrics. It measures correctness, test quality, scope discipline, style, and adherence to codebase standards.

DeepSWE v1.1 evaluates agentic software engineering capabilities on 113 original, long-horizon tasks across 91 active open-source repositories in five languages (TypeScript, Go, Python, JavaScript, Rust). Unlike SWE-bench which mines merged PRs and risks contamination, DeepSWE tasks are written from scratch and never contributed upstream, keeping solutions out of training data. Each task is graded by hand-written functional verifiers rather than inherited test suites. On a matched audit, an independent LLM judge disagrees with DeepSWE’s verifier just 1.4% of the time, compared to 32.4% for SWE-Bench Pro’s inherited tests. Despite prompts being ~half the length of SWE-Bench Pro’s, reference solutions touch 5.5x more code, requiring ~2x more output tokens.

Agentic & Tool-Use Capability

Coding benchmarks test whether a model can produce the right output. Agentic benchmarks go further: can the model take actions across many steps, use tools, and recover when things go wrong? This is what it means to deploy a model as an autonomous assistant rather than a question-answering system.

Terminal-Bench 4.0 and Terminal-Bench Science 0.1

Terminal-Bench tests multi-step agentic terminal operation i.e. writing scripts, debugging shell pipelines, and interpreting command-line output across many turns. Instead of producing a single file, the model must operate an actual terminal environment end-to-end.

Terminal-Bench 4.0 is the latest version, with GPT-6 Astra achieving 57.9% and Claude Fable 5.1 at 55.8%:

Terminal-Bench Science 0.1 tests whether agents can complete scientific research workflows using code and terminal tools, including analyzing data, running simulations, and fitting models. This is a new benchmark that measures scientific coding capability:

OSWorld

OSWorld takes agentic evaluation the furthest: can a model operate a real computer? It tests tasks across operating systems: opening files, navigating a browser, running terminal commands, interacting with GUI applications. Unlike TerminalBench, OSWorld involves full visual interfaces and requires the model to perceive and act on a screen. Accuracy has risen from ~12% to 72.6% (GPT-6 Astra). Human performance is around 72%, meaning models have now reached parity with the human baseline on this benchmark.

Note: Claude Fable 5.1 and 5 scores on OSWorld 2.0 use partial scoring from the benchmark authors’ August 2026 task release, while GPT-6 Astra uses the offline set with partial scoring. Strict scoring for Claude Fable 5.1 is 41.7%.

BrowseComp

BrowseComp tests whether a model can find specific information on the web when the answer requires navigating and synthesizing content across multiple pages. It measures real-world browsing capability:

Agents’ Last Exam

Agents’ Last Exam (ALE) is a benchmark developed by UC Berkeley RDI in collaboration with 300+ industry experts, designed to evaluate AI agents on long-horizon, economically valuable, real-world tasks. It consists of 1500+ tasks spanning 55 subdomains across 13 industry clusters, grounded in the O*NET / SOC 2018 occupational taxonomy. Tasks are sourced from actual professional practice and require interleaving GUI interaction with CLI operations on real OS sandboxes.

AutomationBench

AutomationBench evaluates business workflow automation: can a model complete real-world business tasks like data entry, form filling, and process management?:

BenchCAD

BenchCAD evaluates whether models can reconstruct 3D objects from multi-view renders by generating executable parametric CAD code (CadQuery programs). Through four tasks: Vision2Code (image-to-code generation), Vision-QA, Code-QA, and CodeEdit (instruction-guided program editing), it tests spatial reasoning, engineering design knowledge, and the ability to recover exact parametric dimensions. The benchmark contains 17,900 execution-verified parts across 106 industrial part families (gears, springs, fasteners, brackets, etc.), with 52 families. Scoring is execution-grounded via voxel IoU between the model’s output and ground-truth geometry, no LLM judge.

CursorBench

CursorBench evaluates coding agent performance within the Cursor IDE, measuring real-world software development workflows:

ScreenSpot-Pro

ScreenSpot-Pro tests whether models can accurately read and understand text rendered in images, a key capability for computer use:

Vision, Long-Context & Document Understanding

The benchmarks so far have mostly involved text. But real-world tasks often require more: understanding images, reasoning over long documents, or both at once. These two domains are closely related; both test what happens when you give a model richer, more complex input than a short text prompt.

Long-context benchmarks

They test whether models can actually use their large context windows. Context windows have grown from 4K tokens (GPT-3) to over 1M tokens (Gemini 3 Pro), but accepting a long document and reasoning over it are very different things. RULER and HELMET measure whether models can actually retrieve and connect information spread across very long inputs. A well-known failure mode here is “lost-in-the-middle” where a model handles the beginning and end of a document fine but loses track of content buried deeper inside.

The OpenAI MRCR v2 (Multi-Round Coreference Resolution) is a newer long-context benchmark that tests whether models can track and resolve references across very long contexts. GPT-6 Astra achieves 100% on the 8-needle 256K-512K variant and 96.3% on the 512K-1M variant, demonstrating near-perfect recall across million-token contexts:

Vision and multimodal reasoning

The models that handle both images and text are called vision-language models (VLMs). MMMU (Massive Multidisciplinary Multimodal Understanding) is the standard: college-level questions across six disciplines that require genuinely understanding images, not just reading captions. Top VLMs score in the 75–86% range, with room still to improve. More specific benchmarks test narrower skills:

ChartQA and DocVQA are nearly saturated. MMMU and MMT-Bench still differentiate models, making them the active frontiers for multimodal evaluation.

Human Preference & Safety

All of the benchmarks above measure what a model can do. But two important questions remain: is it actually useful to interact with, and does it behave responsibly?

Human Preference

Chatbot Arena (LMArena) is the most widely trusted human preference evaluation. Users chat with two anonymous models simultaneously without knowing which is which, and vote on which response they prefer. Millions of these votes are aggregated into an Elo rating, the same system used in chess rankings. A higher Elo means a model wins more head-to-head matchups.

Arena Elo Ratings (June 2026):

The top models are clustered within ~45 Elo points, the tightest spread on record. This has a practical implication: choosing a model now means matching it to your use case, not just picking the highest number. Price, latency, context window, and domain fit often matter more than raw capability differences.

The Open LLM Leaderboard v2 (HuggingFace) provides a consistent automated harness for comparing open-weight models across six benchmarks. Its value is standardization, any model can be run through the same evaluation pipeline.

Safety & Alignment

Safety evaluation is the least standardized part of the benchmark landscape, but it is growing in importance. StrongREJECT tests whether a model refuses harmful requests robustly. TruthfulQA evaluates whether models produce plausible-sounding falsehoods. HarmBench covers a broader range of harmful behaviors.

These benchmarks matter because capability and safety do not automatically improve together. A model that tops the coding leaderboard is not necessarily the most honest or the hardest to manipulate. As frontier models are deployed in higher-stakes settings, safety benchmarks will become as standard in model cards as GPQA Diamond or SWE-bench.

Domain-Specific Benchmarks

The benchmarks covered so far measure general capabilities i.e. how well a model reasons, codes, or understands images across a broad range of tasks. But in practice, many teams deploying AI care about a specific domain: will this model handle medical questions accurately? Can it reason about legal contracts? Does it understand financial statements?

Domain-specific benchmarks answer those questions. They tend to be narrower and harder to saturate, because they demand real subject-matter expertise rather than broad pattern-matching.

A model that scores well on GPQA Diamond may still struggle on MedQA, because clinical reasoning requires not just broad science knowledge but familiarity with how medical decisions are framed. Similarly, high Arena Elo does not predict LegalBench performance; legal tasks require precision and citation accuracy that general helpfulness does not capture.

As frontier labs compete for enterprise use cases, domain-specific benchmarks are becoming a primary differentiator. Choosing a model for a specialized deployment increasingly means running it through the relevant domain benchmark, not just checking its MMLU or Arena Elo score.

Cybersecurity Benchmarks

As AI models gain the ability to identify and exploit software vulnerabilities, cybersecurity benchmarks have become critical for evaluating both capability and safety. These benchmarks test whether models can find and develop exploits for real software vulnerabilities.

GPT-6 Astra’s 100% score on ExploitBench and discovery of previously unknown zero-day vulnerabilities during evaluation represent a significant jump in cyber capabilities. Claude Mythos 5.1 (Fable with fewer safeguards) achieves 30.4% on ExploitGym.

Science & Health Benchmarks

AI models are increasingly being used for scientific research and healthcare applications. These benchmarks evaluate domain-specific knowledge and reasoning in biology, medicine, and chemistry.

Claude Fable 5.1 and Mythos 5.1 also demonstrated significant scientific research capabilities, including designing high-affinity protein binders with nearly 50% hit rate and creating a new high-resolution elevation map of Venus from NASA Magellan radar data.

OpenScore String Quartets

OpenScore String Quartets evaluates optical music recognition (OMR), the ability to convert images of sheet music into machine-readable notation. GPT-6 Astra achieves 0.84 on the OMR-NED metric, compared to 0.19 for GPT-5.6 Sol, representing a major advance in multimodal understanding of specialized document types.

Why Benchmarks Break Down

Running through each domain makes clear how much progress the field has made. But it also reveals a deeper problem: benchmarks have a shelf life, and that shelf life is shrinking.

Where the Frontier Stands (September 2026)

Putting it all together, here is the current state across the key benchmarks:

No model leads across all benchmarks. GPT-6 Astra dominates in abstract reasoning (ARC-AGI-3 at 99.9%), mathematics (FrontierMath Tier 4 at 97.6%), and cybersecurity (ExploitBench at 100%), while Claude Fable 5.1 leads on HLE with tools (65.0%) and CursorBench (73.4%). The agentic benchmarks, Terminal-Bench 4.0, OSWorld, and Agents’ Last Exam, are where the most active competition is happening right now.

Conclusion

Benchmarking frontier AI models is a fast-changing field that requires regular updates. Tests saturate, harder ones replace them, and the cycle repeats faster each year. No single benchmark tells the full story, and no single model leads across all domains.

The September 2026 releases of GPT-6 Astra and Claude Fable 5.1 have pushed several benchmarks to saturation: FrontierMath Tier 4 (97.6%), ARC-AGI-3 (99.9%), and ExploitBench (100%). Meanwhile, benchmarks like Terminal-Bench 4.0, AutomationBench, and the science/health benchmarks remain wide open with significant room for improvement.

Choosing a model is less about finding the highest score and more about matching capability to use case, whether that’s knowledge reasoning, coding, agentic tasks, vision, or a specific domain like medicine or law.

And if history is any guide, the benchmarks you read about today will be obsolete within two years. The only constant is that the frontier keeps moving.

References:

• Paper: MMLU, Measuring Massive Multitask Language Understanding (MMLU)
• Paper: MMLU-Pro, A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Leaderboard: MMLU-Pro
• Paper: GPQA, A Graduate-Level Google-Proof Q&A Benchmark
• Leaderboard: GPQA Diamond
• Paper: Humanity’s Last Exam
• Leaderboard: Humanity’s Last Exam
• Paper: LiveBench, A Challenging, Contamination-Free LLM Benchmark
• Paper: GSM8K, Training Verifiers to Solve Math Word Problems
• Website: HellaSwag
• Paper: FrontierMath, A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
• Website: FrontierMath EpochAI
• Paper: ARC-AGI-2
• Paper: ARC-AGI-3
• Leaderboard: ARC-AGI
• Paper: HumanEval, Evaluating Large Language Models Trained on Code
• Paper: SWE-Bench, Can Language Models Resolve Real-World GitHub Issues?
• Leaderboard: SWE-Bench Verified vals.ai
• Leaderboard: SWE-Bench Pro
• Blog: FrontierCode
• Paper: OSWorld, Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• Paper: Agents’ Last Exam
• Leaderboard: Agents’ Last Exam
• Paper: BenchCAD
• Leaderboard: BenchCAD
• Paper: MMMU, A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark
• Paper: Chatbot Arena, An Open Platform for Evaluating LLMs by Human Preference
• Leaderboard: Chatbot Arena
• Paper: Terminal-Bench
• Leaderboard: Terminal-Bench 4.0
• Paper: TruthfulQA, Measuring How Models Mimic Human Falsehoods
• Paper: A StrongREJECT for Empty Jailbreaks
• Paper: MedQA, What Disease does this Patient Have?
• Paper: LegalBench, A Collaboratively Built Benchmark for Measuring Legal Reasoning in LLMs
• Paper: FinanceBench, A New Benchmark for Financial Question Answering
• Blog: GPT-6 Astra: A new generation of intelligence
• Blog: Introducing Claude Fable 5.1 and Claude Mythos 5.1
• Blog: Claude Fable 5 and Claude Mythos 5
• Blog: reviewing GPT‑5.6 Sol: a next-generation model