A benchmark score is only as reliable as the harness that produced it

The Laurenceo ViewLaurenceo Intelligence19 November 2026

Two verified runs of GPT-6.2 Astra sit thirty points apart, so the harness and cost per task now matter as much as the score 1.

ARC Prize has posted verified results for OpenAI's GPT-6.2 Astra showing 100.0% on ARC-AGI-3 through a provider adapter at about $4.1K, and 70.3% on the standard harness at $24.2K 1. Both results describe the same model, yet they differ by thirty points. The 100% figure is also the first such result on that benchmark, which makes the gap central to how the score should be read 1.

The same release lists 95.8% on ARC-AGI-2 at $0.50 per task and 98.5% on ARC-AGI-1 1. Those numbers only become comparable when the cost per task and the evaluation setup are stated beside them. Other vendors are making similar pitches on price: Google says its Gemini 4 Neon delivers about 50 to 80% lower cost per task than top competing models 5, while Microsoft's performance and cost figures are its own claims and remain unverified by the company's release 2. Cost per task is becoming the metric vendors choose to emphasise, and it is rarely presented on a common basis.

For decision makers in Bermuda's insurance, reinsurance and financial sectors, the implication is procedural. Any model considered for underwriting support, claims triage, pricing or regulatory work should be tested on the firm's own tasks, under its own harness, with cost measured per completed task rather than per list price. A procurement file that records a score without the evaluation setup does not show that a model is fit for purpose. Regulators and boards that accept vendor headlines at face value will find it hard to explain later why the chosen configuration performed as it did.

The leaderboard question is the one to watch. Whether the standard-harness result, rather than the adapter result, becomes the figure the market treats as the reference will determine how much weight vendor claims can bear 1. Until that settles, buyers should treat any single verified number as one point on a curve shaped by scaffolding and price. The practical demand is simple: every claim should come with the harness type, the cost per run and the evaluation setup, and a vendor that cannot supply all three has not yet made a comparable claim.

“The same AI model scored 100% in one test and 70.3% in another. The way a test is set up can move the score just as much as the model itself does.”

Jorai LaurenceoJorai Laurenceo
Executive Director, Laurenceo

Laurenceo's Position

A headline AI score describes a specific harness and price, and buyers who do not audit both are comparing the wrong things.

The Story

This editorial responds to the lead story of the briefing for 19 November 2026: ARC Prize verifies GPT-6.2 Astra at 100% on ARC-AGI-3 with provider adapter; 70.3% on standard harness. Numbered links open the sourced story behind each point.

More Analysis

All analysis and editorial →

About The Laurenceo View

The Laurenceo View is the daily editorial of Laurenceo Intelligence: the publication's position on the day's lead story, argued only from Laurenceo's own sourced reporting. It is written by Laurenceo's research system under standing editorial direction set by the Editor, who can amend or withdraw it. How this is made.

Laurenceo Intelligence is the daily AI intelligence briefing published in Bermuda by Laurenceo and edited by Jorai Laurenceo. Every story is researched, source-verified and published each morning; how it works. General information only, not legal, regulatory or financial advice.