Generated hero card for the Terminal X evaluation of GPT-6.1 Sol, headed 'An eval agent that agrees with the senior reviewer', with figures reading '88.6% agreement across 500 scored judgments', '86.4% for the next-best model', '+220 basis points', and 'about 30% lower cost per query', a footer reading 'Figures published by OpenAI in a Terminal X customer story, October 20, 2026; not independently replicated', and the OrcaRouter logo in the bottom-right corner.
Guides & Insights

Terminal X Agrees With Its Senior Investors 88.6% of the Time on GPT-6.1 Sol — and Saves 30% Doing It

Author

Elias Hawthorne

Date Published

Latest models · 20View all models →
Benchmarks: Artificial Analysis · updated daily
Back to all posts

The most interesting number in O​penAI's October 20, 2026 write-up of Terminal X is not the model's name. It is 88.6% — the share of 500 scored investment judgments where an eval agent running on GPT-6.1 Sol reached the same verdict as the firm's senior human reviewer. The next-best model in the same harness managed 86.4%. Two hundred and twenty basis points separate them, and roughly 30% separates their cost per query. Terminal X also runs GPT-6 Luna for the high-volume steps, and the split between those two tiers is the actual engineering story here: judgment where judgment is needed, throughput everywhere else. The documented AI failures and agent incidents behind releases like this one are collected in the OrcaRouter AI Incident Archive where each entry carries its own sources, severity, confidence grade and disputed flag.

One thing to be clear about before the numbers get quoted around a desk: this is a customer story published by the model vendor, measured by the customer, on the customer's own rubric. It is a real result in the sense that the harness exists and runs in production, and it is not an independent benchmark in any sense. Nobody outside Terminal X has reproduced the 88.6%. Treat it as the most detailed public evidence so far about how a GPT-6.1-tier model behaves on professional judgment work — not as a leaderboard entry.

What was actually measured

Terminal X sells what it calls Capital Memory: a firm-owned system that indexes data rooms, models, memos, filings, emails and investment-committee history, then encodes how a particular desk makes decisions into agents the firm owns, deployed on-premise or in its own cloud. The engineering staff are former finance and investment professionals, and they sit with each customer for weeks reverse-engineering the reasoning that never got written down — how the credit desk screens a borrower, which sources the co-investment team pulls before a memo.

The eval layer is a separate agent. Per the story, it reads every production output against a 54-criterion rubric and returns pass or fail per criterion, with a one-line reason and a corrected source for every failure. Those failures become new workflow instructions and new test cases, so the rubric thickens as the firm corrects the system.

The headline comparison comes from 500 scored judgments where a senior human reviewer had already ruled. The agent's agreement rate, by model:

• GPT-6.1 Sol — 88.6% agreement across 500 judgments

• Next-best model — 86.4% agreement on the same 500

• Gap — 220 basis points, or roughly one extra agreement per 45 judgments

Terminal X names one error class where the gap concentrates: style drift. A buyout fund whose mandate is take-private candidates trading at a discount to net asset value should not be recommending a distressed credit position or a rescue financing when scenario models turn. The eval agent on GPT-6.1 Sol caught about 25% more of those departures from mandate than the next-best model did. That is a narrower claim than "88.6% accurate", and a more useful one: the model is better at noticing that a recommendation has wandered off the thesis, which is the failure mode a senior reviewer actually spends their day catching.

Why a fund's eval is a different kind of benchmark

Public benchmarks ask every model the same questions. Terminal X asks each model the questions of one firm, against one rubric, in the firm's own house style. That makes the result more relevant to the buyer and less comparable across buyers — the same harness at a different fund would weight different criteria, and the 220-point gap could easily compress or widen.

It also makes the result expensive to check. Reproducing it requires the 54-criterion rubric and 500 judgments, neither of which is public. What is public is the shape of the claim and the production context around it, and on that basis the result is coherent: roughly 1,700 firm Memories built, more than a million custom workflows run, and weekly active users equal to 81% of monthly actives — a stickiness ratio that suggests the agents are being used inside the working week rather than trialled and abandoned. One private equity firm is cited at $2.4 million in annualised savings from automated due diligence and memo workflows. All of that is the customer's own account.

The cost line matters more than the accuracy line

According to the story, GPT-6.1 Sol ran the same workflows at roughly 30% lower cost per query than the next-best model, at similar speed. Terminal X attributes that to token efficiency on long-context work: the model could test a thesis against the fund's mandate, then against its recent deals, then re-run the uncertain verdicts and reconcile, all inside the same budget. That deeper review is what flagged deals a single pass had waved through.

A 30% cost reduction on a per-query basis is a much larger commercial fact than 220 basis points of agreement, because eval and review agents are run constantly and their bill scales with volume. It is also the kind of claim a vendor story can make loosely — "similar speed" is not a number — so it belongs in the same category as the rest: directionally credible, unverified.

What GPT-6.1 Sol is, on the record

GPT-6.1 Sol is the model O​penAI released on September 29, 2026 for complex coding and professional work. It is the only GPT-6.1-tier model on O​penAI's model list — there is no separate bare "GPT-6.1" identifier, and a request naming one will not resolve. The vendor's current specification, as published:

• Price — $2.00 per million input tokens and $10.00 per million output tokens

• Cached input — $0.10 per million; cache writes $2.50 per million

• Long prompts — above 272,000 input tokens, input and cache cost 2× and output 1.5×

• Context window — 1,050,000 tokens, with 922,000 maximum input and 128,000 maximum output

• Modalities — text and image in, text out

• Reasoning effort — low, medium, high, xhigh and max; none and minimal are rejected

• Knowledge cutoff — April 30, 2026

• Service tiers — Fast at 2× standard, Batch and Flex at 50% below, Ultrafast at 6×, with US and EU data residency

Screenshot of OpenAI's developer documentation page for gpt-6.1-sol, showing the model identifier, a pricing table with $2.00 per million input tokens and $10.00 per million output tokens, a cached input rate of $0.10, a 1,050,000-token context window, and the reasoning effort options low through max.

The independent picture on Artificial Analysis, read on October 21, 2026, is a single Intelligence Index of 52 at maximum effort (index v4.3.2, ten evaluations), priced at $0.72 per index task and a blended $1.47 per million tokens, with output speed of 55.2 tokens per second. The board also records a time to first token of 248 seconds at that effort setting, against a comparator median of 3.69 seconds — which is worth knowing before anyone points a synchronous UI at the maximum effort tier.

A generated scoreboard card for GPT-6.1 Sol with six rows: Intelligence Index 52, price $2.00 input and $10.00 output per million tokens, cached input $0.10 per million, context window 1,050,000 tokens, output speed 55.2 tokens per second, and time to first token 248 seconds at maximum effort, with a footer reading that prices and specifications are per OpenAI while index, speed and latency are per Artificial Analysis on 21 October 2026.

Those two sets of numbers describe different things, and the gap between them is the honest summary of the model: a 1M-context reasoning tier priced like a mid-tier model, whose economics work for batch evaluation and long agentic runs, and do not work for anything a person waits on interactively at max effort.

What the 88.6% does not tell you

• It is not a general capability measure. One rubric, one asset class mix, one firm's standards.

• It is agreement with a human, not correctness. A reviewer with a consistent bias and a model that shares it will score well.

• The comparison set is unnamed. "Next-best model" could be a frontier competitor or a much cheaper tier; without the name, the 220 basis points cannot be priced.

• Cost per query is not defined. Whether that includes the extra review passes the model enables is exactly the kind of detail that changes the conclusion.

• Nothing here is a like-for-like public benchmark result. On the one independent index that does exist, GPT-6.1 Sol sits one point below GPT-6 Astra and four points above the GPT-6 Sol tier it replaced.

Testing a claim like this on your own data

The only way to know whether the mandate-drift result transfers is to run your own 200 to 500 judgments through two models and score them against your own reviewer.

Screenshot of the OrcaRouter model page for openai/gpt-6.1-sol, showing the GPT-6.1 Sol heading, its latency and throughput figures, a 1,000,000-token context panel, and per-million pricing of $2.00 input and $10.00 output with a $0.10 cached-input rate.

That is a cheap experiment now: GPT-6.1 Sol is on OrcaRouter's routes at the vendor's own list price, with no markup added, so a comparable run against GPT-6 Astra, GPT-6 Sol or GPT-6 Luna costs the rate card and nothing else. Because all four tiers sit behind one key, the comparison is a change to a model string rather than a second contract.

Terminal X's own architecture is the pattern worth copying: route the judgment-heavy stage — thesis synthesis, mandate fit, the reconciliation pass — to the strongest tier you can justify, and route extraction, classification and the high-volume steps to something like GPT-6 Luna at $0.10 and $0.50 per million tokens. The routing DSL exists precisely so that split can live in a config rather than in application code, and automatic failover means an evaluation run does not die halfway because one upstream had a bad afternoon.

What to watch

Two things would turn this from a vendor anecdote into a result. First, a second firm publishing its own agreement rate on a different rubric — if 88.6% is a property of the model rather than of Terminal X's harness, a credit desk and an infrastructure fund should land in the same neighbourhood. Second, the named comparison. An 86.4% "next-best model" is the number everyone will want to interrogate, and until it has a name the 220 basis points are a claim about a category, not a comparison between two products.

Until then the defensible reading is narrow and still useful: on one production investment workflow, a GPT-6.1-tier model agreed with a senior reviewer often enough to automate the first pass, caught about a quarter more mandate drift than whatever it was measured against, and did it for about 30% less per query. That is a good reason to run your own test. It is not yet a reason to skip one.

Compared in this article1

Detected from this article · Benchmarks: Artificial Analysis · updated daily