Engineering milestone: October 19, 2026. Added to the News archive October 30, 2026.
Start with the question the model actually received
Two models can disagree about a supplier document for reasons that have little to do with their underlying ability. One may have received a newer file. The question may have changed. A previous answer may have slipped into the context. Even a small difference can turn a comparison into a test of two different tasks.
That is why our model-testing work starts inside the verification record. We want to compare models against the questions Atlas actually asks, using the evidence those questions are allowed to see. The unit of comparison is a defined question with a defined input, followed by a result that someone can review.
The immediate goal is to understand where a model helps our verification workflow and where it still needs scrutiny. A broad benchmark cannot answer that for us.
Freeze the work before changing the model
For comparisons based on the saved question catalog, Atlas reconstructs the question definition, its criteria, the subject, and the relevant evidence inputs. The comparison retains the question and taxonomy revisions, document references, input revision, and evidence fingerprint. Previous model verdicts are excluded from the packet.
Both sides of the comparison can then be evaluated against that saved task. Missing evidence is recorded as a coverage gap rather than quietly replaced with material from a different run. This makes the limits of the experiment visible before we start interpreting its results.
The distinction matters when investigating a change. If the source document changed, a different answer may be appropriate. If the evidence stayed the same and only the model changed, the disagreement deserves a different kind of review. A useful testing system has to preserve enough context to tell those situations apart.
Jev and the broader model have different jobs
Jev is integrated into our verification workflow for focused evaluations with defined inputs and constrained answers. Broader language-model reasoning supports the investigation around those decisions: interpreting context, following evidence relationships, and helping identify what needs examination next. We compare the components at the level of the work they are being asked to do.
The testing system supports a typed-evaluation model and a supported language model selected from the registered model catalog. It records the model configuration and cost metadata used for the experiment. Repeated trials help expose variation, and explicit concurrency and cost limits bound the run.
This is not a claim that one type of model is better at every task. It is a way to make the division of work measurable. The article about our updated supplier record covers the practical economics of that division; this testing work is how we examine the decisions underneath it.
An answer saved is not an answer proved correct
A run can complete and still leave an investigation unresolved. A question can have a stored answer that needs evidence review. A technical failure, an evidence gap, a reused answer, and a newly evaluated question describe different things. Combining them into one completion number would make a dashboard look simpler while making the experiment harder to understand.
Our comparison views retain those distinctions. They match questions by their meaning and scope rather than by new IDs assigned during a run. When duplicate or changed questions cannot be paired unambiguously, the system leaves that ambiguity visible instead of counting an arbitrary match as an improvement.
Review labels belong to their own record. Agreement between two models is an observation; correctness requires a defensible reference judgment. We are building the workflow around that difference, so a completed computation does not become an accuracy claim by accident.
Cost and latency need the same discipline
A faster answer is useful only if it does the required work. A lower processing estimate is meaningful only if the comparison accounts for what was included. Atlas records reported usage, estimated cost where the required usage data exists, and latency alongside the individual trials. Unknown billing is not treated as a free call.
The experiment also retains the evidence and model configuration needed to explain a change between runs. This makes it possible to ask a more useful question than which model is cheapest: what did we receive for that spend, what remained unresolved, and how much review did the result require?
The verification record outlasts the model
Models will change. The questions a trade team needs to answer, the evidence required to answer them, and the responsibility to review consequential findings will remain. We are building Atlas so that those parts of the workflow stay inspectable when an underlying model is replaced.
That is the purpose of this testing work. Preserve the task. Change the component deliberately. Examine the result in enough detail to decide what should happen next.