Your AI Passed the Test. Did It Answer the Business Question?

Your AI Passed the Test. Did It Answer the Business Question?

Before a product team puts an AI assistant into production, it needs a way to determine whether the assistant can be hired to do the job. That is what evaluations, usually shortened to evals, are for. Evals are different the the tests used in traditional software development, and they are different then the performance metrics used to assess human employees. They are also not user acceptance tests.

An eval is a structured test: give the AI a task, observe what it does, and compare the result with expectations established beforehand. Success might mean answering correctly, completing an action, asking a necessary question, or recognizing that the available information cannot support a conclusion.

Evals are more than a successful demonstration or a score showing how quickly an assistant works. They can include familiar software tests and support user acceptance testing, the testing is not about getting the answer right every time, but to understand its behavior under different circumstances. In other words, they examine how the AI handles a range of tasks and conditions. They provide evidence for deciding where the assistant is dependablly competent to do the job following the right process, where it needs help, and what must improve before release.

Teams collect these test cases and run them against different versions of their product. The results help them identify weaknesses that need attention, compare changes to the application or the model over time, and decide what portions of the application are ready for release. The process is repeatable, although the AI’s responses may vary between attempts. Anthropic’s guide to evaluations describes these building blocks.

Consider a fictional company developing an assistant that answers questions about cloud spending. The documents show $120,000 was spent for systems used by customers and employees, $30,000 for software development, and $10,000 for testing. Together, they cover all company cloud costs for the same quarter.

When the assistant is asked to provide a figure for total spending, the assistant should answer $160,000.

Now change the test. Give the assistant only the $120,000 report, which explicitly excludes development and testing. It answers:

“Our total cloud spending last quarter was $120,000.”

That number is present in the report and approves the response.

The number is correct. But is there any support for that conclusion? The company-wide claim is unsupported.

A simple checker is plainly inadequate: it verifies that a value appears in a source without checking whether the source supports the complete claim. But the example exposes a broader design problem. An eval can correctly enforce its scoring rules while those rules fail to capture what the business needs.

A more capable grader helps only if the evaluation gives it the right job.

In Right Numbers, Wrong Questions, posted October 28, 2026, TIFIN researchers examined constructed Indian mutual-fund questions involving ambiguous figures, product variants and missing investor information.

They also tested an AI judge. Providing a dictionary of field definitions helped it identify 40 of 44 mislabeled figures. Yet, without being told the intended interpretation, it accepted 42 of 52 answers that silently chose one reading of a vague question.

This was their key finding. A judge assessing whether an answer addresses “the question as asked” can accept a plausible reading without requiring the assistant to disclose that reading. Better definitions improved one check; they did not establish that the answer matched the user’s unstated intention. Requiring explicit interpretations was proposed, but not tested as a remedy. These were controlled research cases, not failure rates for deployed businesses.

Other research posted that day supplies related evidence. Beyond fabrication found numerical claims in AI-written research that used existing values from the wrong experiment, metric or run. DeoGen found that models performed substantially worse when policy questions lacked enough information for a conclusion. That study used generated cases from four U.S. policy texts, with extended reasoning disabled.

These studies examine different failures. They do not establish how often the problem occurs across enterprises or prove that particular remedies will work in production. They give product teams specific behaviors to investigate: preserving a figure’s meaning, attributing it to the correct result, and recognizing when a necessary fact is missing.

In The Map Is Not the Decision, I argued that analytical outputs need explicit assumptions and boundaries around what they can support. Evaluation makes that requirement concrete. A passing result should depend on whether the answer stays within those boundaries.

For a product team, the work begins before the AI runs.

Someone must assemble representative questions, the information available for each, and the behavior that the job was completed successfully. A test runner can then submit those cases to the product, capture its responses and actions, apply grading rules, and record the results. After a change, the team runs the cases again and compares performance. OpenAI’s evaluation guidance recommends this cycle of defined objectives, representative cases, measurement and continued evaluation.

The grader might be ordinary software checking a calculation, a person assessing an explanation, or another AI applying written criteria. Different parts of the same task can use different graders. An assistant that updates a record should be checked against the resulting record, as well as what it says it did. These distinctions are covered in Anthropic’s evaluation guide.

For the fictional cloud-cost assistant, one test result could look like this:

The failure identifies behavior that needs attention. It does not, by itself, establish the cause. Perhaps the search process missed available records. Perhaps the assistant ignored the report’s exclusions. Perhaps the product never specified what to do with incomplete information.

Running an eval does not repair any of those problems. It gives the team evidence with which to investigate them.

This is why domain specialists belong in evaluation design. Engineers can build the machinery, while finance, operations or customer-support specialists help establish what a correct result means. If finance defines spending as payments made and procurement means purchase orders issued, the product team must resolve that difference—or specify when the assistant should present both measures.

As I argued in How to Communicate in a World of AI, specifications preserve intent. Here, that intent must reach the grading rules. “Be accurate” leaves too much unstated. “Do not present production-only costs as a company-wide total” is a requirement someone can test.

The grader also needs scrutiny. Give it a well-cited answer from the wrong quarter. Give it a correct answer phrased differently from the reference answer. Check whether it rejects the misleading response and accepts the valid one.

Then connect those results to a decision.

Suppose the team is considering a faster, cheaper version of the cloud-cost assistant. Its overall evaluation score improves, but it now presents production-only costs as total company spending.

Before testing, the product owner and finance specialist agreed that this failure would block release. The team therefore holds the update, investigates the defect, makes a change, and reruns the relevant tests. It also checks that the assistant still answers directly when complete records are available.

This is a hypothetical release decision, but it illustrates the practical value of evals. An aggregate improvement does not settle whether a change meets the product’s requirements. The team needs to know which behaviors improved, which deteriorated, and whether any failure violates a requirement it considers essential.

Those requirements should reflect the consequences of the work. A confusing sentence and an unsupported financial total need not carry the same weight.

There is also a cost to excessive caution. The financial-query study found that one intervention reduced wrong answers while introducing unnecessary clarification. Stronger models handled some missing-information tasks successfully. More questions were not a universal improvement. Right Numbers, Wrong Questions.

For our cloud-cost assistant, the team should therefore test both conditions: incomplete records should prevent an unsupported total; complete records should produce a direct answer. If an approved definition of spending already exists, the assistant should use it rather than repeatedly asking the user to define the term.

The team can track consequential errors, unnecessary questions, unresolved requests and time to completion separately. That makes it easier to see whether a proposed improvement helps users finish their work or merely transfers more effort to them.

A passing evaluation remains bounded evidence. It shows how the system performed on selected cases under particular grading rules. It does not guarantee every future response. A live check of an individual answer is a separate control, whose effectiveness can itself be evaluated.

These tests also need to continue as the product changes. A different model, revised instruction, new search process or renamed data field can alter behavior. As Dev, Test, and Prod Still Matter: What Gets Deployed Has Changed explains, assurance needs to follow the system being deployed.

A useful starting point is one answer the product currently passes. Change the scope of its evidence without changing the number. Remove a fact required for its conclusion. Then restore complete information.

Check whether the product limits its claims when the evidence is incomplete—and answers directly when the evidence is sufficient.