One difficult request can reveal a real weakness in an AI system. It cannot prove that the obvious fix is the right one.
Suppose a travel planner misunderstands how a traveler expressed duration. Adding one phrase to a parser may make the failed case pass. The same system might still fail on a paraphrase, accept a route ordinal as a trip length, or behave differently on the next model run.
The useful question is therefore not only, “Can we fix this case?” It is, “What general distinction did this case expose, and how can we test the surrounding region?”
At Platix, we think of that process as counterexample-guided synthetic evaluation. A failure pattern becomes the seed for a privacy-safe family of tests. The family checks whether the improvement generalizes, whether nearby inputs remain correctly rejected, and whether stochastic behavior is stable enough for the product promise being made.
A failure is evidence, not a specification
A memorable failure exerts a strong pull on implementation.
An engineer can add a keyword, special route rule, or narrowly worded prompt instruction and immediately turn one red test green. That is useful feedback, but it does not establish that the system learned the right behavior. The patch may recognize the exact shape of the example rather than the underlying intent.
This is engineering overfitting. It can happen in regular expressions, prompts, routers, schemas, validators, recovery policies, and tool selection just as surely as it can happen in model weights.
Before changing the system, we try to state the exposed principle without referring to the original wording. For a duration problem, the principle might be:
> A duration explicitly expressed in ordinary language should survive intake, while route ordinals and calendar windows should not be converted into total trip length.
That statement is broad enough to guide implementation and precise enough to generate tests. It also gives reviewers something more durable to debate than one phrase.
Reconstruct the behavior, not the person
Production use can reveal valuable failure patterns. It does not turn a customer conversation into evaluation material.
The original prompt may contain destinations, dates, family details, accessibility needs, bookings, budgets, or combinations that remain recognizable after names are removed. Copying it into a repository, benchmark report, or writing tool creates a second data-retention path and weakens the boundary between product operation and engineering content.
The safer approach is abstraction followed by reconstruction:
1. Record the general capability that failed. 2. Remove the original language and all identifying context. 3. Change destinations, quantities, dates, route shape, and constraints. 4. Write a new minimal example from scratch. 5. Have a human verify that it tests the same principle without preserving the person.
Consider this purpose-built synthetic request:
> Plan an easy nine-day rail trip from Lyon to Annecy and Turin, with three nights already booked in Annecy.
It can test duration, route order, transport mode, pace, and a protected stay. It is not a paraphrase of a customer message, and it contains no account or production record.
The distinction matters. An anonymized prompt is still a real prompt. A synthetic fixture is a newly authored test.
Build a neighborhood around the seed
One synthetic case is still only one case. The next step is to map the behavioral neighborhood around it.
Useful families contain several kinds of variation.
Equivalent expressions change wording while preserving the intended structure. “Nine days” might become “eight nights,” “a little over a week,” or a date range that implies the same duration.
Boundary-positive cases use less common but still valid expressions. They test punctuation, modifiers, reordered clauses, multiple destinations, and constraints placed far from the duration phrase.
Negative controls look superficially similar but should produce a different interpretation. “Reach Annecy on night three” describes route timing, not total trip length. “Travel next week” supplies a calendar window, not necessarily a seven-day itinerary.
Structural variants preserve the language pattern while changing the planning shape. A one-base city stay, a linear route, a loop, and a route with a repeated destination can expose different downstream assumptions.
Conversation variants divide the same information across turns. One case may supply the duration first and a booked stay later. Another may explicitly correct an earlier route. These tests reveal whether the system updates one brief or accidentally starts another.
The family should include ordinary cases as well as difficult ones. A suite made entirely of linguistic puzzles can optimize the product for edge cases while making simple requests worse.
Let an LLM propose cases, not declare truth
An LLM is useful for expanding a test family because it can produce paraphrases and combinations much faster than a person can enumerate them.
It can be asked to vary syntax, distribute information across turns, add irrelevant detail, create near misses, and produce both valid and invalid examples. It can also suggest categories the initial author did not consider.
But generated cases are candidates, not ground truth.
The generator may misunderstand the intended distinction. It may create two examples that look different but exercise the same path. It may introduce an accidental ambiguity or quietly change a fixed constraint. If the generator and the system under test use the same model family, they may also share blind spots and stylistic habits.
Every promoted case therefore needs a reviewed contract:
- What facts are explicit?
- What may be inferred or recommended?
- What must remain unchanged?
- Is clarification acceptable or required?
- Which outputs are equivalent?
- What behavior would constitute a material failure?
Human review is especially important for negative controls. A case labeled “should reject” can make a good system look wrong if the label itself is mistaken.
Diversity helps reduce correlated blind spots. Test families should mix human-authored cases, deterministic transformations, LLM-generated candidates, different route shapes, and occasionally a different generator or judge. No single source should define the whole evaluation distribution.
Test invariants before prose
Generated itinerary language can vary without changing the quality of the plan. Evaluation should be strict about the contract and flexible about acceptable expression.
For the synthetic rail request, useful invariants might include:
- The trip contains nine days or the equivalent date allocation.
- Lyon appears before Annecy, and Annecy before Turin.
- The protected Annecy stay remains three nights.
- Rail remains the intended intercity mode unless the system identifies a conflict.
- Easy pacing affects day capacity rather than deleting a destination.
- A follow-up correction updates the active plan instead of creating an unrelated trip.
The test should not require one exact title, one restaurant, or one sentence from the assistant. Those details may be valid in several forms and may depend on provider evidence outside the evaluation boundary.
This separation also improves diagnosis. If the brief loses the booked stay, the failure belongs to intake or normalization. If the brief is correct but the route is reversed, the route planner owns it. If both are correct and the browser opens the wrong trip, the model is not the relevant component.
Stochastic systems need repeated evidence
Temperature is one source of variation in an LLM workflow. It is not the only one.
Sampling, model updates, context ordering, tool results, retry behavior, concurrency, and external provider data can all change an outcome. Even a nominally deterministic model configuration may not guarantee identical output across infrastructure or model revisions.
A single passing run therefore demonstrates possibility, not reliability.
Important semantic cases should run more than once. The report can track required-case pass rate, variation in extracted structure, clarification frequency, failure categories, and whether the same invariant fails repeatedly or intermittently. The first result should be preserved rather than retried until a pass appears.
Repetition should be proportional to risk and cost. Deterministic schema and policy checks can run on every change. A smaller semantic set can run several times on a regular cadence. Broader model and provider evaluations can run weekly or before a significant rollout.
The goal is not to eliminate all variation. It is to determine whether variation stays inside an acceptable behavioral envelope.
Hold out cases that did not inspire the fix
Test-family expansion protects the neighborhood around a failure. A held-out set protects the rest of the system.
Cases used to design a change should remain visible as regression tests, but they should not be the only evidence used to approve it. The evaluation also needs requests that did not motivate the implementation: different destinations, route shapes, durations, languages, correction patterns, and ambiguity levels.
This catches a common failure mode in agentic systems. A stricter validator may fix unsupported duration inference while becoming too eager to ask questions. A broader parser may accept useful phrasing while also converting every mention of “night three” into trip length. A routing improvement for place-led trips may quietly misclassify ordinary destination-first planning.
Positive cases, negative controls, and held-out cases answer different questions. Passing all three is much stronger evidence than repeatedly passing the original counterexample.
Promotion should be deliberate
Not every generated case deserves permanent residence in the regression corpus.
A candidate should be promoted when it represents a meaningful distinction, has a reviewed expected contract, is not redundant with an existing case, and can be evaluated at the correct system boundary. It should receive a stable case identity and a category that explains what it protects.
The stored artifact should contain the synthetic input, expected structured behavior, safe failure codes, and evaluation metadata. It should not contain the production prompt, user identifier, trip identifier, raw trace, or a note that allows someone to reconstruct the original conversation.
Over time, the suite itself needs maintenance. Duplicate cases can distort pass rates. Obsolete contracts can punish intentional product changes. A family generated by one model can become stylistically narrow. Periodic review should examine coverage by capability and route shape, not simply celebrate a growing case count.
Evaluation is not automatically training
This loop can improve the product without changing foundation-model weights.
Engineers may update a prompt, deterministic validator, workflow router, schema, retry rule, or clarification policy after reviewing the evidence. In that setting, the cases are guiding the surrounding system rather than training the model in the conventional machine-learning sense.
If labeled cases later update a learned router or fine-tuned model, terms such as hard-example mining or active learning may become appropriate. Until then, failure-driven evaluation expansion is the more accurate description of the operating practice.
The distinction keeps accountability visible. A generated test does not teach the system by itself. People still decide whether the expected behavior is correct, which component should change, and what evidence is sufficient for release.
The durable lesson
The best response to one AI failure is not one passing test.
It is a carefully constructed family that expresses the underlying principle, probes nearby boundaries, includes cases that should not match, survives repeated stochastic runs, and remains independent of the person whose experience first revealed the weakness.
Counterexample-guided synthetic evaluation turns a painful result into durable engineering knowledge. It helps the product solve the general case without turning customer conversations into training material or mistaking one lucky model response for reliability.
Loading comments...