Three trip-planning tests failed in the same weekly run. Each asked for a different kind of journey, and each stopped at the same frustrating question: how long should this trip be?
The requests had already answered it.
At first glance, this looked like an AI comprehension problem. The model had apparently missed ordinary duration language. The actual failure sat one layer later. The AI had understood the duration, but a deterministic validator rejected its answer.
That distinction matters. Changing the model would not have repaired the system. We needed to repair the contract between probabilistic interpretation and deterministic safety checks.
What the weekly lab evaluates
Platix runs a synthetic Trip Shape Lab against the early stages of trip creation. The lab stops before travel-provider searches and before database writes. It asks whether a planning request can become a coherent trip shell:
- Did the system preserve the requested geography and route order?
- Did it understand the trip length and fixed dates?
- Did it retain stays, constraints, and transport details?
- Did it ask for clarification only when a required decision was genuinely missing?
- Could it produce a usable route and a geographic base for every day?
The corpus contains purpose-built synthetic requests with explicit expectations. That gives us repeatability without exposing user conversations or spending money on place details and photos merely to test prompt understanding.
In this run, 12 of 15 hard cases passed. The three failures looked unrelated until we compared their intake results.
One failure wearing three outfits
Consider three newly constructed synthetic examples:
- “Plan a 12-night loop through northern Spain.”
- “Give us 6 quiet days around Lake Como.”
- “Build a relaxed week in Porto.”
All three communicate a duration. They do it in different grammatical forms: a hyphenated night count, a modifier between the number and the unit, and a descriptive week.
The model interpreted those forms correctly. It proposed structured durations for the trip brief. The validator then checked whether the original request contained explicit evidence for those values.
That check recognized simple phrases such as “six days.” It did not recognize the broader forms above. Because the duration appeared unsupported, the compiler removed it rather than allowing a model inference to become a traveler-owned fact. Intake then did exactly what it had been designed to do when duration was absent: it asked the traveler for clarification.
The safety rule was reasonable. Its language boundary was too narrow.
Why not simply trust the model
A trip brief becomes the foundation for route allocation, dates, lodging nights, and every planned day. Allowing an unsupported number into that structure can create a polished itinerary for a duration the traveler never requested.
The validator therefore serves an important purpose. It distinguishes facts supplied by the traveler from recommendations supplied by the planner. Removing that boundary to make three tests pass would trade a visible clarification for quieter and more damaging assumptions.
The correct fix was to widen the accepted evidence carefully. The duration detector now recognizes bounded natural-language forms including hyphenated counts, short modifiers, and common descriptions of a week.
The negative cases are just as important. “Reach Geneva at night 3” describes route timing, not total duration. “Travel next week” describes a calendar window, not a seven-day trip. Both must remain outside the duration boundary.
The regression tests we wanted
We added tests at the contract boundary rather than testing only the text recognizer in isolation.
For each accepted phrase, the test creates a synthetic model patch, runs it through the same normalization path used by trip intake, and confirms that the duration survives. Separate tests confirm that route ordinals and calendar language do not acquire false duration meaning.
The focused compiler and shape suites passed 129 tests after the change. The frozen hard corpus and all 44 product-authored landing starters also passed their deterministic validation. Those checks prove that the expanded rule is internally consistent and that the corpus contracts remain intact.
They do not prove that a live model will always produce the same result. That requires another semantic run after deployment, and repeated runs are more meaningful than a single pass when the model itself is stochastic.
A lab needs an honest baseline
The investigation exposed a smaller reporting problem too. The weekly job had no clean baseline artifact available for comparison. It correctly failed because required semantic cases failed, but it could not tell us whether those cases had regressed from the previous accepted baseline.
Those are different statements. “Three cases failed this week” is supported. “Three cases regressed this week” requires a comparable baseline.
Evaluation infrastructure needs the same precision as product code. A useful report should identify the code revision, model, corpus hash, pass rate, failure codes, provider boundaries, and baseline it actually compared. When one of those inputs is absent, the report should say so plainly.
The larger lesson
Reliable AI products are systems, not model calls. The model can interpret a request correctly while a validator discards the result. A permissive validator can also accept confident nonsense. Quality lives in the handoffs between those components.
The most useful evaluation cases exercise those handoffs. They ask whether the model understood the request, whether deterministic code preserved the supported parts, whether unsupported assumptions stayed out, and whether the product recovered helpfully when evidence was missing.
That is why three failures were good news in the end. They did not merely lower a score. They revealed one precise boundary that could be improved without weakening the protections around it.
Loading comments...