A travel-planning agent touches unusually changeable systems. A single request can involve a language model, geocoding, place search, routes, opening hours, photos, persistence, and a user-facing itinerary. Running that complete chain for every test is slow, expensive, and difficult to reproduce.
It can also answer the wrong question. When a generated trip has a bad route, did the model misunderstand the request? Did place search return weak candidates? Did live traffic change? Did persistence lose a segment? A test that exercises everything at once tells us only that something went wrong.
A useful harness gives each boundary a smaller question.
Start with the artifact, not the interface
The first harness boundary should stop before maps and storage. Give the planning system a purpose-built synthetic request and inspect the structured artifacts it produces:
- The interpreted trip brief
- The ordered route and overnight allocation
- The number and geographic base of each day
- Fixed dates, transport legs, and traveler constraints
- The day-level planning directives that later guide place search
These artifacts are more stable than generated prose. A sentence can change style without changing meaning, while a missing overnight or reversed route is a material planning defect.
This is also where deterministic assertions are most useful. If a synthetic request explicitly fixes Porto before Lisbon, the harness can require that order. If it states eight nights, the route allocation must account for eight nights. If the request leaves pace open, the harness should not pretend that one particular pace is the only correct answer.
Make forbidden calls observable
Saying that a test does not need live travel providers is not enough. The harness should make those calls impossible or count them as failures.
For the prompt-to-brief and prompt-to-day boundaries, Platix runs with travel-provider credentials absent and persistence disabled. The report records that place, photo, route, and database operations remained at zero. Pending provider checks stay pending rather than being quietly replaced with invented facts.
This creates a useful contract: the harness may judge understanding and planning structure, but it may not claim that a restaurant is open, that a train is available, or that two stops are a precise number of minutes apart.
Provider behavior belongs in a different layer. Recorded fixtures can test response parsing and ranking repeatedly. A small, controlled live smoke can detect vendor contract changes. Neither should be confused with model comprehension.
Use several harness layers
One test mode cannot provide every kind of evidence. A practical travel-agent harness has at least four layers.
Contract validation checks schemas, corpus joins, stable case IDs, expected fields, and forbidden dependencies without calling a model. It is fast enough for ordinary code changes.
Semantic evaluation calls the configured model against synthetic cases but still blocks live travel providers and writes. It measures whether natural language becomes the expected structured plan.
Provider fixtures and controlled smokes test place, route, photo, and metadata behavior independently. Fixture runs are repeatable; live smokes are deliberately small because vendor data and cost can change.
Product regression and human adjudication exercise the released experience: authentication, progress, persistence, maps, editing, and whether the resulting trip is actually useful. This final layer is necessary, but it should not replace the narrower harnesses.
The layers form a diagnostic ladder. A contract failure points to code or fixtures. A semantic failure points to interpretation or planning. A provider failure points to external data or integration. A product failure can then be traced back through evidence from the earlier boundaries.
Build a corpus that can disagree with you
A corpus made only from easy examples becomes a demonstration, not an evaluation. It should contain different route shapes, durations, transport constraints, incomplete requests, corrections, and cases where clarification is the correct result.
The prompts must be synthetic or otherwise explicitly approved for evaluation. Production conversations should never be copied into a repository artifact. When real usage reveals a failure pattern, construct a new fixture from scratch with different wording, places, dates, and quantities. Preserve the behavior that matters without preserving the person.
Each fixture also needs an explicit contract. Some facts are fixed: named destinations, route order, booked stays, accessibility requirements, or flight windows. Others are flexible: which neighborhood makes the best base, how to divide open nights, or what title sounds natural. The harness should be strict about the former and allow bounded alternatives for the latter.
Preserve the first failure
Model behavior is stochastic. Repeating a failed case until it passes destroys useful evidence.
A semantic harness should save the first result, its model and prompt fingerprints, the corpus revision, stable failure codes, latency, and aggregate token usage. Repeated runs can then measure stability, but they should not rewrite the first attempt into a success.
Baselines need similar care. A run can fail required cases without proving a regression. Calling it a regression requires a comparable prior artifact with the same relevant corpus and evaluation contract. When that baseline is missing, the report should say so.
Know what the harness cannot prove
A provider-free harness can prove that the system understood a destination, retained a fixed train leg, allocated the requested nights, and produced day plans for appropriate geographic bases.
It cannot prove that the selected venues are currently open, the photographs still resolve, the route reflects live conditions, or the final itinerary feels good to a traveler. Those claims require provider checks and human review.
That limitation is a strength when it is explicit. The harness becomes trustworthy because it makes a narrow promise and supplies evidence for exactly that promise. Travel AI becomes easier to improve when every test knows where it should stop.
Loading comments...