The first module froze what RoutePilot is allowed to do. That contract gives us an unusual advantage when constructing training data: much of the answer is executable. We do not need a language model to decide whether a closed restaurant is feasible, whether a charger has the right connector, or which candidate wins a declared utility function.
The tempting pipeline asks a powerful teacher to write thousands of request and answer pairs. The safer pipeline creates a structured situation, derives its answer with code, and asks a teacher—if one is used at all—to vary only the surface language.
A RoutePilot scenario has four layers:
Generation follows that order. If prose comes first, an author must reverse engineer all implied constraints and may quietly add assumptions. If structured truth comes first, prose is one representation that can be checked against it.
template + seed
|
v
context + candidates + policy
|
v
hard constraints + utility weights
|
+---- deterministic oracle ----> expected choice
|
+---- renderer/paraphraser ----> request text
The expected choice is never typed into the generator. The generator calls the same oracle used by evaluation:
scenario["expected_choice_id"] = expected_choice(scenario)
This does not make the policy correct. It guarantees the label follows the policy that was actually declared. Reviewers can debate a 25-minute definition of “quick,” change it, version the dataset, and observe every affected label.
Uniform random candidates produce many boring cases. A useful template assigns roles deliberately:
The attractive-but-infeasible case is essential. Without it, a model can ignore hard constraints and still select the same option as the oracle. The feasible alternative is equally important: without it, selection accuracy says nothing about ranking.
Randomness can vary values inside those roles. It should not decide whether a scenario accidentally has no feasible answer unless that is the behavior being generated intentionally.
The committed generator uses a local pseudorandom number generator, a recorded seed, and a generator version. The same version and seed reconstruct the same fixture. That makes a release inspectable and makes a changed JSONL file easy to explain.
A reproducible dataset can still be narrow, biased, or unrealistic. A seed does not answer:
Those questions require a coverage report. Scenario count is only one line in that report.
Suppose one template produces 10,000 requests by changing cuisine names and detour limits. A random 80/20 split puts the same syntax, candidate roles, and field relationships in train and development. The model can learn the generator's fingerprint rather than the product behavior.
RoutePilot assigns each template_family to one split before generating rows.
The first scaffold uses explicit-cuisine and explicit-connector families for
training, while arrival-deadline and context-connector families go to
development. Both splits contain food and charging behavior, but the mechanism
that constructs each example does not cross the boundary.
This is still not a final holdout. A private holdout should contain additional families and remain unavailable to training, autonomous experiment agents, and repeated unlimited submissions.
Self-Instruct demonstrated that model-generated instructions can expand instruction-tuning data, with filtering of invalid and similar generations. Evol-Instruct explored controlled rewriting toward more complex instructions. These results motivate language generation, but they do not require RoutePilot to delegate its verifiable labels.
For this task, a teacher can:
A teacher must not:
Every retained paraphrase needs a constraint-preservation check. At minimum, compare the required typed arguments and numeric values with the rendered text and context. Store the raw generation record, model version, sampling settings, filter decision, and terms governing use of the output.
Benchmark contamination is often discussed as an unknowable property of a model's pretraining corpus. Course authors also control a simpler form: whether their own train and evaluation examples overlap.
Exact-string deduplication is not enough for generated data. Two rows can differ in every restaurant name and number while sharing the same template, candidate roles, and solution path. Family-level splits make that relationship explicit. Later modules should add similarity and contamination reports, but no detector replaces a split designed before generation.
The first generator deliberately stays small. Four template families produce
eight committed scenarios from seed 20260923. The CLI validates every row
before writing it, and tests confirm:
Eight scenarios cannot train RoutePilot. They establish the machinery that a larger release must use. Scaling comes after a coverage matrix, rights review, and baseline measurement show which examples are worth generating.
Module 3 freezes prompts, model revisions, parsing behavior, and sampling settings for base and teacher baselines. If the base model already satisfies the contract reliably, training may not be justified. If failures cluster around constraint extraction, clarification, or ranking, the scenario factory can produce targeted evidence instead of an undirected pile of synthetic text.
Lab: Open the lab — exercises you run against the benchmark on your own machine.