Capstone — A Durable Agent Runtime

On this page
Open the guided lab ↗

This page is original to the course. It restates the exam, gives you the shape a correct answer takes, and puts a real production workflow beside it so you can compare box by box.

The exam

Design AgentRun without opening the documentation. It must:

If you have done modules 1 through 10, none of these is new. The exam is whether you can put them in one picture and defend every box.

The shape that falls out

Swipe to explore diagram →
Client AgentRun Workflow id agent-42 Signals Update Query state pause, resume, cancel_run, inject_context change_goal status step, status, goal, context_summary, attempts, processed_command_ids plan() call_llm execute_tool evaluate ResearchAgent Child Workflow continue_as_new(AgentState) every N steps agent-workflows cpu-tools gpu-tools browser-tools
The AgentRun architecture that falls out of the requirements.

Around the box: a RetryPolicy on every Activity with bad arguments marked non-retryable; a Start-to-Close and Schedule-to-Close timeout on every Activity; a heartbeat on anything that can run longer than half a minute; Worker Versioning with executions pinned and upgraded at continue-as-new; one Task Queue per compute class; replay tests over a fixture of exported histories; and a failure-injection suite that crashes Workers, times out Activities, duplicates executions and cancels runs.

The failure column

The defense document requires one line per box. This is the table you fill in.

Box A Worker dies here On retry What history records
plan() Replayed from history on the next Worker; nothing is lost because nothing happened outside history Not applicable, it is a decision Commands only
LLM Activity The attempt fails at Start-to-Close or heartbeat; the Activity is retried with the same operation key A recorded answer can be returned from the gateway ledger. If the provider charged but the gateway crashed before recording the response, reconcile the ambiguous outcome or rely on provider-side idempotency before calling again Scheduled, Started, Completed with the result
Tool Activity Heartbeat timeout fails the attempt quickly; the next attempt receives the last delivered heartbeat details Activity code reads the details and implements safe resumption; external effects still need idempotency Same, plus heartbeat details in the pending Activity view
ResearchAgent child The child replays on its own; the parent does not notice The child's own policy ChildWorkflowExecution events in the parent
Pause wait The condition is Workflow state; nothing polls while paused Not applicable WorkflowExecutionSignaled
continue_as_new The new run starts atomically as the old one closes Not applicable WorkflowExecutionContinuedAsNew

If you cannot fill a row from memory, that module is where your next hour goes.

The LLM row has an ambiguous commit window: the provider may finish and charge, then the gateway may crash before it stores the answer under the operation key. A retry finds an in-progress key but cannot infer the provider's outcome from that fact alone. A stable key prevents duplicate calls only if the provider honors it, the gateway controls the billable operation, or a durable provider lookup lets you reconcile the first call. The lab's fake ledger has no such crash window, so its one-bill test proves the fake's behavior, not a general exactly-once guarantee.

Swipe to explore diagram →
The ambiguous commit A stable operation key survives the retry; the provider outcome may not. 1 · GATEWAY Store key abc status = IN_PROGRESS 2 · PROVIDER Compute the answer Charge for the call 3 · GATEWAY CRASH Response is lost No completed row saved 4 · RETRY WITH KEY abc The gateway sees IN_PROGRESS, but cannot tell whether the provider charged. Before another paid request, resolve the unknown outcome: Provider honors the sameidempotency key Provider offers durablerequest/result lookup Gateway controls thebillable operation
The provider may charge before the gateway records the response. A retry with the same gateway key sees an unknown outcome until the provider or a reconciliation path resolves it.

For the tool row, Temporal makes the last delivered heartbeat details available to the retry. It does not continue the container job automatically: your Activity must read the details, choose where to restart, and make repeated work safe.

Task Queues as compute lanes

Temporal decides where execution logically is. Kubernetes decides where processes run. They are different layers, and the agent runtime uses both. Workers for gpu-tools run on GPU nodes and poll only that queue. Workers for browser-tools run in a sandbox with a headless browser. The Workflow does not know or care which host picks up a task; it only names the lane. Backlog on a lane is the signal to scale that fleet. Appendix A1 has the details.

Nexus, in one paragraph

When the training team, the inference team and the agent team each own a Namespace, Nexus is how a Workflow in one calls an operation in another with durability, retries, cancellation and visibility intact, instead of an HTTP call that forgets everything on failure. Know that it exists. Do not build the capstone on it. Appendix A2.

The real one

The engine that rendered these slides runs its multi-leg course jobs on Temporal. The design is in docs/design/temporal_agent_loop.md in the mini_lecture repository.

Swipe to explore diagram →
POST /course-jobs workflow id per job Initial Activity produce lectures; retry + heartbeat ProductionYield schedule another Activity CheckpointPause durable Signal wait approve / revise Signal from the API Resume → finish artifacts in the job store
The real one: the course-job workflow inside the engine that rendered these slides.

Compare it with your design:

What it deliberately leaves outside Temporal: the job store, the agent state file, the generated artifacts, usage telemetry and the verifier database. Those are the product-facing and quality systems of record. Temporal history answers what execution happened. It does not answer whether the lecture was good. That boundary is the answer to the tenth question.

Diagnose from history alone

You will be given a broken AgentRun and you may not read the code. From the Web UI or temporal workflow show, answer:

  1. What happened?
  2. What did Temporal think happened?
  3. What side effects may have occurred?
  4. What will happen next?
  5. What will happen if every Worker disappears now?

Appendix A3 maps each question to the surface that answers it.

A ninth contract question, for the product

Every box above answers what survives, what repeats and what is recorded. A product on top of the runtime needs one more answer per box: what does the user see, and from which store? The Workflow's visibility list is the operator's view, not the product's; the product reads its own projection. Appendix A4 works this through on a different scenario, an evaluation console, so the skill transfers rather than the diagram.

The ten questions

  1. What exactly survives when a Worker crashes?
  2. How does replay reconstruct Workflow state?
  3. Why must Workflow code be deterministic?
  4. Why can an Activity execute twice?
  5. What is the difference between Activity retries and Workflow replay?
  6. Signal versus Query versus Update?
  7. Activity versus Child Workflow?
  8. When should you Continue-As-New?
  9. How do you safely deploy changed Workflow code?
  10. Where does Temporal end and Kubernetes, Kafka and Postgres begin?

What the lab will test

Before building the final runtime, state which mechanism protects each requirement: crash recovery, duplicate effects, unbounded history, human input, cancellation, and code upgrades. Use the capstone evidence ledger to test your mapping.

Try it

labs/11-capstone-durable-agent-runtime/: exercise 11.1 assembles AgentRun from the module labs with separate Task Queues; 11.2 runs 200 steps and checks history bounds and Search Attributes; 11.3 runs the failure-injection suite and deploys v2 mid-run; 11.4 is the one-page architecture defense and the ten questions; 11.5 puts a product on top of the Workflow — appendix A4's evaluation console, with a projection, a cursor-paginated list, an SSE stream and an approval gate.