Event History and Replay

The most important hour

Everything else in Temporal follows from one mechanism. The Service keeps a log of what one execution did. A Worker can rebuild that execution's in-memory state from the log — any Worker, on any machine, with no memory of the original. If this hour clicks, the rest of the course stops being surprising: why Workflow code has rules (module 3), why Activities retry the way they do (module 4), why long agents must periodically start over (module 6), why deploying new code is delicate (module 9). Keep your module 1 paragraph beside you.

Commands go up, Events come down

Two nouns, in upstream's exact words, because you will read them off a screen for the rest of the course.

A Command is a requested action issued by a Worker to the Temporal Service after a Workflow Task Execution completes. Each time your Workflow Definition makes an API call to execute an Activity or start a Timer, it doesn't perform the action directly. Instead, it sends a Command to the Temporal Service — ScheduleActivityTask, StartTimer, CompleteWorkflowExecution, and so on. Commands are batched and sent as part of the Workflow Task completion request, after the Workflow Task has progressed as far as it can with the Workflow function.

Events are created by the Temporal Service in response to external occurrences and Commands generated by a Workflow Execution. An Event History is an append-only log of Events for your application. It is durably persisted by the Temporal Service, enabling seamless recovery of your application state from crashes or failures, and it also serves as an audit log for debugging.

So the Worker never writes history. It proposes; the Service validates each Command and records the Event. Some Events are the direct result of a Command (ScheduleActivityTask → ActivityTaskScheduled, StartTimer → TimerStarted). Others are indirect — recorded when a Worker starts or finishes a Task, a timer fires, or a Client acts (ActivityTaskStarted, ActivityTaskCompleted, TimerFired, WorkflowExecutionSignaled). Mechanically, a Workflow's await is a Command waiting for its Event.

The unit of work: a Workflow Task

A Workflow Task advances a Workflow Execution by one step. The Service creates and schedules one whenever something that might affect the Workflow's state happens: the execution is started, a Signal or Update arrives, an Activity completes (successfully or with a failure), a Timer fires, a Child Workflow completes, or a previous Workflow Task failed and needs a retry. The Workflow Task bundles together all new events that have occurred since the last Workflow Task completed.

Each Workflow Task leaves a triplet in history — WorkflowTaskScheduled, WorkflowTaskStarted, WorkflowTaskCompleted — with its Commands' Events following the Completed. That is why a history has far more events than your code has awaits: each costs one Command-Event plus the triplet that wakes the code afterward.

Workflow Task Execution is typically very fast (milliseconds). No actual I/O occurs during it; Activity results come from history. The time spent in a Workflow Task is unrelated to how long Activities or Timers take. For the agent runtime: deciding "call the model again" takes milliseconds; the model call it schedules may take a minute, and if the process dies during that minute nothing in the Workflow was mid-flight.

An Activity Task runs one attempt of an Activity. ActivityTaskScheduled marks the Service queuing it; ActivityTaskStarted marks a Worker picking it up; ActivityTaskCompleted (or a failure or timeout event) marks the Worker yielding back. A detail you will trip over: while the Activity is running and retrying, ActivityTaskScheduled is the only Activity-related Event in history — ActivityTaskStarted is written along with the terminal Event, because the attempt count is an attribute of it.

The module-1 history, annotated

Here is the history module 1's AgentRun produces, started the way Lab 2's starter starts it (a 20-second timer; module 1's lab uses 30), when you kill the Worker during the timer and leave it dead until after the timer fires: one row per Event, with the Command or external cause of each. Event IDs are what the Web UI and temporal workflow show --workflow-id agent-42 --output json (as of 2026-09; see upstream page) show; Lab 2.1 has you verify the numbering. Rows 9, 13 and 14 mention a sticky queue: a queue that only one Worker polls, which the Service uses to send a Workflow's next Task back to the Worker that still holds that Workflow in memory, so it need not replay. "Why two Workers agree", below, explains it, and Lab 2 shows it in every history.

id  Event                        caused by
──  ───────────────────────────  ─────────────────────────────────────────────────
 1  WorkflowExecutionStarted     Client: start_workflow("Summarize the Temporal docs"); input stored here
 2  WorkflowTaskScheduled        Service: consequence of 1; Task placed on "agent-runs"
 3  WorkflowTaskStarted          Worker A polled the Task (identity pidA@host recorded)
 4  WorkflowTaskCompleted        Worker A: plan() ran; RespondWorkflowTaskCompleted, carrying ↓
 5  ActivityTaskScheduled        Command ScheduleActivityTask(call_llm, prompt, "agent-42:0:9c1f…")
 6  ActivityTaskStarted          Worker A dequeued the Activity Task (written at close, attempt=1)
 7  ActivityTaskCompleted        Worker A: RespondActivityTaskCompleted → "answer to: Summarize …"
 8  WorkflowTaskScheduled        Service: 7 may change Workflow state → wake it
 9  WorkflowTaskStarted          Worker A (sticky queue; state still cached)
10  WorkflowTaskCompleted        Worker A, carrying ↓
11  TimerStarted                 Command StartTimer(20s)   ◀── kill -9 Worker A here
    ── no Worker process exists for the rest of the 20 s. The Service keeps going: ──
12  TimerFired                   Service clock, 20 s after 11. NO Worker is involved or required
13  WorkflowTaskScheduled        Service: deliver 12. Offered first on A's sticky queue — and A is dead
14  WorkflowTaskTimedOut         sticky schedule-to-start expired (10 s on the Python Worker;
                                 upstream quotes 5 s) → stickiness dropped
15  WorkflowTaskScheduled        reoffered on "agent-runs", where any Worker may take it.
                                 It waits here, unstarted, until one exists
    ── start a Worker again ──
16  WorkflowTaskStarted          Worker B polled it (identity pidB@host — a different process)
17  WorkflowTaskCompleted        Worker B, after REPLAYING 1–15 to rebuild state, carrying ↓
18  WorkflowExecutionCompleted   Command CompleteWorkflowExecution("answer to: Summarize …")

Events 12 to 15 are the ones worth staring at: they were all written while no Worker process existed anywhere. The timer is the Service's, not the Worker's. Killing every Worker does not pause time; it only means nobody is available to run the Workflow Task the fired timer scheduled, so that task waits. That is the whole architectural boundary in five events, and Lab 1 has you produce them. The sticky-to-shared handoff really does leave the WorkflowTaskTimedOut footprint shown above rather than a single re-scheduling (measured on the labs' dev server, Temporal Server 1.31.2, 2026-09; upstream describes the rescheduling but not its exact footprint, so check yours). Now the other crash shape: kill the Worker inside a Workflow Task instead of during the timer, and the Service learns of it only when the Workflow Task Timeout expires — 10 seconds by default (as of 2026-09; see upstream page) — recording WorkflowTaskTimedOut before scheduling a fresh Task.

Upstream's compact table for the Python walkthrough is worth memorising:

Everything else — string formatting, arithmetic, an if on a value you already have — is an internal step the Worker runs without contacting the Service.

Replay, step by step

A Replay is the method by which a Workflow Execution resumes making progress. During a Replay the Commands that are generated are checked against an existing Event History. That is the definition; here is what it looks like from inside Worker B at event 16.

Worker B receives a Workflow Task containing the complete Event History and has no cached state for agent-42. It calls AgentRun.run("Summarize the Temporal docs") from line one — the input comes from event 1. plan() runs again, on the same state, and returns the same Step with the same key. At await workflow.execute_activity(call_llm, ...) the SDK creates a ScheduleActivityTask Command but does not issue it. It looks at the next unconsumed Events, finds ActivityTaskScheduled for this Activity type (5), ActivityTaskStarted (6) and ActivityTaskCompleted with the answer (7). The Activity already ran, so the Command is matched and discarded and the await resolves immediately with the recorded result. call_llm is not called again; nobody is billed twice.

At await asyncio.sleep(20) the SDK creates StartTimer, finds TimerStarted (11) and TimerFired (12), and resolves at once. Twenty seconds do not pass a second time.

At return answer the SDK creates CompleteWorkflowExecution and finds no matching Event. History is exhausted; the code is live. This Command is issued for real in Worker B's RespondWorkflowTaskCompleted, the Service records events 17 and 18, and handle.result() returns an answer that Worker B never computed. Had the program had a second Activity after the timer, that await would have been the first live Command instead — the mechanism is the same.

Replay is not re-execution of effects. Activities are not re-run; their recorded results are returned. Timers are not re-waited. Only the pure computation between awaits is re-executed, and fed identical inputs it produces identical local variables. Lab 2.1 counts Activity invocations across a forced replay: the counter does not move.

How await asyncio.sleep becomes a durable timer

The table says asyncio.sleep issues StartTimer, and the replay above says it resolves from TimerFired. Your code passes no callback, so it is fair to ask how the Workflow ever learns the timer fired. The callback is inside asyncio.sleep itself. In CPython's asyncio it is, in essence:

future = loop.create_future()
loop.call_later(delay, set_result, future)   # the callback: resolve the future later
await future

loop is whichever event loop is running the coroutine, and inside a Worker that is not asyncio's. The Python SDK runs Workflow code on its own class, a subclass of asyncio.AbstractEventLoop, and its call_later never starts an operating-system timer. It gives the timer a sequence number, adds a StartTimer Command carrying that number and the duration, and parks the callback in a table of pending timers keyed by the number (Python SDK 1.32.0, temporalio/worker/_workflow_instance.py; these are internal names and may move). The coroutine is now waiting on a future that nothing in the process will resolve, the loop has no more ready work, and the Workflow Task completes with the Command. That is event 11.

The Service does the waiting, and the notification is a Workflow Task. At the deadline the Service records TimerFired (12) and schedules a Workflow Task (13) to deliver it. Your Python code never reads Events: the SDK's Rust core, under temporalio.bridge, turns them into activation jobs, and TimerFired arrives as a fire_timer job carrying the same sequence number. The SDK pops the parked callback, the future resolves, and await asyncio.sleep(20) returns.

So which part is durable? Only what is in the Service: the timer's existence (11) and its expiry (12). The future and the pending-timer table live in one Worker's memory and die with it. They do not need to survive, because replay rebuilds them. When Worker B replays, the same line calls call_later again, which produces StartTimer with the same sequence number; the SDK matches it against event 11 instead of sending it, and parks a fresh callback. Then one of two things happens:

• History already holds TimerFired. The same activation carries fire_timer for that number, the fresh callback runs at once, and the twenty seconds do not pass a second time. That is the replay walked through above.
• The timer has not fired yet. The future stays pending, and the replay stops there with nothing new to send. When the Service's clock reaches the deadline, event 12 arrives as a Workflow Task exactly as it would have for Worker A. Measured on the labs' dev server with Lab 1's 30-second timer: the only Worker was killed just after TimerStarted, a fresh one was started, and a __stack_trace Query sent at +3 s made it replay the Workflow (the answer came 10 s later, once the dead Worker's sticky queue timed out). No Event was written by that replay; TimerFired came at +30.18 s and the Workflow completed on the fresh Worker at +30.23 s (2026-09-17).

Compare time.sleep(20). It blocks the thread and produces no Command, so the Service never learns a timer exists, and a crash loses the wait along with the process. That is one reason the sandbox refuses it (module 3).

What your code makes durable

There is no "save" call in Temporal. You never mark a variable or a result as something to persist. What becomes durable is decided by which SDK calls your code makes: each one that crosses from the Worker to the Service is recorded as an Event, with its arguments and its result. Everything else, self.step and every local variable included, is never stored; replay recomputes it. Upstream's three-row table above, extended to the calls this course uses (payload field names checked against the Python SDK 1.32.0 protobuf definitions):

To make a value durable in Workflow code, then, make it the result of something recorded: compute it in an Activity, receive it through a Signal or Update (module 5), or carry it into the next run as continue_as_new input (module 6). In the module-1 history, self.context_summary = answer is stored nowhere. answer is, as event 7's result, and replay re-runs the assignment. No Event holds self.step.

Two traps in Workflow code:

• Raising is not always failing. Only a Temporal failure, such as ApplicationError, fails the Workflow Execution. Any other exception fails the Workflow Task, which the Service retries, and the execution stays open (as of 2026-09; see upstream failures reference). The Worker option workflow_failure_exception_types, empty by default, adds exception types that fail the execution instead.
• Some calls look like effects and write nothing. workflow.now(), workflow.random() and workflow.uuid4() produce no Events: the time comes from the Workflow Task, and the generator is seeded so replay reproduces the same values. A Query reads in-memory state and writes nothing; the Query in the timer measurement above added no Event.

Activity code is recorded only at its end:

• The return value becomes ActivityTaskCompleted's result.
• A raised exception is retried. While it retries, ActivityTaskScheduled is the only Event for that Activity, and only the final outcome, such as ActivityTaskFailed, is written.
• activity.heartbeat(details) writes no Event. The Service keeps the latest details on the pending Activity, where describing the execution shows them, and hands them to the next attempt as activity.info().heartbeat_details; in history they surface only inside a timeout failure, as last_heartbeat_details. A Worker that crashes may not have delivered its last heartbeat (module 4).
• Everything else the Activity does (a database row, a file, an API call) is durable only if that system makes it so. Temporal records the Activity's outcome, never its side effects; that gap is why module 4 gives every call an idempotency key.

Every argument and result in the table is stored in history, so keep payloads small and pass references to large data. "History has a size", below, gives the limits.

Why two Workers agree

Same code plus same history yields the same sequence of Commands, hence the same state. That equivalence is the contract, and it is why Workers are disposable.

Sticky Execution is the optimisation that makes the common case cheap. Workers cache the state of the Workflow they execute. Once a Worker picks up the first Workflow Task, it continues polling the original Task Queue but also begins polling an additional queue the Service shares exclusively with that Worker — the Sticky Queue, with an automatically generated name. Subsequent Workflow Tasks go there, and the Worker applies only the new Events to its cached state instead of replaying from scratch. If the Worker fails to start a Workflow Task in the Sticky Queue shortly after it's scheduled (within a short window: the upstream docs quote five seconds; the Python Worker sets its sticky schedule-to-start timeout to ten seconds, which is what Lab 2 observes), the Temporal Service disables stickiness for that Workflow Execution and reschedules the Workflow Task in the original queue, allowing any Worker to pick it up. On the labs' dev server that wait shows up only when no Worker is polling at all: with another Worker polling when the timer fired, the Task reached it about 0.02 s after TimerFired, with no WorkflowTaskTimedOut (Lab 1.5, measured 2026-09-16). If a Workflow Task fails, the Worker evicts that execution from its cache and the Task goes back to the original queue.

So Worker A in the lab was receiving deltas, and Worker B rebuilt from scratch, and you cannot tell from outside which happened. Both reach the identical point, or the design is broken — module 3 is about what your code must promise to keep it unbroken.

Not a DAG engine

Notice what is not stored here. A DAG engine stores node states — task X succeeded, task Y pending — and your logic is the shape of the graph. Temporal stores a replayable program plus a history of its interactions with the outside world. Your logic is ordinary code.

That is why Workflow code may contain if, loops, try/except, variables that accumulate across steps, child workflows, timers raced against Signals, and concurrency. The Service needs only that the code, re-run against the same history, makes the same requests in the same order. For an agent this is the difference between "a pipeline of stages" and "a loop that decides the next step from everything the model has said so far" — the second is what you want, and it is a Workflow.

History as the operator's instrument

Because the log is complete, an operator can answer five questions about a stuck execution without reading the code: what happened; what did Temporal think happened; what side effects may have occurred (every ActivityTaskStarted is a possible effect, even without a Completed); what happens next (the pending Commands); and what if every Worker vanished now (everything up to the last recorded Event survives). Module 10's exam asks exactly this of a Workflow you have never seen.

History has a size

The Service logs a warning after 10,240 Events or 10 MB, and the Workflow Execution is terminated when its history exceeds 51,200 Events or 50 MB, contains more than 2,000 Updates, or more than 10,000 Signals (as of 2026-09; see upstream page). There is no limit on how long a Workflow may run — only on how much it may record.

Count the annotated history: one Activity call costs about six events (three for the Activity, three for the Workflow Task that delivers its result). An agent loop calling one tool per step passes the warning threshold in under two thousand steps and the hard limit in under ten thousand. A million-step agent cannot be one history — Continue-As-New, module 6. For now, notice the number is finite and you can compute it.

What the lab will test

Before restarting a Worker, mark the last recorded Event and the next Command the Workflow will need to issue. In Lab 2, compare your prediction with the real history and Activity invocation counter.

Try it

Go to labs/02-event-history-and-replay/. All four exercises use the module-1 Workflow on the dev server.

• 2.1 Run agent-42 again and annotate its real history: for every Event, the Command or external cause, as above. Check the sticky handoff footprint and the Worker identities.
• 2.2 With the timer pending, predict the next three Events. Then check.
• 2.3 Two-Worker handoff: stop Worker 1 after event N, start Worker 2, diff the histories — identical prefix, continuation appended.
• 2.4 Hand-replay: given a history JSON and the code, say what the Worker does next and which await — or return — is the first to go live.

Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.