A6 — Interview drill: twenty-five questions

An append-only, durably persisted log of every interaction one Workflow Execution had with the Temporal Service: started with this input, scheduled this Activity, got this result, started this timer. The Service writes it; the Worker only proposes Commands.

It holds no program counter, no serialized Python stack, no heap snapshot. Temporal stores enough to reconstruct where execution logically is, never the position itself (modules 1 and 2).

Because reconstruction is re-execution. A Workflow Definition is deterministic if every execution of it produces the same Commands in the same sequence given the same input; on replay the SDK compares Command N against Event N, and a mismatch is a non-determinism error. A Workflow that could have gone either way from one history cannot be rebuilt: the SDK's only evidence of which way it went is that history (module 3).

The method by which a Workflow Execution resumes making progress: the SDK runs your function from line one and, at each await, matches the Command your code emits against the next recorded Event and hands back the recorded answer. When history runs out the code is live and the next Command is issued for real.

It does not re-run effects. Activities are not called again and timers are not re-waited; only the computation between awaits re-executes (module 2).

Workflow = decisions, Activity = effects. A decision is a function of the Workflow's input and its recorded history; anything that observes the world — clock, random source, network — is an effect, and lives in an Activity whose result is recorded and returned unchanged on replay.

In AgentRun, plan() is Workflow code and never calls a model. The model's answer is an effect; what you do with it is a decision (module 3).

Queries are read requests that cannot block and are never written to history. Signals are asynchronous write requests, recorded as WorkflowExecutionSignaled and accepted whether or not a Worker is alive, returning nothing. Updates are synchronous, tracked write requests: a validator may reject one and leave no Event at all, while an accepted one is recorded and returns a result or an error.

AgentRun uses all three — pause, resume, cancel_run, inject_context; status(); change_goal with a validator (module 5).

An Activity is one operation on the external world with one recorded result. A Child Workflow has its own history, steps, timers, Signals and retries. call_llm is an Activity, because retrying it means retrying the whole thing, which is correct. ResearchAgent is a Child Workflow, because thirty sources with a human review in the middle is a lifecycle: a step you can retry, a history you can open.

The test: would anyone want to inspect it on its own? If not, it is an Activity (module 6).

Because history has a size and an agent loop does not: one step with a model call and a tool call costs well over ten Events. A fresh run also starts cleanly on the current deployment.

The trigger is workflow.info().is_continue_as_new_suggested(), checked where the loop already turns — not a hard-coded step count, not the number 51,200. You pass what the next run needs as AgentState and get the same Workflow ID, a new Run ID and an empty history (module 6).

A lightweight, dynamically allocated queue that one or more Worker Entities poll for Tasks. It is a name created on demand. Workers poll outbound, so a queue with no pollers is stalled rather than broken.

It is not a Kafka topic. A Task is a short-lived instruction to some Worker, not a message you own at an offset, and the record of what happened is History. Any Worker may take any Task on a queue, so every Worker on it registers everything on it — which makes gpu-tools a contract about capability (modules 1 and 10, appendix A1).

Everything recorded up to the last Event, held by the Service: the history, the queued Workflow and Activity Tasks, and the durable timers, which keep counting with nobody alive. What does not survive is the unrecorded work of the in-flight Activity attempt — it may have happened, and nothing says so.

A returning Worker polls, replays the history to rebuild state, reaches the first Command with no matching Event and issues it. A different process finishes a run it never started.

None of them, for the effect you care about. Temporal records one terminal outcome per Activity Execution, so the Workflow sees a single result — but the Activity body may run more than once, and may partially complete more than once, before that outcome is recorded. Say it that way rather than reaching for a phrase like "exactly-once observation": it is not Temporal's own vocabulary, and an interviewer will make you defend it. The default Retry Policy makes execution at-least-once; maximum_attempts=1 makes it at-most-once.

Exactly-once effects are yours, through an idempotency key enforced by the service the Activity calls, keyed on the Workflow ID and step rather than the Run ID, which changes at every continue-as-new (module 4).

It never completes. Start-to-close bounds a single attempt, so every attempt is killed at five minutes with twenty-five minutes of work left; the Retry Policy schedules another and the clock resets. If the Activity does not call activity.heartbeat() at least every thirty seconds, each attempt dies at thirty seconds instead. About ten attempts fit in the hour, then schedule-to-close — the budget for the whole Activity Execution, retries included — fires and the Workflow gets an ActivityError caused by a TimeoutError.

Set start-to-close longer than the slowest legitimate attempt. Only a tool that checkpoints into heartbeat details makes progress under this configuration at all (module 4).

Because the next Workflow Task goes to a Worker running today's code, which replays a four-month-old history. If the branch changes which Commands are emitted or their order, replay fails with a non-determinism error — a Workflow Task failure, so the execution stays Open while the Service retries with backoff and nothing progresses.

Three ways out. Patch it: workflow.patched("insert-verify-step") records a marker for new executions and returns False for the old history, keeping that run on the old branch. Version it: declare AgentRun PINNED, so that execution finishes on the Worker Deployment Version it started on. Or move the change to the continue-as-new boundary, where a fresh history has nothing to disagree with. Auto-upgrade is not a fourth way — it still requires replay compatibility, kept by hand with patching (module 9).

The Service records WorkflowExecutionCancelRequested and schedules a Workflow Task; asyncio.CancelledError is raised at the Workflow's next await; your handler runs cleanup, re-raises, and the run closes as Cancelled. The tool keeps running: Activities must heartbeat to receive cancellations, because the request rides back on the heartbeat response.

So "the Workflow is Cancelled" is not "the work has stopped" — the job runs on, holding a reservation and a bill. Cancelling reaches no subprocess even when the Activity heartbeats; proc.kill() is code you write in the except (module 7).

The Service saw an attempt that never completed — it learns of the death at the heartbeat timeout, or at start-to-close if there is none — and schedules the next attempt. It cannot know whether the charge happened, so attempt 2 charges again.

ActivityTaskStarted is written along with the terminal Event, so intermediate attempts leave no Events and history names only the terminal attempt's Worker; the attempt count and last failure are mutable state, read with temporal workflow describe. The fix is an idempotency key enforced by the provider (module 4).

The execution is Open and, from the Service's side, healthy: the failure is a Workflow Task failure, retried with backoff, so the run neither fails nor advances. No Activity is running and nothing is duplicated: every Worker that takes the Task runs the same code and fails the same way. The TemporalReportedProblems Search Attribute finds every execution in this state.

Recovery is a code change: roll the fleet back to the build that produced the history, then export it into tests/histories/ as a replay-test fixture and reintroduce the change behind patched() (modules 3, 8 and 9).

Nothing. The Task waits in the queue, unstarted, until a Worker that registers AgentRun polls agent-runs; Tasks persist until one recovers.

Event 44 is the sticky-to-shared handoff: the Task was offered first on the dead Worker's Sticky Queue, its schedule-to-start window expired, and stickiness was disabled for this execution. temporal task-queue describe answers it: no pollers, backlog age rising (modules 2 and 10).

TimerFired at the Service's clock thirty seconds later, then WorkflowTaskScheduled to deliver it, then WorkflowTaskTimedOut when the sticky offer to the dead Worker expires, then WorkflowTaskScheduled again on agent-runs, where it waits.

All four are written while no Worker process exists anywhere. Killing every Worker does not pause time; it only means nobody is available to run the Workflow Task the fired timer scheduled (modules 1, 2 and 5).

Attempts 1 and 2 ran and left no Events of their own, and lastFailure describes the attempt before this one, which died on a missed heartbeat. Both may have done their work; history cannot say. The key in event 52's input decides whether attempt 3 doubles the effect.

Next is either ActivityTaskCompleted, or another heartbeat timeout — which schedules attempt 4, resuming from heartbeat details if the tool checkpoints them, or exhausts the policy and hands the Workflow an ActivityError with a TimeoutError cause (modules 4 and 8).

A Workflow Task is scheduled to process the cancellation; asyncio.CancelledError is raised at the Workflow's next await, once — a cancellation request is a state, not an event, so a Workflow that swallows it cannot be asked again. The Workflow runs the compensations list in reverse under asyncio.shield, re-raises, and closes as Cancelled. The finetune Activity learns of it on its next heartbeat response, and never if it does not heartbeat.

Terminate is the other shape: the Service appends WorkflowExecutionTerminated, no Workflow Task is scheduled, no Workflow code runs and no cleanup happens — the GPU stays reserved (module 7).

Whatever code it runs decides. A Worker on the build that produced the history replays cleanly, reaches the first Command with no matching Event, issues it, and the execution moves. A Worker on the build that wrote event 89 replays to event 8, emits StartTimer where history has an Activity being scheduled, and fails the Task again, indefinitely, while the execution stays Open.

Read the message as a location: the code asked for a timer and at event 8 history has something else. Somebody put a sleep before an existing Activity call (modules 3, 8 and 9).

AgentRun declared PINNED under Worker Versioning, so an execution completes on the version it started on and a bad build is rolled back by pointing set-current-version at the previous build. The loop checks is_continue_as_new_suggested() where it already turns, drains with wait_condition(workflow.all_handlers_finished), and continues-as-new with AgentState — the boundary at which a pinned run upgrades, which upstream marks Public Preview.

Two constraints: moving is lazy, so an agent parked at a 72-hour checkpoint learns of a new target version only when it executes a step; and the new version must accept the old run's continue-as-new input, so AgentState stays additive (modules 6, 8 and 9).

In your own system of record — rows your Activities write, keyed by the same idempotency keys they enforce, outliving the Namespace's retention. History answers what ran: this attempt started, that one completed with this payload. It does not answer whether the result was good, what the provider charged, or what the customer now owns.

That gap is what Temporal is built around: every ActivityTaskStarted with no terminal Event is an attempt that may have done its work before the Worker died, with no Event either way (module 10, lab 8.5).

Ask, do not count: check is_continue_as_new_suggested() at the same point in the loop each time and snapshot AgentState. Hard-coding 51,200 is wrong, because a step's Event cost changes the moment you add a step to the loop.

Carry context_summary, not the transcript, and claim-check large tool output to object storage behind a URI. Carry processed_command_ids across the boundary or the first retried Signal in the new run is an undetected duplicate. Delegate multi-step work to ResearchAgent children (modules 4, 5 and 6).

Match the tool to the Workflow's lifetime. A short Workflow that finishes before the next deploy is PINNED and never needs a patch. One that spans deploys with no continue-as-new is AUTO_UPGRADE plus patching, because it moves to each new version and replay-safety becomes your job. A long-lived agent that continues-as-new is PINNED and upgrades at the boundary. A fleet on rolling deploys uses patching, because rolling is incompatible with versioning.

Patching costs scar tissue: three deploys per change — patched(), then deprecate_patch(), then nothing — with a replay test gating each step, and the newest branch at the top of the if/elif chain (module 9).

From history: what happened in order, with inputs and results; which attempt is outstanding — the last ActivityTaskScheduled with no terminal Event; its timeouts, Retry Policy and idempotency key; how many attempts there have been, from describe rather than the log; and what a compatible Worker does next, which is replay to the first Command with no matching Event and issue it.

Not from history: the Workflow's own variables, because you do not have the code; and whether the outstanding attempt's side effect reached the world. That is lab 8.5's lesson — the two places your answers are wrong are the two things history does not record (modules 2, 8 and 10).