Activities and Failure Semantics

Where your code touches the world

Workflow code is constrained, so it is hard to get wrong. Activities are where your code touches the world — the model call, the tool run, the row written — and the world does not offer exactly-once. This page is about what Temporal promises about an Activity and what it cannot.

Upstream's definition, kept exact: "An Activity is a normal function or method that executes a single, well-defined action (either short or long running), such as calling another service, transcoding a media file, or sending an email message. Activity code can be non-deterministic." Temporal recommends that it be idempotent, and its own list of typical Activities includes "a read that should be memoized, like an LLM call" — the agent's model call is a textbook Activity.

When the function returns, the Worker sends the result to the Temporal Service, which appends an ActivityTaskCompleted Event to history. Everything below lives in the gap between "the function returned" and "the Event was recorded".

The charge that may happen twice

A Worker polls the Task Queue, accepts the Activity Task for charge(order_id), and calls the payment provider. The card is charged. Then the Worker process dies — OOM, kill -9, a node drained — before it reports completion.

The Service saw an attempt that never completed and scheduled the Activity again. It could not know whether the charge happened; neither can you, without a key. Upstream states the guarantee precisely: "For an Activity with a Retry Policy that allows retries, Temporal guarantees that the Activity will be observed as completed exactly once. However, the Activity may be executed multiple times and may even partially complete more than once during this process."

Assume every Activity can execute more than once. Temporal guarantees durable orchestration — the Activity is scheduled, retried and observed as completed exactly once in history — not exactly-once execution of your external side effect. With the default Retry Policy, "Activities follow an at-least-once execution model". maximum_attempts=1 "means a single execution attempt and no retries", which is at-most-once: the charge either happened or the Activity failed. Neither setting is exactly-once. charge(order_id) is the canonical illustration; execute_tool with a key is the same problem in the agent's clothes. For agent infrastructure the dangerous duplicates are not send_email but "launch a GPU sandbox", "start an agent session" and "submit a fine-tune job" — resources you pay for and must later tear down.

Design for re-execution

"An Activity is idempotent if multiple Activity Task Executions do not change the state of the system beyond the first Activity Task Execution." A non-idempotent Activity does not break the platform — it breaks your application, quietly.

The standard tool is an idempotency key, and note where it is enforced: "These are enforced by the service you are calling from your Activity, not by the Activity itself." Upstream suggests "a combination of the Workflow Run ID and the Activity ID as an idempotency key since this is guaranteed to be consistent across retry attempts but unique among Workflow Executions." For an agent that will continue-as-new (module 6), the Run ID changes every run, so key on the Workflow ID and the step instead:

import hashlib
from temporalio import activity

@activity.defn
async def call_llm(step: int, prompt: str) -> str:
    info = activity.info()
    digest = hashlib.sha256(prompt.encode()).hexdigest()[:16]
    key = f"{info.workflow_id}:{step}:call_llm:{digest}"
    if (cached := await cache.get(key)) is not None:
        return cached                      # attempt 2 gets attempt 1's answer, unbilled
    answer = await provider.complete(prompt, idempotency_key=key)
    await cache.set(key, answer)
    return answer

A tool call gets the same treatment: ToolCall.key = f"{workflow_id}:{step}:{sha256(args)[:12]}", computed once in Workflow code and enforced inside execute_tool. Where the downstream system accepts no key, prefer conditional writes — upsert instead of insert, check-then-do — or accept duplicates where they are harmless (a monotonic counter, a log line). And keep Activities small: a three-step Activity that fails at step three reruns all three on retry.

The cache in this example is only an optimization. The provider must honor the same key across retries: if it charged for the first call and the Worker died before cache.set, the cache is empty on retry. A check followed by an external write has the same gap unless the downstream operation is atomic or can be reconciled. For a provider without idempotency or result lookup, an in-progress gateway record cannot prove whether the first paid call completed.

Retries are the default

"Temporal's default behavior is to automatically retry an Activity that fails, so transient or intermittent failures require no action on your part." Workflows, by contrast, are not retried by default. The default Retry Policy (as of 2026-09; see upstream page):

Initial Interval     = 1 second
Backoff Coefficient  = 2.0
Maximum Interval     = 100 × Initial Interval
Maximum Attempts     = ∞
Non-Retryable Errors = []

The retry interval is the smaller of initial_interval × backoff_coefficient ^ retries and maximum_interval. maximum_attempts=0 means unlimited, 1 means no retries; upstream prefers bounding total duration with Schedule-To-Close over attempt counts.

from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

result = await workflow.execute_activity(
    call_llm, args=[step, prompt],
    start_to_close_timeout=timedelta(seconds=120),
    retry_policy=RetryPolicy(
        initial_interval=timedelta(seconds=2),
        backoff_coefficient=2.0,
        maximum_attempts=3,
        non_retryable_error_types=["InvalidToolArguments"],
    ),
)

Some failures cannot succeed on retry — bad tool arguments, an unsupported model name. Non-retryable errors "are matched against the type field of the Application Failure", or the Activity marks itself:

raise ApplicationError("tool 'shell' rejects argument 'rm -rf'",
                       type="InvalidToolArguments", non_retryable=True)

One trap: "the Temporal Service only inspects the top-level failure_info" when deciding retryability. Re-raise a non-retryable ApplicationError wrapped in a plain Exception and the flag is gone — the Activity retries. Wrap in another ApplicationError if you must add context.

Vocabulary: a retry is the same operation again; compensation (module 7) is undoing a previous success.

The four timeouts

Each timeout bounds a different span. The definitions below are upstream's; the last column is ours.

"An Activity Execution must have either this timeout (Start-To-Close) or the Schedule-To-Close Timeout set" (as of 2026-09; see upstream page).

When Start-To-Close fires, an ActivityTaskTimedOut Event is written to mutable state; if the Retry Policy allows, the Service schedules another Activity Task, the attempt count increments, and the Start-To-Close timer resets.

Heartbeats and checkpoints

"An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Service. Each ping informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed." Without a heartbeat timeout, a ten-minute tool run whose Worker dies at minute one is not retried until minute ten. A heartbeat can carry details, and "if an Activity Task Execution times out due to a missed Heartbeat, the next Activity Task can access and continue with that payload." That is a checkpoint:

@activity.defn
async def run_tool_in_container(job: ToolJob) -> ToolResult:
    details = activity.info().heartbeat_details      # empty on attempt 1
    done = details[0] if details else 0
    for i in range(done, len(job.chunks)):           # killed at 60%? resumes at 60%
        await container.process(job.chunks[i])
        activity.heartbeat(i + 1)
    return await container.collect()

# in the Workflow
await workflow.execute_activity(
    run_tool_in_container, job,
    start_to_close_timeout=timedelta(minutes=10),
    heartbeat_timeout=timedelta(seconds=10),
)

The Worker throttles sends to the smaller of heartbeat_timeout × 0.8 (or 30 s if unset) and 60 s (as of 2026-09; see upstream page), so call heartbeat() freely — every loop iteration is fine. The last heartbeat before a failure is never throttled, but a Worker that crashes cannot deliver it, so a checkpoint may be one step stale: make the resumed step safe to repeat.

Temporal delivers the last recorded details to the next attempt; the Activity code above explicitly reads activity.info().heartbeat_details and starts its loop at done. If it ignored those details, the retry would start at zero. A heartbeat reports the Worker's progress, not proof that a separate API call or database transaction committed.

Heartbeats are also the only channel for Cancellation: "Activities that don't Heartbeat can't receive a Cancellation" (module 7). The Workflow cannot read heartbeat details while the Activity runs. Upstream's test for whether to heartbeat is not time but "How do you know you are making progress?" — a quick API call does not need it; a container run does.

How failure reaches your Workflow

Any exception raised in an Activity becomes an ApplicationError (type name, message and stack preserved, non_retryable=False). When the Activity Execution finally fails — retries exhausted, non-retryable, or cancelled — the Workflow gets an ActivityError whose cause is the original failure:

from temporalio.exceptions import ActivityError, ApplicationError, TimeoutError

try:
    answer = await workflow.execute_activity(call_llm, args=[step, prompt], ...)
except ActivityError as e:
    if isinstance(e.cause, TimeoutError):
        answer = await workflow.execute_activity(call_llm_fallback_model, ...)
    elif isinstance(e.cause, ApplicationError) and e.cause.type == "InvalidToolArguments":
        raise ApplicationError("agent produced an unusable plan", type="PlanRejected") from e
    else:
        raise

The decision after a failure is Workflow code; the failure itself is history. The two kinds of Workflow error are not symmetric: "In Workflow code, throwing a Temporal failure fails the Workflow Execution, while throwing any other error fails the Workflow Task and is retried automatically." A ValueError in the Workflow is a bug you fix and redeploy without losing state; an unhandled ActivityError fails the run.

In history, a completed Activity is three Events: ActivityTaskScheduled, ActivityTaskStarted, ActivityTaskCompleted. ActivityTaskStarted is not written until the Activity closes, so a retrying Activity shows only ActivityTaskScheduled — the attempt count is in the Web UI's Pending Activities.

History is not a blob store

Every Activity input and result is a payload in Event History, which every later Workflow Task replays. That makes it the right place for decisions and small results — the model's chosen step, a tool's exit code and summary line, the URI of an artifact — and the wrong place for LLM transcripts, giant tool outputs, checkpoint blobs or model artifacts. The Service enforces a per-payload limit, 2 MB by default and fixed on Temporal Cloud, and a history is capped at 51,200 Events or 50 MB with a warning at 10,240 Events or 10 MB (as of 2026-09; see upstream pages). A payload-heavy history slows every Workflow Task well below either limit; upstream names the agent case: "Long conversation histories grow with each turn, and the cumulative size can degrade Workflow performance."

The pattern is the claim check: keep large data in object storage and pass a reference through Temporal. execute_tool writes the container's full log to S3 and returns ToolResult(key, exit_code, summary, output_uri); call_llm receives a prompt built from context_summary, never the transcript. Temporal's External Storage feature applies the claim check inside the Data Converter: payloads at or above a threshold (256 KiB by default in Python) are uploaded by an S3StorageDriver and replaced by a reference token that the Worker resolves before your code runs. It is Public Preview as of 2026-09, you own the bucket's lifecycle policy, and the design rule stands: keep what the Workflow holds and passes between steps small enough to replay cheaply.

Asynchronous Activity completion

An expert extension, and the answer to a question the GPU lane forces. A fine-tuning job on gpu-tools takes 45 minutes. As a normal Activity it holds a Worker slot for 45 minutes, heartbeating; a preempted node means a fresh attempt from the last checkpoint. That is correct, but the Worker is doing nothing except waiting on a scheduler that already knows when the job ends.

Upstream's feature for this: "Asynchronous Activity Completion enables the Activity Function to return without causing the Activity Execution to complete. The Temporal Client can then be used from anywhere to both Heartbeat Activity Execution progress and eventually complete the Activity Execution and provide a result." Three steps, per the Python guide: the Activity hands the external system identifying information — a Task Token, or Namespace plus Workflow Id plus Activity Id; the Activity function returns in a way that marks it as waiting; a Temporal Client, anywhere, heartbeats and completes it.

@activity.defn
async def execute_tool(call: ToolCall) -> ToolResult:
    if call.tool == "finetune":                                   # the 45-minute GPU job
        token = activity.info().task_token
        await gpu_scheduler.submit(call.args, callback_token=token, key=call.key)
        activity.raise_complete_async()                           # the function returns; the Activity does not complete
    ...

# in the callback handler your GPU scheduler calls when the job ends — any process with a Client
handle = client.get_async_activity_handle(task_token=token)
await handle.heartbeat()                                          # optional progress, from outside
await handle.complete(ToolResult(key=key, output=artifact_uri))   # or handle.fail(...)

From the Workflow nothing changes: ActivityTaskScheduled stands, the await waits, timeouts and heartbeat rules apply exactly as before, and no Worker waits. Two cautions, both upstream. A Task Token is unique per Activity attempt, so if the Activity fails after handing out the token but before raise_complete_async, the retry issues a new one and the scheduler holds a dead token; passing Workflow Id plus Activity Id instead is the robust form. And a Start-To-Close timeout still bounds the wait, so set it to the job's real ceiling. The idempotency key carries through: a retried submission with the same key must return the same job.

Local Activities

A Local Activity "executes in the same Worker process as the Workflow Execution that schedules it": no Task Queue, no Service round trip, one MarkerRecorded Event instead of three. It is durable only once that marker is written, cannot heartbeat, and blocks Signals while it runs. Upstream reserves it for work that finishes in a few seconds, retries included. Default to a normal Activity.

The agent's tool call, specified

await workflow.execute_activity(
    execute_tool, ToolCall(key=key, tool=name, args=args),
    start_to_close_timeout=timedelta(minutes=2),    # slowest legitimate single attempt
    schedule_to_close_timeout=timedelta(minutes=15), # total budget; bounded without attempt maths
    heartbeat_timeout=timedelta(seconds=10),         # container runs report progress every few s
    retry_policy=RetryPolicy(
        maximum_attempts=3, backoff_coefficient=2.0,  # transient tool failures, not forever
        non_retryable_error_types=["InvalidToolArguments"],  # cannot succeed; fail fast
    ),
)
# key = f"{workflow_id}:{step}:{sha256(args)[:12]}" — computed in Workflow code, enforced inside execute_tool

Every number has a reason you can say out loud. That is the exercise.

What the lab will test

Suppose a charge succeeds externally, but its Activity completion never reaches the Service. Predict the retry outcome with and without an idempotency key; Lab 4 lets you inspect the ledger.

Try it

Lab 4 is in labs/04-activities-and-failure-semantics/:

• 4.1 — four Activities, each configured to hit exactly one timeout; record the history Event and the exception class for each.
• 4.2 — a heartbeating Activity that checkpoints a counter in heartbeat details; kill the Worker at 60% and watch the next attempt start at 60%.
• 4.3 — a non-idempotent "charge" executed twice under a forced Worker death, then fixed with a key.
• 4.4 — design the agent's execute_tool Activity: timeouts, retry policy, idempotency key, heartbeat — with a one-line justification each.
• 4.5 — asynchronous completion: the finetune tool hands out its Task Token and returns without completing; finish the Activity from a separate process with complete_async.py, and note what did not change — the Workflow code, the ActivityTaskScheduled Event, and the Start-To-Close that still bounds the wait.

Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.