AgentRun's compute is heterogeneous. A planning step needs nothing but a CPU; an embedding step wants a GPU; a browsing tool needs a live Chromium process; a frontier-model call needs a rate-limited, expensive API key. The Workflow that decides what to do next must not care which machine does it. Task Queues are how Temporal separates the two.
Semantics versus today's numbers
This page mixes two kinds of fact, and only one of them is worth memorising. The semantics — Workers pull, a Task Queue routes, the registration on a queue must be identical across its Workers — are what an interviewer is testing and what your architecture rests on. The numbers are current implementation detail: every one of them is tagged (as of 2026-09; see upstream page) where it appears. Four partitions, a five-minute unload, a 90% poll-success floor — read those as "what it does today", check them against upstream before you tune on them, and never recite one as though it were a guarantee.
The upstream definition is precise, so keep it: a Task Queue is a lightweight, dynamically allocated queue that one or more Worker Entities poll for Tasks. There are three types of Task Queues: Activity Task Queues, Workflow Task Queues, and Nexus Task Queues.
Task Queues don't require explicit registration. They're created on demand when a Workflow Execution, Activity, or Nexus Operation is invoked, and/or when a Worker Process subscribes to start polling. When a named Task Queue is created, individual Task Queues for Workflows, Activities, and Nexus are created using the same name. A Temporal Application can use, and the Temporal Service can maintain, an unlimited number of Task Queues.
Read the second paragraph again, because it is the source of the most common first-week bug and of the most useful design freedom you have. There is no CREATE QUEUE step. The first time your Client starts AgentRun on agent-workflows, that queue exists. The first time a Worker polls gpu-tools, that queue exists. Nothing checks that anyone will ever poll the queue you started a Workflow on.
If you arrive from Kafka or SQS, the word "queue" will mislead you. A Task Queue is not a durable log of messages that consumers read through at their own pace, and a Task is not a message you own. A Task is a short-lived instruction from the Temporal Service to some Worker: "make progress on this Workflow Execution" or "run this Activity attempt". The record of what happened is the Event History, never the queue.
Workers poll for Tasks in Task Queues via synchronous RPC. Upstream lists what this buys you; the parts that matter for a runtime like AgentRun are:
So the mental model is: the Service holds work it wants done, and Workers pull it when they have room. Nobody pushes to a Worker. That single fact explains why a Worker fleet can be scaled to zero and back with no data loss, why there are no consumer groups or offsets to manage, and why a queue with no pollers is not an error condition, only a stalled one.
Ordering is deliberately weak. Task Queues can be scaled by adding partitions; by default each Task Queue has 4 partitions (as of 2026-09; see upstream page), each Task is assigned to a random partition, and partitions act roughly FIFO. Upstream is explicit that this is about the ordering of individual Tasks and does not apply to the ordering of Events in a single Workflow Execution. The order of Events in a Workflow Execution is guaranteed to remain constant once they have been written to that Workflow Execution's History. AgentRun's steps are ordered by its History, not by any queue.
Any Worker can pick up any Task on a given Task Queue. You must ensure that if a Worker accepts a Task that it can process that task using one of its registered Workflows, Activities, or Nexus Operation handlers. This means that all Workers listening to a Task Queue must register all Workflows, Activities, and Nexus Operations that live on that Queue.
When Workers don't have a registered handler for a given Task, the Task will fail with a "Not Found" error. "Not Found" Workflow Tasks and Activity Tasks are treated as retryable errors, so the Workflow Execution does not fail; it stalls until a Worker that knows the type polls. The two exceptions upstream allows are Worker Versioning during a rollout and dynamic Workflow or Activity stand-ins.
For AgentRun this rule is the topology rule. If gpu-tools Workers must register every Activity on gpu-tools, then the set of Activities you put on gpu-tools is exactly the set that needs a GPU. The queue name is a contract about capability.
Upstream lists five places the Task Queue name can be set. In the Python SDK (names verified against temporalio 1.32.0, as of 2026-09; see upstream page):
client.start_workflow(..., task_queue=...). Required.Worker(client, task_queue=..., workflows=[...], activities=[...]). Required.workflow.execute_activity(..., task_queue=...). Optional; an Activity Execution inherits the Task Queue name from its Workflow Execution if one is not provided.workflow.start_child_workflow(..., task_queue=...). Optional; a Child Workflow Execution inherits the Task Queue name from its Parent Workflow Execution if one is not provided.The inheritance defaults in 3 and 4 are why a single-queue tutorial works with no routing at all, and why AgentRun has to opt in to routing on every tool call.
Since Task Queues are created dynamically when they are first used, a mismatch between the name the Client uses and the name the Worker uses does not result in an error. Instead, it will result in the creation of two different Task Queues. Consequently, the Worker will not receive any tasks from the Temporal Service and the Workflow Execution will not progress.
Upstream's recommendation is to define the Task Queue name in a constant that is referenced by the Client and Worker. Do that, and do it for every lane:
# shared.py — the only place a Task Queue name is spelled out
AGENT_WORKFLOWS = "agent-workflows"
PREMIUM_MODELS = "premium-models"
CPU_TOOLS = "cpu-tools"
GPU_TOOLS = "gpu-tools"
BROWSER_TOOLS = "browser-tools"
from shared import AGENT_WORKFLOWS
client = await Client.connect("localhost:7233", namespace="default")
handle = await client.start_workflow(
AgentRun.run,
task,
id=f"agent-{task.id}",
task_queue=AGENT_WORKFLOWS,
)
from shared import GPU_TOOLS
worker = Worker(
client,
task_queue=GPU_TOOLS,
activities=[embed_documents, run_vision_model],
)
Upstream notes the constant is not always possible: the Client that starts the Workflow may run on another system or be written in another language. In that case the name is part of your published interface, and you version it like one.
The rest of naming is convention, not upstream rule. The conventions this course uses are ours:
gpu-tools, not ml-team-queue or node-17. The identical-registration rule means a queue name is a promise about what Workers on it can do.prod.gpu-tools / staging.gpu-tools. Better still, one Namespace per environment and no prefix.gpu-tools.v2). Upstream calls Task Routing the simplest way to version code; if you are using Worker Versioning instead (module 9), keep the name stable.temporal task-queue describe, and in the TaskQueue Search Attribute (A3), so make it greppable.Upstream's own routing examples are the ones you need: "Some Workers might exist on GPU boxes versus non-GPU boxes. In this case, each type of box would have its own Task Queue and a Workflow can pick one to send Activity Tasks." Applied to AgentRun:
Temporal Service (one Namespace)
┌──────────────────────────────────────────────────────────────────┐
│ Task Queue Worker fleet that polls it │
│ │
│ agent-workflows ◄── workflow Workers registers AgentRun, │
│ (small, many) ResearchAgent only │
│ │
│ premium-models ◄── LLM Workers registers call_llm; │
│ (holds the API key, rate-limited per │
│ few slots) queue │
│ │
│ cpu-tools ◄── general Workers registers search, │
│ (cheap, autoscaled) parse, fetch, verify │
│ │
│ gpu-tools ◄── GPU Workers registers embed, │
│ (scarce, preemptible) vision, rerank │
│ │
│ browser-tools ◄── browser Workers registers open_page, │
│ (one Chromium each, click, extract; │
│ + per-host queue) routed by host │
└──────────────────────────────────────────────────────────────────┘
AgentRun (on agent-workflows) decides, then dispatches:
plan() ── Workflow code, no queue
call_llm(...) ── task_queue = premium-models
execute_tool("search", ...) ── task_queue = cpu-tools
execute_tool("embed", ...) ── task_queue = gpu-tools
open_page(url) ── task_queue = browser-tools
click(sel) ── task_queue = <host queue returned by open_page>
ResearchAgent child ── task_queue = agent-workflows (inherited)
The Workflow Worker fleet on agent-workflows is deliberately boring: it registers Workflow types and no Activities that need anything but a CPU, so it is the fleet you can run many of and deploy twice a day. Each tool lane registers only its own Activities, and each is sized and provisioned differently. The Workflow code chooses a lane per call:
@workflow.defn
class AgentRun:
@workflow.run
async def run(self, task: Task) -> Result:
while not self.done:
step = self.plan() # a decision; no Task Queue involved
if step.kind == "llm":
out = await workflow.execute_activity(
call_llm, step.request,
task_queue=PREMIUM_MODELS,
start_to_close_timeout=timedelta(minutes=5),
retry_policy=LLM_RETRY,
)
elif step.kind == "tool":
out = await workflow.execute_activity(
execute_tool, step.request,
task_queue=LANE_FOR_TOOL[step.tool], # cpu-tools | gpu-tools | browser-tools
start_to_close_timeout=timedelta(minutes=30),
heartbeat_timeout=timedelta(seconds=30),
retry_policy=TOOL_RETRY,
)
...
LANE_FOR_TOOL is a plain dict in Workflow code. Because it is read during a Workflow Task, it is replayed exactly like any other decision, so the lane a step went to is part of the History, and a redeploy that moves a tool between lanes needs a patch (module 9) if a run can be mid-step.
The two systems are answering different questions, and the design breaks when you make either one answer both.
Worker Processes are external to a Temporal Service. Temporal Application developers are responsible for developing Worker Programs and operating Worker Processes. Said another way, the Temporal Service (including Temporal Cloud) doesn't execute any of your code (Workflow and Activity Definitions) on Temporal Service machines. The Temporal Service is solely responsible for orchestrating State Transitions and providing Tasks to the next available Worker Entity.
So Kubernetes owns the processes: one Deployment per lane, with the GPU lane on a node pool that has GPUs, the browser lane with a Chromium sidecar, the premium-models lane with the secret mounted, each with its own replica count and rollout policy. Kubernetes never sees a Workflow; it sees five Deployments that all make outbound gRPC connections to the same address.
Temporal owns the execution: which Workflow Execution is at which step, which Activity attempt is outstanding, and which Task Queue that attempt is waiting on. When a GPU node is preempted, Kubernetes reschedules the pod somewhere else, and that is the whole of what Kubernetes does. Temporal, separately, times out the Activity attempt's heartbeat, records it, and re-queues the Task on gpu-tools, where the new pod picks it up. Neither system had to tell the other anything. This is also the answer to the tenth question in the capstone: Kubernetes ends at the process boundary, and Temporal begins at the Task Queue.
A Worker's identity is where the two worlds meet in logs. By default Temporal SDKs set a Worker Identity to ${process.pid}@${os.hostname()}, which upstream points out is nearly useless in containers: the process ID is always 1, and the hostname is a randomly generated string. Set it explicitly (Worker(..., identity=...) in Python, as of 2026-09; see upstream page) to something you can grep in both systems, such as the pod name plus the lane. The identity appears in Event History and in the poller list, so a broken run's history tells you which pod ran the failing attempt.
Because Workers pull, "is this lane under-provisioned?" has a direct answer: is work waiting? The Temporal Service reports, separately for each Task Queue type (as of 2026-09; see upstream page):
ApproximateBacklogCount — the approximate count of Tasks currently backlogged in this Task Queue.ApproximateBacklogAge — the approximate age of the oldest Task in the backlog, based on the creation time of the Task at the head of the queue.TasksAddRate and TasksDispatchRate — approximate Tasks-per-second added to or dispatched from the Task Queue, averaged over the most recent 30-second interval.BacklogIncreaseRate — the net Tasks per second added to the backlog, calculated as TasksAddRate - TasksDispatchRate. Positive means growing, negative shrinking.Upstream says you can rely on the count and age when making scaling decisions, and that while the individual add and dispatch rates may be inaccurate (eager dispatch and sticky queues bypass them), BacklogIncreaseRate reliably reflects the rate at which the backlog is shrinking or growing for backlogs older than a few seconds.
Read them from the CLI, per lane and per Task type:
temporal task-queue describe --task-queue gpu-tools --task-queue-type activity
The same command lists the pollers and their LastAccessTime. Upstream's heuristic: a value over one minute may mean the fleet is at capacity or has shut down; under five minutes usually means at capacity (every slot full); over five minutes usually means Workers are gone, since Workers are removed if 5 minutes have passed since the last poll request (as of 2026-09; see upstream page).
For AgentRun that gives one autoscaling rule per lane, and it is the same rule for all five: scale on ApproximateBacklogAge (or BacklogIncreaseRate) of that lane's Activity Task Queue, not on CPU. A gpu-tools pod at 20% CPU with a five-minute backlog age is the bottleneck; a cpu-tools pod at 90% CPU with an empty backlog is fine. Upstream's per-Worker demand calculation is ApproximateBacklogCount divided by the number of Workers. On the SDK side the matching metric is activity_schedule_to_start_latency (prefixed temporal_ when emitted, as of 2026-09; see upstream page): the time from when a Task is scheduled to when a Worker starts it. Rising schedule-to-start on one lane, with the others flat, is the signature of a starved fleet.
Two accuracy caveats matter operationally. If a Task Queue sees no activity for approximately 5 minutes it is unloaded from memory and ApproximateBacklogCount reports zero until the next poll, Task, or API call, so an idle lane with no Workers can look empty when it is not. And it is possible to have too many Workers: upstream's Poll Success Rate, (poll_success + poll_success_sync) / (poll_success + poll_success_sync + poll_timeouts), should be above 90% under steady load (as of 2026-09; see upstream page); low poll success plus low schedule-to-start plus idle hosts means size down.
A Worker that consumes from a Task Queue asks for an Activity Task only when it has available capacity, so it is never overloaded by request spikes. If Activity Tasks get created faster than Workers can process them, they are backlogged in the Task Queue. Capacity is expressed as slots: a Worker Task Slot represents the capacity of a Temporal Worker to execute a single concurrent Task, and a slot supplier decides how many exist (fixed, resource-based, or custom; in Python via max_concurrent_activities or a tuner, as of 2026-09; see upstream page).
Throttling is separate from capacity. The rate at which each Activity Worker polls for and processes Activity Tasks is configurable per Worker, and there is also support for global Task Queue rate limiting, which works across all Workers for the given Task Queue and is frequently used to limit load on a downstream service that an Activity calls into. That last sentence is the premium-models lane in one line: put every frontier-model call on its own queue and cap the queue's dispatch rate (Worker(..., max_task_queue_activities_per_second=...) in Python, as of 2026-09; see upstream page) at what the provider's rate limit allows. The Workflow never learns about the limit; its call simply waits in the backlog.
Most tools are stateless and any Worker on the lane will do. Some are not. Upstream's example is file processing: download a file on any host, then process and upload it on the same host, because the file is local. AgentRun's version is the browser lane. open_page starts a Chromium session in one Worker Process; click and extract on that page must run in the same process, or they find no browser.
Upstream's answer: to route Activity Tasks to a specific host or process, use a dedicated Task Queue — a unique Task Queue would exist for each Worker Process involved. Concretely, each browser Worker Process polls two queues: the shared browser-tools, and a private one named for itself. The first Activity returns the private queue name; later Activities are dispatched to it.
# browser worker process: one Chromium, two queues
HOST_QUEUE = f"{BROWSER_TOOLS}.{os.environ['POD_NAME']}"
shared = Worker(client, task_queue=BROWSER_TOOLS, activities=[open_page])
private = Worker(client, task_queue=HOST_QUEUE, activities=[click, extract, close_page])
await asyncio.gather(shared.run(), private.run())
# in AgentRun
session = await workflow.execute_activity(
open_page, url, task_queue=BROWSER_TOOLS, start_to_close_timeout=timedelta(minutes=1)
)
text = await workflow.execute_activity(
extract, session.page_id,
task_queue=session.host_queue, # returned by open_page
start_to_close_timeout=timedelta(minutes=1),
schedule_to_start_timeout=timedelta(seconds=30), # the host may be gone
)
The schedule_to_start_timeout is the important line. A private queue has exactly one poller, so if that pod dies the Task sits in a queue no one will ever drain. A short Schedule-to-Start timeout turns "the host vanished" into an Activity failure the Workflow can handle by opening the page again on whatever host is up. Without it, the run hangs.
Some SDKs provide a Session API that ensures Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names, with concurrent session limitations and Worker failure detection built in. As of 2026-09 the upstream page links only the Go guide for it (Worker Sessions), and the Python SDK does not expose one; the per-host queue above is the portable pattern.
If your use case involves more than one priority, upstream offers two routes: one Task Queue per priority with a Worker pool per priority, or Task Queue Priority, which assigns priority levels to Tasks within a single Task Queue and avoids the overhead of managing multiple queues and pools. For AgentRun, an interactive run that a user is watching and a batch run that nobody is watching are the same Workflow type; priority within one lane is the right tool, not a sixth lane.
Task Routing is also the simplest way to version your code: a backward-incompatible Activity Definition can start on a different Task Queue. Worker Versioning (module 9) tags Workers with a version and routes Workflow and Activity Tasks to specific versions without separate Task Queues, and supports Pinned and Auto-Upgrade Workflows. It went GA in March 2026 and its API shape is expected to move again; see Worker Versioning for the current form rather than this page.
This section is ours, in the format the capstone defense requires.
| Lane | Every Worker on it dies now | What the Workflow sees | Recorded |
|---|---|---|---|
agent-workflows |
Workflow Tasks stay in the queue; no state changes; nothing polls | nothing — it is not running | nothing new until a Worker returns and replays |
premium-models |
in-flight call_llm attempts time out (Start-to-Close); Task re-queued |
the await continues to wait; retry policy governs attempts |
ActivityTaskScheduled stands; a TimedOut attempt appears in describe under pending activities |
gpu-tools |
heartbeat timeout fires on the running attempt; Task re-queued; backlog age climbs | same; the next attempt receives the last delivered heartbeat details, which Activity code must read to resume | heartbeat details on the pending activity |
browser-tools (private queue) |
the only poller is gone; the Task can never be dispatched | Schedule-to-Start timeout → Activity failure → Workflow reopens the page | ActivityTaskTimedOut with SCHEDULE_TO_START |
| any | — | — | temporal task-queue describe shows no pollers and a growing backlog |
Lab 10.1 (labs/10-operating-temporal-in-production/) has you assemble AgentRun with separate Task Queues for the LLM, the CPU tools, and a mock GPU tool, and run one Worker fleet per queue from separate terminals. Kill the GPU fleet mid-run and watch temporal task-queue describe --task-queue gpu-tools --task-queue-type activity report the backlog and the Workflow's pending activity wait; bring it back and watch the backlog drain. Lab 11.4's tenth question, "where does Temporal end and Kubernetes begin?", is answered by the section above on processes versus execution, in one paragraph, from memory.
This page adapts material from Temporal's MIT-licensed documentation and samples (© Temporal Technologies Inc.; © Uber Technologies, Inc.). Adapted text is rewritten for this course; the upstream pages are the reference of record and may have changed since the commit linked here. Temporal is a trademark of Temporal Technologies; this course is independent and not endorsed by Temporal.