Temporal developer on retainer: uuid.uuid4() called directly inside @workflow.run method generates different correlation ID on Kubernetes rolling restart replay — WorkflowTaskFailedError: nondeterminism detected on 27 in-flight payment workflows — 14 payments in uncertain state — Temporal developer on monthly retainer
Temporal Python SDK 1.5.1 • Python 3.11 • GKE 1.27 • PaymentProcessingWorkflow • correlation_id = str(uuid.uuid4()) in @workflow.run • Kubernetes rolling node upgrade triggers history replay • UUID differs from recorded history • WorkflowTaskFailedError: nondeterminism • 27 workflows stuck in Failed • 14 payments uncertain • 4.5h retainer engagement
On September 15, 2026, at 2:47 AM UTC, the SRE team at Vantage Fintech Operations Inc. received an alert: 27 PaymentProcessingWorkflow instances had transitioned to the Failed state within a two-minute window. The Temporal UI showed WorkflowTaskFailedError: nondeterminism detected on every affected workflow. The timing correlated exactly with a Kubernetes rolling node upgrade that had started at 2:45 AM and restarted the Temporal worker pods. The Temporal retainer developer was paged at 3:00 AM and identified the root cause in thirty minutes: str(uuid.uuid4()) was called directly inside the @workflow.run method to generate a correlation ID for the payment activities. The UUID returned by uuid.uuid4() on the restarted worker pod was different from the UUID recorded in each workflow’s history during the original execution. Temporal detected the mismatch and failed all 27 workflows.
Vantage Fintech Operations processed retail payment authorizations using a PaymentProcessingWorkflow that orchestrated four sequential activities: validate_payment (validates card details and merchant limits), check_fraud (calls a third-party fraud scoring API), settle_payment (sends the settlement request to the payment network), and notify_customer (sends SMS and email confirmation). Each activity received a correlation_id to use for idempotency and tracing. The developer had placed correlation_id = str(uuid.uuid4()) at the top of the @workflow.run method, reasoning that generating the UUID once at the start of the workflow and passing it to all activities would produce consistent tracing. This worked correctly on every execution from June 2026 through September 14, 2026 — because the UUID was only needed once during the original execution, and no worker restarts had occurred while workflows were in-flight.
# payment_workflow.py — Vantage Fintech Operations Inc.
# Temporal Python SDK 1.5.1, Python 3.11, GKE 1.27
import uuid
from temporalio import workflow, activity
from temporalio.common import RetryPolicy
from datetime import timedelta
@workflow.defn
class PaymentProcessingWorkflow:
@workflow.run
async def run(self, payment_request: PaymentRequest) -> PaymentResult:
# BUG: uuid.uuid4() is non-deterministic.
# On original execution (June–September 14 2026):
# correlation_id = "3f2a1b4c-8d9e-4f0a-b1c2-d3e4f5a6b7c8"
# This UUID is recorded in the workflow history as the argument
# to the first ScheduleActivityTask command (validate_payment).
# On replay (September 15 2026, after Kubernetes rolling restart):
# uuid.uuid4() returns "9a8b7c6d-5e4f-3a2b-1c0d-e9f8a7b6c5d4"
# Temporal compares this against the recorded "3f2a1b4c-..." and detects mismatch.
# WorkflowTaskFailedError: nondeterminism detected.
correlation_id = str(uuid.uuid4()) # BUG: non-deterministic in workflow context
# Four sequential payment activities
validation_result = await workflow.execute_activity(
validate_payment,
args=[payment_request, correlation_id], # correlation_id used here
start_to_close_timeout=timedelta(seconds=30),
retry_policy=RetryPolicy(maximum_attempts=3)
)
fraud_result = await workflow.execute_activity(
check_fraud,
args=[payment_request, validation_result, correlation_id],
start_to_close_timeout=timedelta(seconds=10),
retry_policy=RetryPolicy(maximum_attempts=2)
)
if not fraud_result.approved:
return PaymentResult(status="DECLINED", correlation_id=correlation_id)
settlement_result = await workflow.execute_activity(
settle_payment,
args=[payment_request, fraud_result, correlation_id],
start_to_close_timeout=timedelta(minutes=2),
retry_policy=RetryPolicy(maximum_attempts=5)
)
await workflow.execute_activity(
notify_customer,
args=[payment_request, settlement_result, correlation_id],
start_to_close_timeout=timedelta(seconds=15)
)
return PaymentResult(status="COMPLETED", correlation_id=correlation_id)Temporal’s execution model is built on deterministic replay: when a Temporal worker restarts mid-workflow, it reconstructs the workflow’s state by replaying the recorded command history. During replay, the @workflow.run method executes again from the beginning. Temporal doesn’t re-execute activities whose results are already in the history — it returns the recorded results immediately when the workflow code schedules the corresponding activity. But it does re-execute every line of the workflow function that is not an await workflow.execute_activity() or await workflow.sleep() call. The correlation_id = str(uuid.uuid4()) line ran again during replay and produced a different UUID. When the replay code reached await workflow.execute_activity(validate_payment, args=[payment_request, correlation_id]), the correlation_id argument was “9a8b7c6d-...”. The recorded history showed the same ScheduleActivityTask command with correlation_id=“3f2a1b4c-...”. Temporal detected the argument mismatch, determined that the replay had diverged from the original execution, and raised WorkflowTaskFailedError: nondeterminism detected.
# Investigation — Temporal retainer developer, September 15 2026 03:00 UTC
# Vantage Fintech Operations Inc., Temporal Python SDK 1.5.1, GKE 1.27
# Step 1: Query Temporal for all failed workflows
tctl workflow list \
--namespace vantage-payments-prod \
--status FAILED \
--query "WorkflowType='PaymentProcessingWorkflow' AND CloseTime > '2026-09-15T02:40:00Z'"
# 27 failed workflows, all closed 2026-09-15T02:45:XX to 2026-09-15T02:47:XX
# All have FailureMessage: "WorkflowTaskFailedError: nondeterminism detected"
# Step 2: Inspect one failed workflow's history
tctl workflow show --workflow_id "pay-wf-abc123" --namespace vantage-payments-prod
# History event 1: WorkflowExecutionStarted (original UUID generated here, not visible)
# History event 2: WorkflowTaskScheduled
# History event 3: WorkflowTaskStarted
# History event 4: WorkflowTaskCompleted
# → ScheduleActivityTask: validate_payment, args=["...", "3f2a1b4c-8d9e-4f0a-b1c2-d3e4f5a6b7c8"]
# History event 5: ActivityTaskScheduled: validate_payment
# History event 6: ActivityTaskStarted: validate_payment
# History event 7: ActivityTaskCompleted: validate_payment
# History event 8: WorkflowTaskScheduled (replay starts here on new worker pod)
# History event 9: WorkflowTaskStarted
# History event 10: WorkflowTaskFailed
# Cause: NONDETERMINISM_ERROR
# Message: "Workflow code attempted to schedule activity validate_payment with args
# ['...', '9a8b7c6d-5e4f-3a2b-1c0d-e9f8a7b6c5d4']
# but history recorded activity scheduled with args
# ['...', '3f2a1b4c-8d9e-4f0a-b1c2-d3e4f5a6b7c8']"
# → correlation_id mismatch confirmed: uuid.uuid4() returned different value on replay
# Step 3: Identify uncertain payment states (14 of 27)
# 13 workflows: validate_payment + check_fraud NOT in history → no payment actions taken
# 14 workflows: validate_payment + check_fraud COMPLETED, settle_payment NOT in history
# → fraud check approved but settlement not yet attempted
# → need manual reconciliation: confirm no duplicate settlements before re-running
# Step 4: Fix
# Replace: correlation_id = str(uuid.uuid4())
# With: correlation_id = str(workflow.uuid4())
# workflow.uuid4() is deterministic: returns UUID derived from workflow run ID and a counter
# Same workflow run ID → same UUID on every replayThe fix was a one-line change: replacing str(uuid.uuid4()) with str(workflow.uuid4()). The Temporal Python SDK’s workflow.uuid4() function generates a UUID deterministically based on the workflow run ID and an internal counter. Every replay of the same workflow history with the same run ID produces the same sequence of UUIDs, so the ScheduleActivityTask command arguments match the recorded history on every replay. The fix was deployed to the production Temporal workers after the 14 uncertain payments were manually reconciled (verifying that no duplicate settlement requests had been submitted during the failed replay attempts).
The manual reconciliation took two hours: the retainer developer and an on-call SRE queried the payment network’s idempotency log for each of the 14 correlation_id values from the recorded workflow histories, confirmed that no settlement had been submitted for any of them, then terminated and re-ran each of the 14 workflows. The 13 workflows where no payment activities had completed were simpler: they were terminated and re-run without reconciliation. All 27 workflows completed successfully under the new code.
Why non-deterministic code in workflow functions is a recurring Temporal retainer pattern
Temporal’s determinism requirement is fundamental to its durability guarantee: a workflow can survive arbitrary worker failures, restarts, and code deployments because its state can always be reconstructed by replaying its history. This guarantee breaks if the workflow function produces different outputs on different executions — which happens whenever any non-deterministic source (random number generator, clock, UUID generator) is called inside the workflow function without going through the Temporal SDK’s determinism-preserving equivalents.
The failure is invisible until a replay is triggered. In production, replays happen during worker restarts (planned or unplanned), Temporal cluster upgrades, and history compaction events. A workflow with a uuid.uuid4() call can run in production for months without issue — as long as no worker restarts occur while the workflow is in-flight. The nondeterminism is present from the first deployment but only manifests when a replay occurs. In the Vantage Fintech case, the uuid.uuid4() call had been in the code for three months. A Kubernetes rolling node upgrade — a routine maintenance event — triggered replays for all 27 in-flight payment workflows simultaneously, producing a mass failure event at 2:45 AM.
A Temporal retainer developer who runs a quarterly determinism audit catches this failure class before it affects in-flight workflows. The standard practice: use Temporal’s workflow replay testing API to replay a sample of production workflow histories against the current codebase. Any WorkflowTaskFailedError: nondeterminism in the replay test suite indicates non-deterministic code in a workflow function and must be fixed before the next worker restart event.
The 4.5-hour engagement covered diagnosis (0.5h — WorkflowTaskFailedError: nondeterminism detected on all 27 workflows; correlation_id UUID mismatch in event 10 history vs. replay; uuid.uuid4() call identified in workflow function), payment state assessment and reconciliation (2.5h — 0.5h categorize 27 workflows: 13 with no activity completions, 14 with validate_payment + check_fraud completed; 2.0h manual reconciliation: payment network idempotency log queries for 14 correlation_ids; no duplicate settlements confirmed; 14 workflows terminated and re-run; 13 workflows terminated and re-run), fix implementation (0.5h — uuid.uuid4() → workflow.uuid4(); deployed to production workers; replay test against 27 fixed workflow runs: zero nondeterminism errors), and prevention (1.0h — CI integration test added: replay last 7 days of workflow histories using Temporal workflow replay API; any NONDETERMINISM_ERROR fails the CI build; worker deployment runbook updated: quarterly determinism audit scheduled).
In a HourTab retainer log, the incident session reads: “Temporal WorkflowTaskFailedError nondeterminism investigation: Temporal Python SDK 1.5.1, Python 3.11, GKE 1.27; PaymentProcessingWorkflow; correlation_id = str(uuid.uuid4()) in @workflow.run method; deployed June 2026; September 15 2026 02:45 UTC Kubernetes rolling node upgrade restarts Temporal worker pods; 27 in-flight workflows enter replay; uuid.uuid4() returns different UUID on replay; ScheduleActivityTask validate_payment args mismatch vs. recorded history; WorkflowTaskFailedError: nondeterminism detected; 27 workflows → Failed; 14 payments with validate_payment + check_fraud completed, settle_payment not started (uncertain state); manual reconciliation: no duplicate settlements; 27 workflows re-run successfully; fix: workflow.uuid4(); prevention: CI replay test; 4.5h.”
HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.
FAQ: Temporal developer retainers
What does a Temporal developer on retainer typically do?
A Temporal developer on monthly retainer covers workflow and activity architecture (designing @workflow.defn classes for long-running business processes; implementing @activity.defn functions for external calls; configuring activity timeouts and retry policies; managing signals and queries), determinism compliance (auditing workflow functions for non-deterministic code: time.time(), datetime.now(), random.*, uuid.uuid4(); replacing with Temporal SDK equivalents: workflow.now(), workflow.random(), workflow.uuid4(); using workflow.patched() for breaking code changes to in-flight workflows), workflow lifecycle operations (namespace configuration; worker rollout procedures; Temporal visibility queries for operational monitoring), and recurring retainer activities (quarterly determinism audit using the workflow replay testing API).
What Temporal work is most commonly underlogged on retainer?
The most systematically underlogged Temporal retainer risk is non-deterministic code inside @workflow.run methods — any call to uuid.uuid4(), datetime.now(), random.*, or other functions that return different values on repeated calls. The code looks correct and runs without issues until a worker restart triggers a history replay, at which point the different value produces a different command sequence and Temporal raises WorkflowTaskFailedError: nondeterminism detected. In the Vantage Fintech case, uuid.uuid4() in a payment workflow function ran undetected for three months before a Kubernetes rolling node upgrade triggered 27 simultaneous replays, failing all 27 in-flight payment workflows.
What are typical Temporal developer retainer rates?
Entry-level Temporal developers with workflow and activity operations and basic incident response typically bill at $80 to $130 per hour. Mid-level Temporal engineers with workflow architecture, determinism compliance, incident response, and worker rollout management typically bill at $120 to $190 per hour. Senior Temporal architects with enterprise workflow governance, reliability engineering (determinism monitoring, replay testing harness), and cross-platform architecture typically bill at $160 to $270 per hour. Monthly retainer ranges: $9,000 to $17,000 per month for Temporal workflow operations and business process management; $16,000 to $28,000 per month for enterprise workflow orchestration platform architecture.
What should a Temporal developer retainer agreement include?
A Temporal developer retainer agreement should specify: Temporal SDK version, namespace, all production workflow and activity type names, and the determinism policy (prohibited non-deterministic operations inside workflow functions and their Temporal SDK replacements). The most critical clause is the determinism requirement: @workflow.run methods must never call uuid.uuid4(), datetime.now(), random.*, or any I/O operation directly. The agreement must define a worker rollout procedure using workflow.patched() for breaking changes to in-flight workflows. The agreement should mandate a quarterly determinism audit replaying production workflow histories against current code via the Temporal replay testing API, and a 1-hour incident response SLA for WorkflowTaskFailedError on financially consequential workflow types.
How should Temporal developer retainer hours be logged?
Log each Temporal retainer session with: Temporal SDK version, namespace, workflow type name, activity types, issue type (WorkflowTaskFailedError nondeterminism, workflow stuck in Running, activity timeout, signal channel overflow, history size limit, worker connectivity failure), root cause, payment/business state assessment, and hours by component. For the Vantage Fintech incident: Temporal Python SDK 1.5.1, GKE 1.27; PaymentProcessingWorkflow; uuid.uuid4() in @workflow.run; 27 workflows Failed; 14 uncertain payments; 4.5h total (0.5h diagnosis + 2.5h payment reconciliation + 0.5h fix + 1.0h prevention).
See also: Apache Airflow developer retainer · Prefect developer retainer · ZenML developer retainer