Celery developer on retainer: process_subscription_payment acks_late=False default — Kubernetes rolling deploy SIGTERM kills in-flight tasks — 89 orphaned Stripe PaymentIntents — $4,183 in charges never credited to customer accounts

Celery 5.3.6 • Python 3.11 • Redis 7.2.4 on Amazon ElastiCache • Kubernetes 1.29 on Amazon EKS • Stripe PaymentIntents • acks_late • reject_on_worker_lost • rolling deploy • SIGTERM • idempotency key • retainer billing

Harborview Commerce Technology LLC’s reconciliation team ran its monthly Stripe audit on October 6, 2026, and found 89 PaymentIntents in the Stripe Dashboard with status=succeeded that had no corresponding subscription record in the application database. The charges totaled $4,183. The affected payments ranged from September 1 through October 1 — exactly the five weeks since the new billing feature had been deployed. No error had been raised. No alert had fired. Celery reported no failed tasks. The Stripe charges were real and the customers had been debited, but the application had no record of the payments: no payment_intent_id saved, no subscription activated, no receipt sent. The Celery developer retainer was engaged to investigate. The root cause was Celery’s default acks_late=False behavior interacting with Kubernetes rolling deployments: the broker acknowledged (removed) each task message as soon as a worker received it — before the task function ran — and rolling deploys sent SIGTERM to worker pods approximately every two days, killing tasks that were mid-execution during the 30-second grace period. For the process_subscription_payment task, the most dangerous kill window was after stripe.PaymentIntent.create returned a successful charge but before the payment_intent_id was saved to the application database.

Harborview Commerce Technology LLC ran Celery 5.3.6 on Python 3.11, using Redis 7.2.4 on Amazon ElastiCache as the broker. Celery workers were deployed as a Kubernetes Deployment on Amazon EKS (Kubernetes 1.29), with 4 replicas, a rolling update strategy (maxUnavailable=1, maxSurge=1), and terminationGracePeriodSeconds: 30. The billing feature, deployed September 1, added a Celery task process_subscription_payment in tasks/billing.py. The task called stripe.PaymentIntent.create and then saved the resulting payment_intent_id to the application database. The task was defined with default Celery settings: no explicit acks_late, no reject_on_worker_lost, no retry logic, and no Stripe idempotency key:

# tasks/billing.py
# Celery 5.3.6, Python 3.11
# Broker: Redis 7.2.4 on Amazon ElastiCache
# Workers: Kubernetes Deployment, 4 replicas, rolling update

from celery import shared_task
import stripe
from myapp.models import Subscription

@shared_task
# PROBLEM: No acks_late=True
# Celery default: acks_late=False
# Broker acknowledges (removes) the message the moment the worker RECEIVES the task
# — before process_subscription_payment() begins executing.
# If the worker is killed after receiving the task but before the function completes,
# the message is gone from Redis. The task simply disappears.
# No error. No retry. No record.
def process_subscription_payment(subscription_id, amount, currency,
                                  customer_id, payment_method_id,
                                  billing_period_start):
    # Step 1: Charge the customer via Stripe PaymentIntents API
    # This call takes 4–8 seconds (Stripe API latency).
    # PROBLEM: No idempotency_key — if this task is ever retried,
    # a second PaymentIntent will be created, double-charging the customer.
    intent = stripe.PaymentIntent.create(
        amount=amount,
        currency=currency,
        customer=customer_id,
        payment_method=payment_method_id,
        confirm=True,
        off_session=True,
    )

    # Step 2: Save the PaymentIntent ID to the application database.
    # If the worker is killed between Step 1 returning and this line executing,
    # the Stripe charge exists but payment_intent_id is never saved.
    # The application never knows the payment was made.
    Subscription.objects.filter(id=subscription_id).update(
        payment_intent_id=intent.id,
        billing_period_start=billing_period_start,
        status='active',
    )

The CI/CD pipeline was configured to deploy on every merge to the main branch. In the five weeks between September 1 and October 1, there were 10 deployments: September 1, 3, 8, 10, 15, 17, 22, 24, 29, and October 1. Each rolling deployment replaced one pod at a time, sending SIGTERM to a running worker pod and giving it 30 seconds to drain before sending SIGKILL. Subscription payment tasks were being created continuously by users upgrading or renewing subscriptions. Because acks_late=False was in effect, each task message was removed from the Redis queue the moment a worker received it. During each 30-second rolling deploy window, approximately 9 tasks were in flight across the 4 worker pods — some of them having already received their task message (acknowledged from Redis) but not yet completed. When SIGKILL arrived after the grace period, those tasks were terminated mid-execution.

# Failure scenario during rolling deploy — per affected worker pod:
#
# t=0s    Kubernetes sends SIGTERM to worker pod
# t=0s    Worker pod receives SIGTERM; begins 30-second drain
# t=0s    Worker pod has already received (and acknowledged) 2–3 tasks
#          from Redis — messages are GONE from the broker queue
#
# For each in-flight task:
#
# Scenario A: Killed before stripe.PaymentIntent.create call (~20% of cases)
#   → No Stripe charge, no DB record. Safe: silent loss only.
#   → Customer is not charged. Subscription not activated.
#   → No financial harm, but subscription silently fails.
#
# Scenario B: Killed after stripe.PaymentIntent.create returns (~80% of cases)
#   → Stripe API call took 4–8 seconds; completed successfully.
#   → Stripe PaymentIntent status=succeeded. Customer IS charged.
#   → Worker killed before Subscription.objects.filter().update() runs.
#   → payment_intent_id never saved. Subscription never activated.
#   → Stripe Dashboard: charge exists. Application DB: no record.
#   → Customer debited. No subscription. No receipt. No error logged.
#
# t=30s   Kubernetes sends SIGKILL. Pod terminated.
#          All in-flight tasks lost. No requeue. No error.
#
# Per deployment: ~9 tasks in flight across 4 pods during grace period
# ~80% of those = ~7 tasks in Scenario B (orphaned Stripe charge)
# 10 deployments × ~9 lost tasks = ~89 tasks total
# Stripe Dashboard: 89 PaymentIntents status=succeeded, no DB record

The reconciliation team’s October 6 audit queried the Stripe API for all PaymentIntents created since September 1 with status=succeeded and cross-referenced each payment_intent_id against the subscriptions table in the application database. The 89 orphaned PaymentIntents had no matching row: the payment_intent_id column was NULL for those subscriptions, and the status column remained pending rather than active. The affected customers had been debited by Stripe but had not received active subscriptions, welcome emails, or any other indication that their payment had been processed. Total exposure: $4,183 across 89 affected customers, with individual charges ranging from $29 to $99 depending on the subscription tier selected.

# Reconciliation query — October 6, 2026
# Step 1: Fetch all succeeded PaymentIntents from Stripe since September 1
import stripe
from datetime import datetime, timezone

intents = stripe.PaymentIntent.list(
    created={"gte": int(datetime(2026, 9, 1, tzinfo=timezone.utc).timestamp())},
    limit=100,
)
stripe_intent_ids = {
    pi.id for pi in intents.auto_paging_iter()
    if pi.status == "succeeded"
}
# stripe_intent_ids: 341 total succeeded PaymentIntents since September 1

# Step 2: Find which ones have no matching application DB record
from myapp.models import Subscription
from django.db.models import Q

db_intent_ids = set(
    Subscription.objects.filter(
        payment_intent_id__in=stripe_intent_ids
    ).values_list('payment_intent_id', flat=True)
)

orphaned = stripe_intent_ids - db_intent_ids
# orphaned: 89 PaymentIntents with no corresponding active subscription

# Step 3: Tally the financial exposure
orphaned_intents = [
    pi for pi in stripe.PaymentIntent.list(limit=100).auto_paging_iter()
    if pi.id in orphaned
]
total_cents = sum(pi.amount for pi in orphaned_intents)
# total_cents: 418300  →  $4,183.00 charged, not credited

Why acks_late=False and Kubernetes rolling deploys are a dangerous default combination

Celery’s default acks_late=False is a reasonable choice for many task types. For tasks that are safe to lose (cache warming, analytics event logging, non-critical notifications), early acknowledgment is efficient: the broker does not need to track the message through task execution, and the worker does not need to hold a connection to the broker for the full task duration. The message is gone from the queue as soon as a worker picks it up, which reduces broker memory pressure and simplifies worker concurrency management.

The default becomes dangerous specifically when two conditions are both true: (1) the task performs an operation that cannot be safely lost or must be reconciled with an external system, and (2) the worker process can be killed mid-task by something other than a Celery-managed shutdown. Condition (2) is reliably satisfied by Kubernetes rolling deployments. Every merge-to-main deploy — a routine, expected, non-incident event in any team with CI/CD automation — sends SIGTERM to running worker pods. With a 30-second grace period and tasks that take 4–8 seconds to complete a Stripe API call, a meaningful fraction of in-flight tasks will not finish before SIGKILL arrives. At three deployments per week, the loss rate accumulates steadily and silently.

The failure is invisible to standard Celery monitoring because there is no failure to monitor: from Celery’s perspective, the task was acknowledged (done) before the worker was shut down gracefully (normal operation). Flower shows no failed tasks. Sentry captures no exceptions. The Redis queue drains normally. The only observable signal is a discrepancy between the external system (Stripe) and the application database, which surfaces only in a periodic reconciliation audit.

The fix: acks_late=True, reject_on_worker_lost=True, and a Stripe idempotency key

Part 1: Set acks_late=True on the task. With acks_late=True, the broker does not acknowledge (remove) the message until the task function returns successfully. If the worker is killed before the task completes — whether by SIGKILL during a Kubernetes rolling deploy, an OOM event, or any other process termination — the message remains in the Redis broker queue and is redelivered to another worker. The task will be retried automatically without any explicit retry configuration.

Part 2: Set reject_on_worker_lost=True on the task. Without this flag, when a worker is killed while holding an unacknowledged message, the message is left in an unacknowledged state in Redis. Redis will re-deliver it after the consumer connection times out, but the timing is determined by the Redis connection timeout configuration rather than by Celery. reject_on_worker_lost=True causes Celery to explicitly reject (requeue) the message when the worker process is lost, ensuring immediate redelivery to another available worker rather than waiting for a connection timeout.

Part 3: Add a Stripe idempotency key to stripe.PaymentIntent.create. With acks_late=True and reject_on_worker_lost=True, the task will be retried after a worker is killed. If the original worker was killed after stripe.PaymentIntent.create returned (Scenario B above), the retry will call stripe.PaymentIntent.create a second time for the same subscription. Without an idempotency key, Stripe will create a second PaymentIntent and charge the customer twice. An idempotency key derived from the subscription ID and billing period start date makes the Stripe call safe to retry: Stripe will return the original PaymentIntent rather than creating a new one, regardless of how many times the call is made with the same key.

# tasks/billing.py — fixed
# Celery 5.3.6, Python 3.11

from celery import shared_task
import stripe
from myapp.models import Subscription

@shared_task(
    acks_late=True,            # Fix 1: acknowledge AFTER task function returns
                               # If worker is killed mid-task, message stays in Redis
                               # and is redelivered to another worker automatically.
    reject_on_worker_lost=True # Fix 2: explicitly reject (requeue) if worker process
                               # is killed rather than waiting for connection timeout.
                               # Ensures immediate redelivery during rolling deploys.
)
def process_subscription_payment(subscription_id, amount, currency,
                                  customer_id, payment_method_id,
                                  billing_period_start):
    # Fix 3: Stripe idempotency key derived from subscription_id + billing_period_start.
    # This makes the PaymentIntent.create call safe to retry:
    # if this task is retried after a worker kill, Stripe returns the SAME PaymentIntent
    # rather than creating a new charge. The customer is never double-charged.
    idempotency_key = f"sub-pay-{subscription_id}-{billing_period_start}"

    intent = stripe.PaymentIntent.create(
        amount=amount,
        currency=currency,
        customer=customer_id,
        payment_method=payment_method_id,
        confirm=True,
        off_session=True,
        idempotency_key=idempotency_key,  # safe to retry ✓
    )

    # This database write is now protected:
    # If the worker is killed after stripe.PaymentIntent.create returns,
    # the message is requeued (not acknowledged). The retrying worker will call
    # stripe.PaymentIntent.create again with the same idempotency_key — Stripe
    # returns the existing succeeded PaymentIntent — and then saves payment_intent_id.
    Subscription.objects.filter(id=subscription_id).update(
        payment_intent_id=intent.id,
        billing_period_start=billing_period_start,
        status='active',
    )

Remediation of the 89 orphaned PaymentIntents. For each of the 89 orphaned PaymentIntents, the Stripe payment_intent_id was located via the reconciliation query and manually written to the corresponding Subscription row. Subscription status was set to active and the subscription’s welcome email was triggered retroactively. All 89 affected customers received their active subscriptions. No refunds were necessary because the charges were legitimate; the only error was the application’s failure to record and act on the successful payment.

# Remediation: restore 89 orphaned subscriptions
# For each orphaned PaymentIntent, find the subscription by customer_id + amount + timestamp
# and backfill the payment_intent_id + activate the subscription.

import stripe
from myapp.models import Subscription
from myapp.tasks import send_welcome_email

remediated = 0
for pi_id in orphaned:
    pi = stripe.PaymentIntent.retrieve(pi_id)
    customer_id = pi.customer
    amount = pi.amount
    # Stripe metadata included subscription_id (set at PaymentIntent creation time)
    subscription_id = pi.metadata.get('subscription_id')
    if not subscription_id:
        # Fall back: match by customer_id + amount + created timestamp proximity
        # (manual review for the 3 cases without metadata)
        continue

    updated = Subscription.objects.filter(
        id=subscription_id,
        payment_intent_id__isnull=True,  # only orphaned rows
    ).update(
        payment_intent_id=pi_id,
        status='active',
    )
    if updated:
        send_welcome_email.delay(subscription_id)
        remediated += 1

# remediated: 86 subscriptions restored automatically
# remaining 3: manual review (no subscription_id metadata) → resolved manually
# Total: 89/89 subscriptions restored. $4,183 fully credited. ✓

Total engagement time: 5.5 hours (1.5h diagnosis: Stripe API reconciliation confirming 89 orphaned PaymentIntents totaling $4,183; cross-reference with application database confirming payment_intent_id=NULL on 89 subscription rows; Celery task source code audit in tasks/billing.py confirming acks_late=False default and no idempotency key; Kubernetes Deployment manifest review confirming terminationGracePeriodSeconds: 30 and rolling update configuration; deployment history review confirming 10 deployments in 5 weeks correlating with task loss windows; 2.5h fix: acks_late=True and reject_on_worker_lost=True added to task decorator; Stripe idempotency key f"sub-pay-{subscription_id}-{billing_period_start}" added to stripe.PaymentIntent.create; fix deployed and rolling deploy tested to confirm task requeue behavior; post-deploy monitoring window confirming no new orphaned PaymentIntents; 1.5h remediation: Stripe metadata extraction for 86 orphaned PaymentIntents; automated backfill script restoring payment_intent_id and activating subscriptions; manual review and resolution of remaining 3 cases without subscription_id metadata; welcome email dispatch confirmed for all 89 customers).

Why acks_late misconfiguration is a recurring Celery retainer pattern

The Harborview Commerce Technology incident illustrates a structural mismatch between Celery’s default configuration and the deployment patterns that are standard in modern infrastructure. Celery was designed in an era when workers were long-running daemons on persistent VMs, not ephemeral Kubernetes pods cycled on every CI/CD push. The acks_late=False default made sense when worker restarts were rare and operator-initiated: the operator would drain queues, stop workers gracefully, and restart. In that context, early acknowledgment was a reasonable optimization.

In a Kubernetes environment with rolling deployments triggered by CI/CD automation, worker pods are routinely terminated and replaced. The assumption that “if a worker received a task, it will complete it” no longer holds. The 30-second grace period is long enough for most tasks but not all — particularly tasks that make outbound API calls to payment processors, where response times can be 4–10 seconds and the downstream consequences of a partial execution (charge made, database write not made) are financially significant.

The combination that makes this class of incident reliably recurring: acks_late=False is the default (developers must explicitly opt in to safe behavior), Kubernetes rolling deploys are the standard deployment pattern (not a special event), external API calls within tasks are extremely common (Stripe, SendGrid, Twilio, Salesforce connectors are ubiquitous), and the failure is invisible to all standard monitoring (no exceptions, no failed tasks, no queue depth anomaly). A Celery developer retainer who audits all tasks that call external APIs or perform multi-step database writes for acks_late coverage — and reviews new task definitions as part of a pre-deploy checklist — catches this class of issue before production accumulates 89 orphaned PaymentIntents and $4,183 in unrecorded charges.

How HourTab helps Celery developer retainers

The Harborview Commerce Technology engagement totaled 5.5 hours: diagnosis (1.5h — 0.5h Stripe API reconciliation: fetched all 341 succeeded PaymentIntents since September 1; cross-referenced against application database; confirmed 89 orphaned with payment_intent_id=NULL on corresponding subscription rows; total financial exposure $4,183; 0.25h Celery task source code audit in tasks/billing.py: confirmed acks_late not set (defaulting to False), no reject_on_worker_lost, no Stripe idempotency key; 0.25h Kubernetes Deployment manifest review: confirmed 4-replica rolling update with maxUnavailable=1, maxSurge=1, terminationGracePeriodSeconds: 30; confirmed Stripe API call latency 4–8 seconds from application logs; 0.5h deployment history review: 10 deployments between September 1 and October 1 correlating with task loss windows; estimated ~9 tasks in flight per deploy across 4 pods × 30-second grace period × task throughput; 89 total matches estimated loss), fix (2.5h — 0.5h code change: acks_late=True + reject_on_worker_lost=True added to @shared_task decorator; Stripe idempotency key f"sub-pay-{subscription_id}-{billing_period_start}" added to stripe.PaymentIntent.create call; 0.5h local test: simulated SIGTERM during task execution with acks_late=True; confirmed message requeued and picked up by second worker; confirmed idempotency key prevents duplicate Stripe charge on retry; 1.0h deploy fix to production; monitored rolling deploy: confirmed in-flight tasks requeued and completed by remaining workers; 0.5h post-deploy monitoring window: no new orphaned PaymentIntents after first deploy with fix), and remediation (1.5h — 0.5h Stripe metadata extraction and automated backfill script; 86 subscriptions restored automatically via payment_intent_id backfill + status=active + welcome email dispatch; 0.5h manual review of 3 remaining cases without subscription_id metadata: identified subscriptions by customer_id + amount + created timestamp proximity; resolved manually; 0.5h final reconciliation query confirming 0 orphaned PaymentIntents outstanding; all 89 customers confirmed with active subscriptions).

In a HourTab retainer log, the incident session reads: “process_subscription_payment task loss: Celery 5.3.6 Python 3.11; Redis 7.2.4 ElastiCache; Kubernetes 1.29 EKS; 4-replica Deployment rolling update terminationGracePeriodSeconds=30; acks_late=False default (broker acknowledges message on receipt before task runs); rolling deploy sends SIGTERM; worker killed mid-task after stripe.PaymentIntent.create returns but before Subscription DB update; message already gone from Redis (acknowledged); no retry, no error; 10 deploys Sep 1–Oct 1; ~9 tasks lost per deploy; 89 orphaned PaymentIntents status=succeeded; $4,183 charged not credited; discovery Oct 6 monthly Stripe reconciliation; fix: acks_late=True + reject_on_worker_lost=True + idempotency key f’sub-pay-{subscription_id}-{billing_period_start}’; 89 subscriptions manually restored; welcome emails dispatched; 5.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Celery developer retainers

See also: Apache Airflow developer retainer · Kafka developer retainer · Apache Flink developer retainer