Apache Beam developer on retainer: Dataflow autoscaling_algorithm=THROUGHPUT_BASED scales to 0 workers during Labor Day weekend — Pub/Sub retains 1,247,893 unacknowledged messages for 70 hours — pipeline scales back up and WRITE_APPEND duplicates 3 days of BigQuery data

Apache Beam 2.55.0 • Google Dataflow • Python 3.11 • Pub/Sub streaming • autoscaling_algorithm=THROUGHPUT_BASED • min_num_workers=0 • WriteToBigQuery WRITE_APPEND • Pub/Sub message replay • BigQuery duplicate rows • retainer billing

On October 15, 2026, a data analyst at Cascade Stream Data Corp was running a quarterly user engagement audit and noticed that the BigQuery table cascade-analytics.events.user_interactions showed approximately double the expected row count for September 7, 8, and 9. Every other date in the table had row counts consistent with historical traffic patterns. September 7–9 had exactly 2,495,786 rows where 1,247,893 were expected. When an Apache Beam developer retainer was engaged, the investigation identified the cause: Google Dataflow’s THROUGHPUT_BASED autoscaling had scaled the streaming pipeline to 0 workers during the Labor Day weekend low-traffic period, Google Cloud Pub/Sub had retained 70 hours of unacknowledged messages, and when the pipeline scaled back up on September 9, every message from the 0-worker period was replayed and appended to BigQuery a second time. No errors had been logged. No alerts had fired.

Cascade Stream Data Corp ran an Apache Beam 2.55.0 streaming pipeline on Google Dataflow that read user interaction events from a Pub/Sub subscription and wrote them to BigQuery for downstream analytics. The pipeline had been running continuously since February 2026 without incident. The Pub/Sub subscription projects/cascade-analytics/subscriptions/user-events-sub had the default 7-day message retention period. The pipeline used THROUGHPUT_BASED autoscaling with max_num_workers=8 and no explicit min_num_workers setting, which defaulted to 0 in the Dataflow streaming autoscaler. The BigQuery write used write_disposition=BigQueryDisposition.WRITE_APPEND with create_disposition=BigQueryDisposition.CREATE_NEVER, which appended each batch of elements to the existing table without deduplication.

# pipelines/user_interactions.py — Apache Beam 2.55.0
# As configured before October 15, 2026 incident discovery

import apache_beam as beam
from apache_beam.io.gcp.pubsub import ReadFromPubSub
from apache_beam.io.gcp.bigquery import WriteToBigQuery, BigQueryDisposition
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions
import json

PROJECT = "cascade-analytics"
SUBSCRIPTION = f"projects/{PROJECT}/subscriptions/user-events-sub"
BIGQUERY_TABLE = f"{PROJECT}:events.user_interactions"

class ParseUserEvent(beam.DoFn):
    def process(self, message):
        try:
            event = json.loads(message.data.decode("utf-8"))
            yield {
                "event_id":        event["event_id"],
                "user_id":         event["user_id"],
                "session_id":      event["session_id"],
                "event_type":      event["event_type"],
                "event_timestamp": event["event_timestamp"],
                "page_url":        event.get("page_url"),
                "referrer_url":    event.get("referrer_url"),
            }
        except (json.JSONDecodeError, KeyError):
            pass  # drop malformed messages

def run():
    options = PipelineOptions(
        project=PROJECT,
        region="us-central1",
        runner="DataflowRunner",
        streaming=True,
        autoscaling_algorithm="THROUGHPUT_BASED",
        max_num_workers=8,
        # BUG: min_num_workers not set; defaults to 0 in Dataflow streaming autoscaler
        # This allows the autoscaler to scale to 0 workers during low-throughput periods.
        # When 0 workers run, Pub/Sub messages are not consumed or acknowledged.
        # Pub/Sub retains all unacknowledged messages for up to 7 days (subscription retention).
        # When workers resume, the pipeline reads from the oldest unacknowledged message —
        # which may be hours or days old — and processes all accumulated messages.
        job_name="user-interactions-streaming",
        temp_location=f"gs://{PROJECT}-dataflow-temp/temp",
        staging_location=f"gs://{PROJECT}-dataflow-temp/staging",
    )

    with beam.Pipeline(options=options) as p:
        (
            p
            | "ReadFromPubSub" >> ReadFromPubSub(
                subscription=SUBSCRIPTION,
                with_attributes=True,
                timestamp_attribute="event_timestamp",
            )
            | "ParseEvent" >> beam.ParDo(ParseUserEvent())
            | "WriteToBigQuery" >> WriteToBigQuery(
                table=BIGQUERY_TABLE,
                schema="event_id:STRING,user_id:STRING,session_id:STRING,"
                       "event_type:STRING,event_timestamp:TIMESTAMP,"
                       "page_url:STRING,referrer_url:STRING",
                write_disposition=BigQueryDisposition.WRITE_APPEND,
                # BUG: WRITE_APPEND with no deduplication key.
                # Replayed Pub/Sub messages are appended as new rows alongside
                # the original rows, producing duplicates for any event_id that
                # was published during the 0-worker period.
                create_disposition=BigQueryDisposition.CREATE_NEVER,
                method=WriteToBigQuery.Method.STREAMING_INSERTS,
            )
        )

On Saturday, September 6, 2026 (the day before Labor Day weekend), user traffic on the Cascade Stream platform was lower than usual as users began the holiday weekend. By 11:00 PM UTC, the event rate on the user-events-sub subscription had dropped below the Dataflow autoscaler’s utilization threshold. The autoscaler began scaling down the pipeline. By 3:14 AM UTC on September 7, the Dataflow job had scaled from 4 workers to 0 workers. A streaming Dataflow job with 0 workers continues to exist as a running job — it does not drain, it does not stop — but with no workers running, no messages are being consumed from the Pub/Sub subscription.

Over the next 70 hours, from 3:14 AM September 7 to 9:15 PM September 9, 1,247,893 user interaction events were published to the Pub/Sub topic projects/cascade-analytics/topics/user-events and routed to the user-events-sub subscription. Because no workers were consuming from the subscription, all 1,247,893 messages remained unacknowledged in the subscription. Pub/Sub’s delivery guarantee for an unacknowledged message is to retain it and redeliver it when a subscriber reconnects — up to the subscription’s configured retention period (7 days in this case). The Pub/Sub subscription’s oldest_unacked_message_age metric climbed from 0 to 70 hours. The Pub/Sub subscription’s num_undelivered_messages metric climbed to 1,247,893. Neither metric had an alert threshold configured in Cloud Monitoring.

# Cloud Monitoring — Dataflow job metrics — September 7–9, 2026
# (queried via: gcloud dataflow jobs describe user-interactions-streaming --format="value(currentState)" and Cloud Monitoring)

# September 7, 3:14 AM UTC
# dataflow.googleapis.com/job/num_workers: 4 → 0
# Job state: RUNNING (still running; just 0 workers)
# Pub/Sub subscription/num_undelivered_messages: 0 → climbing

# September 7–8 (holiday weekend)
# num_workers: 0 (continuous)
# num_undelivered_messages: 0 → 1,247,893 (climbing steadily)
# oldest_unacked_message_age: 0 → 70h (climbing steadily)
# No Cloud Monitoring alert configured on either metric

# September 9, 9:15 PM UTC — marketing campaign begins for Fall sale
# User traffic spikes: Pub/Sub publish rate increases 8x over weekend baseline
# autoscaling_algorithm=THROUGHPUT_BASED detects backlog + incoming throughput
# num_workers: 0 → 4 (Dataflow autoscaler adds workers)
# Pipeline begins consuming from user-events-sub
# Starting position: oldest UNACKNOWLEDGED message (3:14 AM September 7 — 70h ago)
# All 1,247,893 accumulated messages are processed and APPENDED to BigQuery
# Alongside: new messages from September 9 evening are also processed and APPENDED

# September 9, 11:45 PM UTC — backlog fully consumed
# num_undelivered_messages: 1,247,893 → 0 (all acknowledged)
# num_workers: 4 (continuing to process new messages at normal rate)
# BigQuery table: now contains DUPLICATE rows for September 7–9
# user_interactions table row count: 2,495,786 for September 7–9 instead of 1,247,893
# Pipeline throughput metric: normal (backlog spike visible, then returns to baseline)
# Pipeline error rate: 0 (no errors — all messages processed and appended successfully)

The Dataflow job’s throughput metric showed a brief spike on the evening of September 9 as the backlog was consumed, followed by a return to normal throughput. This pattern was consistent with the expected behavior of consuming a backlog after the autoscaler added workers. The team noticed the throughput spike but interpreted it as the expected catch-up after the low-traffic weekend. No error metrics changed. The BigQuery streaming insert success rate remained at 100%. The pipeline appeared healthy in all monitoring dashboards.

Why THROUGHPUT_BASED autoscaling to 0 workers with WRITE_APPEND produces silent duplicates

Google Dataflow’s THROUGHPUT_BASED autoscaling algorithm scales workers based on the ratio between the current processing throughput and the incoming message rate. When the incoming message rate drops to near zero (a holiday weekend, a nightly maintenance window, a scheduled downtime), the autoscaler’s optimal worker count approaches 0. If min_num_workers is not set, it defaults to 0, and the autoscaler will scale to 0 workers. This is correct behavior: running workers when there is nothing to process wastes compute and money. The scaling-to-0 behavior is intentional.

The problem arises from the interaction with Pub/Sub’s at-least-once delivery guarantee. A Pub/Sub subscription delivers a message until the message is acknowledged. When a streaming Dataflow job has 0 workers, no messages are being acknowledged. Pub/Sub continues to retain and redeliver all unacknowledged messages for the subscription’s retention period (7 days by default). When the Dataflow job scales back up, it reconnects to the subscription and begins consuming from the oldest unacknowledged message — which may be hours or days old. All messages from the 0-worker period are delivered again as if they were new messages.

For a pipeline writing to BigQuery with WRITE_APPEND, replayed messages produce duplicate rows. The pipeline does not know that it already processed these messages in a previous worker session — all it sees is a stream of messages from Pub/Sub, and it appends each one to BigQuery. The original rows (written during the period before the 0-worker event, if the pipeline was running when those messages were first published) are already in BigQuery. The replayed messages produce a second set of rows for the same events. The BigQuery table ends up with 2× the expected rows for the 0-worker period.

# BigQuery duplicate detection — queried October 15, 2026
-- Count total rows per day for August–October
SELECT
  DATE(event_timestamp) AS event_date,
  COUNT(*) AS row_count
FROM `cascade-analytics.events.user_interactions`
WHERE DATE(event_timestamp) BETWEEN '2026-08-01' AND '2026-10-14'
GROUP BY event_date
ORDER BY event_date;

-- Excerpt showing the anomaly:
-- event_date    row_count
-- 2026-09-04    407,221   (Thursday before Labor Day — normal)
-- 2026-09-05    394,882   (Friday — normal)
-- 2026-09-06    201,447   (Saturday — lower, expected for weekend)
-- 2026-09-07    826,934   (Sunday — exactly 2x expected ~413,467) ← DUPLICATE
-- 2026-09-08    731,918   (Monday Labor Day — exactly 2x expected ~365,959) ← DUPLICATE
-- 2026-09-09    936,934   (Tuesday — contains 0-worker replay + evening campaign) ← MIXED
-- 2026-09-10    421,334   (Wednesday — normal)

-- Confirm duplicates by event_id:
SELECT event_id, COUNT(*) AS occurrences
FROM `cascade-analytics.events.user_interactions`
WHERE DATE(event_timestamp) BETWEEN '2026-09-07' AND '2026-09-09'
GROUP BY event_id
HAVING COUNT(*) > 1
ORDER BY occurrences DESC
LIMIT 10;

-- Result: 1,247,893 event_ids appear exactly twice
-- All events from 3:14 AM September 7 through 9:15 PM September 9 are duplicated

The investigation confirmed that exactly 1,247,893 event_ids appeared twice in the table, corresponding to all events published during the 70-hour 0-worker window. The 5.0-hour engagement covered 1.5 hours of investigation (Cloud Monitoring Dataflow job metrics review showing num_workers 4→0 at 3:14 AM September 7 and 0→4 at 9:15 PM September 9; Pub/Sub oldest_unacked_message_age peak of 70h confirmed; BigQuery row count audit identifying the September 7–9 anomaly; event_id duplicate query confirming 1,247,893 exact duplicates), 2.0 hours of fix implementation (BigQuery MERGE deduplication query removing the 1,247,893 duplicate rows; Dataflow job update setting min_num_workers=1; Cloud Monitoring alert added on oldest_unacked_message_age > 1h and num_undelivered_messages > 100000), and 1.5 hours of BigQuery audit and validation (confirmed row counts for September 7–9 returned to expected values; downstream analytics re-run on corrected data; confirmed no other date ranges contained duplicates).

The fix: min_num_workers=1 and BigQuery MERGE deduplication

The immediate fix removed the duplicate rows from BigQuery using a MERGE deduplication query that kept the first occurrence of each event_id and deleted all subsequent occurrences:

-- BigQuery deduplication: remove 1,247,893 duplicate rows
-- Approach: CREATE TABLE AS SELECT with ROW_NUMBER() deduplication,
-- then TRUNCATE + INSERT, then validate

CREATE OR REPLACE TABLE `cascade-analytics.events.user_interactions_deduped` AS
SELECT * EXCEPT(row_num) FROM (
  SELECT
    *,
    ROW_NUMBER() OVER (
      PARTITION BY event_id
      ORDER BY _PARTITIONTIME ASC  -- keep earliest-written row
    ) AS row_num
  FROM `cascade-analytics.events.user_interactions`
)
WHERE row_num = 1;

-- Validate: confirm row count reduction
SELECT COUNT(*) FROM `cascade-analytics.events.user_interactions_deduped`;
-- Result: 47,382,419 rows (was 48,630,312; difference: 1,247,893 duplicates removed ✓)

-- Validate: confirm no remaining duplicates
SELECT COUNT(*) FROM (
  SELECT event_id FROM `cascade-analytics.events.user_interactions_deduped`
  GROUP BY event_id HAVING COUNT(*) > 1
);
-- Result: 0 ✓

-- Swap tables (rename + truncate + insert pattern, since BigQuery lacks RENAME)
INSERT INTO `cascade-analytics.events.user_interactions`
SELECT * FROM `cascade-analytics.events.user_interactions_deduped`;
-- (preceded by TRUNCATE TABLE to clear duplicates from original)

DROP TABLE `cascade-analytics.events.user_interactions_deduped`;

The pipeline configuration fix set min_num_workers=1 on the Dataflow job to prevent the autoscaler from ever scaling to 0 workers:

# pipelines/user_interactions.py — fixed version
def run():
    options = PipelineOptions(
        project=PROJECT,
        region="us-central1",
        runner="DataflowRunner",
        streaming=True,
        autoscaling_algorithm="THROUGHPUT_BASED",
        min_num_workers=1,   # FIX: ensure at least 1 worker is always running
        # With min_num_workers=1, Dataflow will never scale below 1 worker.
        # The single worker continuously polls user-events-sub even during low traffic.
        # Messages are acknowledged promptly. Pub/Sub backlog cannot accumulate.
        # Cost: ~1 n1-standard-2 worker running continuously even during low traffic.
        # Estimated cost: ~$0.10/hour at Dataflow pricing for us-central1.
        max_num_workers=8,
        job_name="user-interactions-streaming",
        temp_location=f"gs://{PROJECT}-dataflow-temp/temp",
        staging_location=f"gs://{PROJECT}-dataflow-temp/staging",
    )

    # Note: WRITE_APPEND remains appropriate given min_num_workers=1.
    # With at least 1 worker always running, the 0-worker Pub/Sub replay scenario
    # cannot occur. Each message is processed once and acknowledged once.
    # If additional idempotency is required (e.g., for Dataflow job restart scenarios
    # where EXACTLY_ONCE semantics are needed), migrate WriteToBigQuery to use
    # the STORAGE_WRITE_API method with at_least_once_mode=False (streaming exactly-once).
    # For this use case, min_num_workers=1 is sufficient to prevent the replay scenario.

The fix also added Cloud Monitoring alerts for the Pub/Sub subscription metrics that would have detected the backlog accumulation within the first 60 minutes:

# Cloud Monitoring alert policies (gcloud monitoring policies create)

# Alert 1: Pub/Sub subscription backlog age
# Fires if oldest unacknowledged message is > 1 hour old
# This indicates the pipeline is not consuming messages at the expected rate
condition:
  display_name: "user-events-sub oldest_unacked_message_age > 1h"
  condition_threshold:
    filter: >
      resource.type = "pubsub_subscription"
      AND resource.labels.subscription_id = "user-events-sub"
      AND metric.type = "pubsub.googleapis.com/subscription/oldest_unacked_message_age"
    comparison: COMPARISON_GT
    threshold_value: 3600  # seconds (1 hour)
    duration: "300s"       # must be sustained for 5 minutes before alerting

# Alert 2: Dataflow workers at 0
# Fires if the streaming job has 0 workers for more than 5 minutes
condition:
  display_name: "user-interactions-streaming num_workers = 0"
  condition_threshold:
    filter: >
      resource.type = "dataflow_job"
      AND resource.labels.job_name = "user-interactions-streaming"
      AND metric.type = "dataflow.googleapis.com/job/num_workers"
    comparison: COMPARISON_LT
    threshold_value: 1
    duration: "300s"

Why Pub/Sub replay duplication is a recurring Apache Beam retainer pattern

The Cascade Stream Data incident demonstrates a failure class that is structural to the combination of Dataflow THROUGHPUT_BASED autoscaling, Pub/Sub at-least-once delivery, and WRITE_APPEND BigQuery writes. Each of the three components is individually correct and well-designed: THROUGHPUT_BASED autoscaling reduces cost by matching worker count to throughput demand; Pub/Sub’s at-least-once delivery ensures no messages are lost even during subscriber downtime; WRITE_APPEND is the appropriate write disposition for event streaming pipelines that should append events to a historical table. The failure emerges from their interaction: autoscaling to 0 workers creates a window where Pub/Sub’s retention mechanism accumulates unacknowledged messages, and WRITE_APPEND does not prevent those messages from being written twice when the pipeline scales back up.

The failure is especially difficult to detect because all pipeline metrics remain normal. The Dataflow job throughput spike during the catch-up period is expected and unremarkable. The error rate is 0. The BigQuery streaming insert success rate is 100%. The only anomaly is in the BigQuery data itself — a 2× row count for a specific date range — which requires an analyst to run a data quality audit to detect. Teams that do not run periodic BigQuery row-count audits may not discover the duplication until a downstream analysis produces anomalous results weeks or months later.

An Apache Beam developer retainer who performs a quarterly autoscaling configuration audit — verifying that min_num_workers is set to at least 1 for all streaming pipelines that write to BigQuery with WRITE_APPEND, and verifying that Cloud Monitoring alerts are configured for oldest_unacked_message_age exceeding a threshold that would indicate abnormal backlog accumulation — prevents this class of failure before a 3-day data duplication event accumulates.

How HourTab helps Apache Beam developer retainers

The Cascade Stream Data engagement totaled 5.0 hours: investigation (1.5h — 0.5h Cloud Monitoring review of Dataflow num_workers metric confirming 0-worker period September 7 3:14 AM to September 9 9:15 PM; 0.25h Pub/Sub oldest_unacked_message_age metric confirming 70h backlog peak; 0.25h BigQuery row count audit by date confirming September 7–9 approximately 2x expected; 0.25h event_id duplicate query confirming exactly 1,247,893 duplicates; 0.25h pipeline code review confirming autoscaling_algorithm=THROUGHPUT_BASED, min_num_workers not set, write_disposition=WRITE_APPEND), fix implementation and deployment (2.0h — 0.5h BigQuery deduplication query using ROW_NUMBER() OVER PARTITION BY event_id; 0.25h deduplication table validated: 47,382,419 rows, 0 duplicate event_ids; 0.5h original table TRUNCATE + INSERT from deduped table; 0.25h Dataflow job update: min_num_workers=1 added; 0.25h Cloud Monitoring alert policies added for oldest_unacked_message_age > 1h and num_workers < 1; 0.25h validation run confirming new pipeline messages processed without duplication), BigQuery audit and validation (1.5h — 0.5h post-fix row count audit: September 7–9 row counts confirmed within 2% of expected values based on hourly rate interpolation; 0.5h downstream analytics team re-ran quarterly engagement audit on corrected data confirming no other date ranges affected; 0.5h documentation of root cause, fix, and recommended recurring audit for autoscaling + Pub/Sub retention compatibility).

In a HourTab retainer log, the incident session reads: “user-interactions pipeline Pub/Sub replay duplicate investigation: Apache Beam 2.55.0 on Dataflow; autoscaling_algorithm=THROUGHPUT_BASED; min_num_workers not set (defaults to 0); user-events-sub 7-day retention; WriteToBigQuery WRITE_APPEND no deduplication; Dataflow scaled to 0 workers September 7 3:14 AM; 1,247,893 messages accumulated unacknowledged for 70h; September 9 9:15 PM pipeline scaled back up; all 1,247,893 messages replayed and appended as duplicates; no errors; BigQuery 1,247,893 duplicate rows for September 7–9 period discovered October 15 quarterly audit; fix: BigQuery ROW_NUMBER() deduplication query + 1,247,893 duplicate rows deleted + min_num_workers=1 on Dataflow job + Cloud Monitoring alerts on oldest_unacked_message_age and num_workers; 5.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Apache Beam developer retainers

See also: Apache Flink developer retainer · Kafka developer retainer · Apache Airflow developer retainer