Elasticsearch developer on retainer: dynamic mapping locks session_duration_seconds as long — upstream float string triggers mapper_parsing_exception — Kafka Connect DLQ disabled silently drops 44,940 documents over 35 days

Elasticsearch 8.11.3 • Elastic Cloud • Kafka Connect Elasticsearch Sink 14.0.12 • Confluent Cloud • dynamic mapping • mapper_parsing_exception • dead-letter queue • silent data loss • Black Friday cohort • retainer billing

Northpoint Digital Commerce Corp.’s analytics team was preparing a Black Friday 2026 audience segmentation in early October when they noticed an unexpected anomaly: the high-session-duration user cohort — defined as users in the top 15% of session_duration_seconds for the trailing 90 days — had shrunk by approximately 35% compared to the same cohort constructed 35 days earlier. The drop was not visible in total user counts (which remained stable) but was clearly visible in the session_duration_seconds distribution: values above 200 seconds had a suspicious gap starting September 3, 2026. When the Elasticsearch developer retainer was engaged, the investigation identified that 44,940 session documents had been silently dropped from the product-sessions-v1 Elasticsearch index over a 35-day period. The root cause was a Kafka Connect Elasticsearch Sink connector configured with errors.tolerance=all and no dead-letter queue (DLQ) topic. When an upstream Node.js application bug began sending session_duration_seconds as a float string (“123.45”) instead of an integer on September 3, Elasticsearch rejected every affected document with a mapper_parsing_exception — because the index’s dynamic mapping had already inferred session_duration_seconds as type long from the first documents indexed at index creation, and a long field cannot accept a float string value. With errors.tolerance=all and no DLQ, the connector silently discarded each rejected document, logged the error at DEBUG level, and moved on. Over 35 days, approximately 1,284 documents per day — those whose session_duration_seconds values happened to be sent as float strings by the buggy code path — were dropped without any alert, dead-letter record, or observable change in the Kafka Connect connector status.

Northpoint Digital Commerce Corp. ran a Confluent Cloud Kafka cluster with a user-session-events topic. A Kafka Connect Elasticsearch Sink connector (version 14.0.12) consumed from this topic and indexed documents into Elasticsearch 8.11.3 hosted on Elastic Cloud. The product-sessions-v1 index was created in March 2026 without an explicit index mapping, relying on Elasticsearch’s dynamic mapping to infer field types from the first ingested documents. The initial documents contained session_duration_seconds as an integer value (for example, 5284), and Elasticsearch correctly inferred the field type as long. The Kafka Connect connector was configured as follows:

# Kafka Connect Elasticsearch Sink connector configuration
# connector: product-sessions-elasticsearch-sink
# Confluent Kafka Connect, Elasticsearch Sink Connector 14.0.12

{
  "name": "product-sessions-elasticsearch-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "3",
    "topics": "user-session-events",
    "connection.url": "https://northpoint-prod.es.eastus2.azure.elastic-cloud.com:9243",
    "type.name": "_doc",
    "key.ignore": "true",
    "schema.ignore": "true",
    "errors.tolerance": "all",
    # errors.tolerance=all: connector continues processing after any error
    # Consequence: mapper_parsing_exception does NOT stop the connector task
    # The rejected document is logged at DEBUG level and skipped

    # Missing: errors.deadletterqueue.topic.name
    # Without this, rejected documents are NOT written to any DLQ topic
    # They are permanently lost from the Elasticsearch index
    # The only record is in the connector's internal error metric counter

    "errors.log.enable": "true",
    "errors.log.include.messages": "false",
    # errors.log.include.messages=false: error log does not include the message payload
    # Even the log entry is not actionable for recovery

    "batch.size": "2000",
    "linger.ms": "1000",
    "flush.timeout.ms": "10000"
  }
}

On September 3, 2026, a Node.js application update deployed to production introduced a bug in the session event serialization layer. The session_duration_seconds field, previously always serialized as a JSON integer ("session_duration_seconds": 5284), was now serialized as a string representation of a float for a subset of sessions where the duration was computed via a floating-point division that was not rounded to an integer before serialization ("session_duration_seconds": "123.45"). Approximately 8% of session events per day were produced by the affected code path. Each of these documents, when the Kafka Connect connector attempted to index them into Elasticsearch, was rejected:

// Elasticsearch rejection response for affected documents:
// HTTP 400 Bad Request
{
  "error": {
    "root_cause": [{
      "type": "mapper_parsing_exception",
      "reason": "failed to parse field [session_duration_seconds] of type [long]
                 in document with id 'sess_8f4a2b1c9d3e'. Preview of field's value:
                 '123.45'"
    }],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [session_duration_seconds] of type [long]
               in document with id 'sess_8f4a2b1c9d3e'. Preview of field's value:
               '123.45'",
    "caused_by": {
      "type": "json_parse_exception",
      "reason": "Current token (VALUE_STRING) not numeric, can not use
                 numeric value accessors at [Source: UNKNOWN; line: -1, column: -1]"
    }
  },
  "status": 400
}

// Kafka Connect connector behavior with errors.tolerance=all + no DLQ:
// 1. Connector receives HTTP 400 from Elasticsearch for this document
// 2. errors.tolerance=all: connector does NOT fail or pause the task
// 3. No errors.deadletterqueue.topic.name: document NOT written to any DLQ
// 4. Error logged at DEBUG level (not visible in standard monitoring):
//    "Failed to index document sess_8f4a2b1c9d3e: mapper_parsing_exception"
// 5. errors.count connector metric incremented by 1
// 6. Connector moves on to next document in the batch
// 7. Document permanently absent from product-sessions-v1 index

// Daily impact:
// Total daily session events: ~16,050
// Affected by float string bug: ~8% = ~1,284 documents/day
// Duration: September 3 through October 7 = 35 days
// Total dropped: 35 × 1,284 = 44,940 documents

The Kafka topic user-session-events retained messages for 7 days. By the time the Elasticsearch developer retainer was engaged on October 9, all 35 days of dropped documents had already been purged from the Kafka topic by the retention policy. The errors.count connector metric had accumulated to 44,940 over 35 days — a counter that was not connected to any alert. The Kibana monitoring dashboard for the product-sessions-v1 index showed the document count as 552,915 (total events minus dropped documents) without any visual indication that documents were missing. Only the Black Friday cohort analysis, which compared the session_duration_seconds distribution against a prior-period baseline, surfaced the gap.

Why dynamic mapping type-lock combined with silent error tolerance is a recurring Elasticsearch retainer failure

Elasticsearch’s dynamic mapping feature is designed for rapid development: a new index can start receiving documents without any upfront schema work, and Elasticsearch will automatically create a schema that matches the first data it sees. This is convenient during prototyping but creates a production risk: the inferred schema is based on one document (or the first few documents), and the resulting field types are locked in. The session_duration_seconds field was first seen as an integer, so it became a long. There is no way to change this without reindexing — Elasticsearch explicitly states: “The data type for an existing field mapping is not supported.” If a subsequent source change causes the same field to arrive as a different type, every document containing the incompatible value is rejected.

The key compounding factor is the connector’s errors.tolerance=all configuration without a DLQ. This configuration was chosen in development to prevent connector task failures from halting the entire ingestion pipeline during integration testing, when occasional schema mismatches were expected and tolerated. The setting was never revisited for production deployment. In production, errors.tolerance=all without a DLQ means that any systematic indexing failure — not just the transient schema-mismatch during testing — will silently accumulate without triggering any alert:

// Diagnosis Step 1: Compare Kafka topic message count vs. Elasticsearch document count
// (Kafka retention expired for the 35-day window, so this is retrospective)
// Current Elasticsearch document count:
GET /product-sessions-v1/_count
{
  "query": {
    "range": {
      "session_start": {
        "gte": "2026-09-03",
        "lt": "2026-10-09"
      }
    }
  }
}
// Result: { "count": 507,015 }

// Expected based on daily event volume:
// 16,050 events/day × 35 days = 561,750 expected (had no documents been dropped)
// 561,750 - 507,015 = 54,735 gap
// But 8% affected × 16,050 × 35 = 44,940 confirmed via errors.count metric
// Gap of 54,735 - 44,940 = 9,795 explained by other minor issues (connector restarts, etc.)

// Diagnosis Step 2: Check connector errors.count metric
// Confluent Cloud Kafka Connect metrics:
// connector: product-sessions-elasticsearch-sink
// Metric: kafka_connect_sink_task_metrics_error_record_failures_total
// Value as of October 9: 44,940  ✓ (accumulated since September 3)

// Diagnosis Step 3: Check Elasticsearch mapping for session_duration_seconds
GET /product-sessions-v1/_mapping
// Response (excerpt):
// "session_duration_seconds": {
//   "type": "long"   ← inferred from first document, March 2026
// }

// Diagnosis Step 4: Check sample source documents in upstream application log
// Node.js application event log (Cloudwatch Logs, September 3 onward):
// { "session_duration_seconds": "123.45", "user_id": "u_7f2a3b1c", ... }
// { "session_duration_seconds": "67.20", "user_id": "u_4d8e9f2b", ... }
// Type: float string — incompatible with Elasticsearch long mapping  ✓

// Diagnosis Step 5: Confirm DLQ is not configured
// Connector config review:
// errors.deadletterqueue.topic.name: (not set)
// errors.tolerance: all
// → No DLQ → rejected documents permanently lost from Elasticsearch  ✓

Diagnosis was complete in 1.5 hours. The root cause was the combination of: (1) product-sessions-v1 index using dynamic mapping, which had locked session_duration_seconds as type long from the first document indexed in March 2026; (2) a Node.js application bug introduced on September 3 that serialized session_duration_seconds as a float string for approximately 8% of session events; and (3) the Kafka Connect Elasticsearch Sink configured with errors.tolerance=all and no DLQ, which silently discarded every rejected document without any durable record or alert.

The fix: explicit mapping with float type, upstream bug fix, bulk reindex from application event log

Part 1: Fix the upstream Node.js serialization bug. The application bug was confirmed to be a missing Math.round() call on the session duration calculation. The fix was a one-line change: session_duration_seconds: Math.round(sessionEndMs - sessionStartMs) / 1000 replaced by session_duration_seconds: Math.round((sessionEndMs - sessionStartMs) / 1000). The fix was deployed on October 9, stopping new document drops immediately.

Part 2: Update the Elasticsearch index mapping and configure DLQ. The product-sessions-v1 index was reindexed into a new product-sessions-v2 index with an explicit mapping that defined session_duration_seconds as type float (to accommodate both integer and decimal values from the source system). All other frequently-queried fields were also given explicit types to prevent future dynamic mapping type-lock issues. The Kafka Connect connector was updated to target the new index and to include DLQ configuration:

// Step 1: Create product-sessions-v2 with explicit mapping
PUT /product-sessions-v2
{
  "mappings": {
    "properties": {
      "session_id":              { "type": "keyword" },
      "user_id":                 { "type": "keyword" },
      "session_start":           { "type": "date", "format": "epoch_millis||strict_date_optional_time" },
      "session_end":             { "type": "date", "format": "epoch_millis||strict_date_optional_time" },
      "session_duration_seconds": { "type": "float" },
      // ↑ Changed from dynamically-inferred long to explicit float
      // Accepts both integer (5284) and decimal (123.45) values from source
      "page_views":              { "type": "integer" },
      "event_count":             { "type": "integer" },
      "device_type":             { "type": "keyword" },
      "referrer_domain":         { "type": "keyword" }
    }
  }
}

// Step 2: Reindex existing data from v1 to v2
// (Note: existing long values in v1 are compatible with float in v2 — no data loss)
POST /_reindex
{
  "source": { "index": "product-sessions-v1" },
  "dest":   { "index": "product-sessions-v2" }
}
// 507,015 documents reindexed ✓

// Step 3: Update connector to add DLQ and point to v2
PUT /connectors/product-sessions-elasticsearch-sink/config
{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "tasks.max": "3",
  "topics": "user-session-events",
  "connection.url": "https://northpoint-prod.es.eastus2.azure.elastic-cloud.com:9243",
  "type.name": "_doc",
  "key.ignore": "true",
  "schema.ignore": "true",
  "index": "product-sessions-v2",
  "errors.tolerance": "all",
  "errors.deadletterqueue.topic.name": "user-session-events-dlq",
  // ↑ Now: rejected documents are written to DLQ topic for recovery
  "errors.deadletterqueue.context.headers.enable": "true",
  // ↑ Includes rejection reason in DLQ message headers for diagnosis
  "errors.log.enable": "true",
  "errors.log.include.messages": "true",
  // ↑ Log the rejected message payload for immediate visibility
  "batch.size": "2000",
  "linger.ms": "1000",
  "flush.timeout.ms": "10000"
}

Part 3: Reindex the 44,940 dropped documents from the upstream application event log. The Node.js application wrote all session events to AWS CloudWatch Logs (log group /prod/northpoint/session-events) in addition to publishing them to Kafka. CloudWatch Logs retained logs for 90 days. The 44,940 dropped documents were recovered by exporting the relevant CloudWatch log events (September 3–October 8 where session_duration_seconds matched the float string pattern), transforming them to the correct format, and bulk-indexing them directly into product-sessions-v2 via the Elasticsearch Bulk API:

# Export dropped documents from CloudWatch Logs
aws logs filter-log-events \
  --log-group-name /prod/northpoint/session-events \
  --start-time 1725321600000 \  # September 3, 2026 00:00 UTC
  --end-time 1728345599000 \    # October 8, 2026 23:59 UTC
  --filter-pattern '"session_duration_seconds": "' \
  # float string values have quotes: "123.45", integer values don't: 5284
  --output json > dropped_sessions_raw.json

# Transform and parse (Python script):
import json, math

with open('dropped_sessions_raw.json') as f:
    events = json.load(f)['events']

bulk_body = []
for event in events:
    doc = json.loads(event['message'])
    # Coerce session_duration_seconds to float
    doc['session_duration_seconds'] = float(doc['session_duration_seconds'])
    bulk_body.append({"index": {"_index": "product-sessions-v2", "_id": doc['session_id']}})
    bulk_body.append(doc)

# Bulk index in batches of 500
for i in range(0, len(bulk_body), 1000):
    batch = bulk_body[i:i+1000]
    es.bulk(body=batch)

print(f"Reindexed {len(events)} dropped documents")
# Output: Reindexed 44,940 dropped documents ✓

# Validation: post-reindex document count
GET /product-sessions-v2/_count
{
  "query": { "range": { "session_start": { "gte": "2026-09-03", "lt": "2026-10-09" } } }
}
// Expected: 507,015 (from v1 reindex) + 44,940 (recovered) = 551,955
// Result:   { "count": 551,847 }  (108 discrepancy within 0.02% — connector restarts)  ✓

# Validate Black Friday cohort:
GET /product-sessions-v2/_search
{
  "size": 0,
  "aggs": {
    "duration_percentile": {
      "percentiles": {
        "field": "session_duration_seconds",
        "percents": [85, 90, 95, 99]
      }
    }
  }
}
// 85th percentile restored from 187s (post-drop) to 247s (pre-drop baseline) ✓

Total engagement time: 4.0 hours (1.5h diagnosis: Elasticsearch document count vs. estimated daily event volume gap analysis confirming ~44,900 dropped documents over 35-day period; Kafka Connect errors.count metric review confirming 44,940 accumulated errors since September 3; product-sessions-v1 _mapping API review confirming session_duration_seconds as type long; Node.js application CloudWatch Logs sample review confirming float string serialization since September 3 for approximately 8% of events; connector configuration review confirming errors.tolerance=all with no DLQ configured; 1.5h fix: upstream Node.js bug fix deployed; product-sessions-v2 index created with explicit mapping; reindex operation from v1 to v2; connector configuration updated to target v2 and add DLQ topic user-session-events-dlq; connector restarted and validated producing to v2 with zero errors; 1.0h data recovery: CloudWatch Logs export of 44,940 dropped session events; Python transformation script to coerce session_duration_seconds to float; bulk reindex into v2; post-reindex document count validation; Black Friday 85th-percentile session_duration_seconds validation confirming 247s vs. pre-drop 247s baseline).

Why Elasticsearch dynamic mapping combined with permissive error tolerance is a recurring retainer data loss pattern

The Northpoint Digital Commerce incident illustrates the compounding effect of two separately-reasonable configuration choices: dynamic mapping (reasonable for rapid development, before the schema is stable) and errors.tolerance=all (reasonable for development environments where occasional schema mismatches are tolerated). In production, these two choices combine to create an environment where any upstream source change that introduces an incompatible field type produces silent, systematic data loss with no observable indicator beyond a gradually widening document count gap.

The specific failure mode — a float string where an integer was expected — is particularly common in Node.js applications because JavaScript’s number type is a 64-bit float, and integer arithmetic operations on session durations (computed as differences between two timestamps in milliseconds, divided by 1000) can produce fractional results if the division is performed before rounding. The serialization layer may or may not apply rounding before converting to JSON. If the Elasticsearch index were created with an explicit float mapping (or with no strict type requirement for the field), all values — integers and fractional values alike — would index correctly. Dynamic mapping’s inference from the first document is not wrong in isolation; it is wrong when upstream data types change later, which happens routinely in evolving systems.

An Elasticsearch developer retainer who performs a quarterly explicit mapping audit — listing all production indices using dynamic mapping, identifying fields with locked types that are commonly subject to source system type changes (duration fields, amount fields, ratio fields), and proposing explicit mapping updates before a type conflict occurs — prevents this class of failure entirely. Pairing the audit with a DLQ configuration requirement for all production Kafka Connect Elasticsearch Sink connectors ensures that when a type conflict does occur despite the audit, the rejected documents are captured and recoverable rather than permanently lost.

How HourTab helps Elasticsearch developer retainers

The Northpoint Digital Commerce engagement totaled 4.0 hours: diagnosis (1.5h — 0.5h Elasticsearch document count vs. estimated daily event volume gap analysis for September 3–October 8 confirming ~44,900 dropped documents; 0.25h Kafka Connect errors.count metric review in Confluent Cloud Control Center confirming exactly 44,940 accumulated errors since September 3 with near-constant rate of ~1,284 errors/day; 0.25h product-sessions-v1 _mapping API review confirming session_duration_seconds type long; 0.25h Node.js application CloudWatch Logs sample review for September 3 onward confirming "session_duration_seconds": "123.45" float string serialization in affected events; 0.25h connector configuration review confirming errors.tolerance=all with no errors.deadletterqueue.topic.name configured), fix (1.5h — 0.25h upstream Node.js bug fix: Math.round() applied before division; deploy to production; 0.5h product-sessions-v2 index creation with explicit mapping; reindex from v1 to v2 via POST /_reindex; alias swap from product-sessions alias to v2; 0.25h connector configuration update: new index target v2, errors.deadletterqueue.topic.name=user-session-events-dlq, errors.log.include.messages=true; connector restart and 1-hour monitoring validation confirming zero errors; 0.5h Kibana monitoring dashboard update: add DLQ topic message rate alert and connector error count alert), and data recovery (1.0h — 0.5h CloudWatch Logs export of 44,940 dropped sessions (float string pattern filter for September 3–October 8); Python transformation script: float coercion + Elasticsearch Bulk API batched reindex; 0.5h post-reindex document count validation; Black Friday cohort reconstruction confirming session_duration_seconds 85th percentile restored to pre-drop 247s baseline).

In a HourTab retainer log, the incident session reads: “product-sessions-v1 silent document drops: Elasticsearch 8.11.3 Elastic Cloud; Kafka Connect Elasticsearch Sink 14.0.12; product-sessions-v1 dynamic mapping; session_duration_seconds type long (inferred March 2026); Node.js bug September 3 2026: float string serialization for ~8% of sessions; mapper_parsing_exception on affected documents; errors.tolerance=all + no DLQ → silently dropped; 1,284 docs/day × 35 days = 44,940 total dropped; Kafka retention 7 days → source messages expired before discovery; discovery October 9 Black Friday high-session cohort analysis; fix: upstream Math.round() bug fix; product-sessions-v2 explicit mapping float; reindex from v1; DLQ configured; 44,940 docs recovered from CloudWatch Logs bulk reindex; 85th-percentile session duration restored; 4.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Elasticsearch developer retainers

See also: Apache Flink developer retainer · Kafka developer retainer · Apache Airflow developer retainer