TIBCO BusinessWorks developer on retainer: Invoke_WMS_GetInventory HTTP Request activity no timeout — all 8 worker threads consumed by stalled WMS calls — 4,847 orders queued — TIBCO BW developer on monthly retainer

TIBCO ActiveMatrix BusinessWorks 6.7.2 • TIBCO EMS • HTTP Request activity • worker thread pool exhaustion • substvar timeout • fault handler DLQ routing • TIBCO Hawk • retainer billing

At Coastal Shipping Partners LLC, the order synchronization pipeline had been running cleanly for months when it silently stopped processing incoming customer orders on the morning of September 12, 2026. The TIBCO ActiveMatrix BusinessWorks engine showed no errors. No alerts fired. The EMS queue that delivered orders to the BW process was receiving messages normally — 4,847 of them over four hours — but not a single order was completing. The operations team noticed only because an order management portal showed a growing backlog of unconfirmed shipments. They paged the TIBCO BusinessWorks developer retainer at 14:04. The root cause was an HTTP Request activity in the BW process with no timeout property set, combined with a partial network interface card failure on the warehouse management system API server. Every BW worker thread in the entire two-engine deployment was occupied, indefinitely, waiting for an HTTP response that would never arrive.

Coastal Shipping Partners ran TIBCO ActiveMatrix BusinessWorks 6.7.2 on RHEL 8.3. Two BW engine containers — bw-engine-01 and bw-engine-02 — ran the CSP_OrderSync application (version 2.3.1). Each engine was configured with 4 worker threads (tibco.bw.engine.threadcount=4), giving 8 total worker threads across both engines. The application deployed a single BW process: OrderSyncHTTP.process, which read incoming customer orders from TIBCO EMS queue CSP.ORDERS.INBOUND on EMS server ems-prod-01, invoked the warehouse management system REST API via an HTTP Request activity, and wrote order confirmations to CSP.ORDERS.CONFIRMED.

The HTTP Request activity within the process was named Invoke_WMS_GetInventory. Its target was http://wms-internal.csp.local:8080/api/inventory/check — a POST call with a JSON payload sent to the internal warehouse management system. The activity’s Timeout property was not set. In TIBCO BW 6.x, an HTTP Request activity with no Timeout property configured waits indefinitely for the server to respond. There is no platform-level default timeout. The BW HTTP Client Shared Resource used by the activity also had no connect or read timeout configured. This was not an oversight that was ever visible in normal operation: under normal conditions the WMS API responded in under 200 milliseconds, so the absence of a timeout had never caused any observable problem.

At 09:47 AM on September 12, a network interface card on wms-app-01 (10.12.0.45, the WMS application server) began failing. The NIC started dropping approximately 15% of TCP packets. The failure mode was partial and asymmetric: ICMP traffic was unaffected, and TCP connections from the BW engines to the WMS API server established successfully — the three-way handshake completed. But the WMS server’s response packets were dropped at the degraded NIC before they left the host. HTTP requests from the BW engines arrived at the WMS server and were processed, but the HTTP response never reached the BW engine. From the BW process instance’s perspective, the HTTP Request activity was waiting for a response that the network had silently discarded.

How TIBCO BusinessWorks HTTP Request thread blocking works

TIBCO ActiveMatrix BusinessWorks 6.x executes process instances on a worker thread pool. The thread pool size is controlled by the engine property tibco.bw.engine.threadcount, set in the engine configuration or in the BW application’s engine properties. Each active process instance holds exactly one worker thread for the duration of any blocking activity — including an HTTP Request activity waiting for a response.

-- BW Engine configuration: both bw-engine-01 and bw-engine-02
-- tibco.bw.engine properties (engine.tra or BW application substvar):

tibco.bw.engine.threadcount=4
# Total worker threads per engine: 4
# Total across both engines: 8

-- CSP_OrderSync.substvar (before fix — WMS timeout property absent):
# WMS.HttpTimeout   (not set — property does not exist in substvar file)

-- Invoke_WMS_GetInventory HTTP Request activity configuration (before fix):
# Target URL:   http://wms-internal.csp.local:8080/api/inventory/check
# Method:       POST
# Content-Type: application/json
# Timeout:      (not set — field left blank)
# BW 6.x behavior: no timeout = indefinite wait (no Java SocketTimeout thrown)

-- BW HTTP Client Shared Resource (before fix):
# Connection Timeout:  (not set)
# Read Timeout:        (not set)
# Both fields blank = indefinite wait at Java HttpURLConnection layer

When the WMS NIC began failing, incoming BW process instances entered the Invoke_WMS_GetInventory activity and blocked. Each blocked instance held its worker thread. The EMS server continued to deliver incoming order messages to the BW engines: as each new message arrived, it was assigned to a new process instance, which ran until it reached Invoke_WMS_GetInventory and blocked in turn. Over the first 36 minutes (09:47 to 10:23), all 8 worker threads — 4 per engine — were consumed by blocked instances. After 10:23, no new process instances could execute. Incoming EMS messages were acknowledged by the BW JMS receiver but queued internally pending a free thread. The EMS queue depth — which reflects messages not yet handed to a BW process — climbed steadily from 0 to 4,847 over the 4-hour outage.

No entry appeared in the BW Engine logs indicating a problem. The 8 blocked process instances were in a valid state: active, not faulted. The BW Engine log records faults and exceptions; an HTTP Request activity waiting for a response is not a fault. The only indication of the problem in the BW layer was the growing internal queue of unstarted process instances, visible in the TIBCO Hawk agent metrics but not surfaced to any alert because EMS queue-depth alerting had not been configured.

Diagnosis: BW Engine admin console and TIBCO Hawk monitoring

The TIBCO BW developer retainer connected to the BW Engine admin console at 14:09. The admin console’s Process Instances view showed the state immediately:

-- BW Engine Admin Console → Process Instances → Active (bw-engine-01):
Process Name              State                      Activity                     Start Time
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      09:47:03
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      09:52:18
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      09:59:44
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      10:08:31
-- (4 instances total on bw-engine-01 — all 4 threads occupied)

-- BW Engine Admin Console → Process Instances → Active (bw-engine-02):
Process Name              State                      Activity                     Start Time
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      09:51:07
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      09:55:22
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      10:14:09
OrderSyncHTTP.process     Active                     Invoke_WMS_GetInventory      10:23:45
-- (4 instances total on bw-engine-02 — all 4 threads occupied)

-- Summary: 16 active process instances across both engines.
-- All 16 in state Invoke_WMS_GetInventory.
-- All started between 09:47 and 10:23. Zero completed since 09:47.

The TIBCO Hawk console confirmed the thread pool saturation. Hawk agent bw-hawk-01 exposed the BW Engine microagent metrics:

-- TIBCO Hawk Console: bw-hawk-01 microagent
-- com.tibco.plugin.threadpool.ExecutionManager (per engine, both identical):

ExecutionManager.activeThreads   = 8    (max 8 — 4 per engine × 2 engines)
ExecutionManager.maxThreads      = 8
ExecutionManager.queueDepth      = 4847 (EMS messages acknowledged, waiting for free thread)
ExecutionManager.completedJobs   = 0    (since 10:23 AM — no completions in 3+ hours)

-- BW Engine Hawk microagent: com.tibco.bw.core.runtime.engine
-- (bw-engine-01)
ProcessInstanceCount.active      = 8
ProcessInstanceCount.completed   = 2,314   (pre-outage completions from earlier this morning)
ProcessInstanceCount.failed      = 0       (no faults — blocked ≠ failed in BW)

The developer coordinated with the network team to diagnose the WMS connectivity problem. ICMP ping to wms-internal.csp.local from bw-engine-01 succeeded, confirming the host was reachable. TCP connect to port 8080 also succeeded, confirming the WMS application was listening. But an HTTP request with a 5-second timeout hung until the timeout was reached:

-- Network diagnostics from bw-engine-01 (14:15):

$ ping wms-internal.csp.local
PING wms-internal.csp.local (10.12.0.45): 56 data bytes
64 bytes from 10.12.0.45: icmp_seq=0 ttl=64 time=0.412 ms
64 bytes from 10.12.0.45: icmp_seq=1 ttl=64 time=0.389 ms
-- ICMP succeeds (NIC drops TCP but not ICMP)

$ telnet wms-internal.csp.local 8080
Trying 10.12.0.45...
Connected to wms-internal.csp.local.
-- TCP connect succeeds (TCP handshake completes before NIC enters drop state)

$ curl -v --max-time 5 http://wms-internal.csp.local:8080/api/inventory/check \
    -H "Content-Type: application/json" -d '{}'
* Connected to wms-internal.csp.local (10.12.0.45) port 8080
* upload completely sent off: 2 out of 2 bytes
-- [hangs for 5 seconds — no HTTP response received]
curl: (28) Operation timed out after 5000 milliseconds with 0 bytes received
-- WMS server receives request but response packets dropped at NIC

-- Network team (14:20): confirmed NIC degradation on wms-app-01 (10.12.0.45).
-- ~15% TCP packet drop rate at egress NIC.
-- NIC replacement ordered. ETA: ~1 hour.

The BW application property review confirmed the timeout omission. Reviewing the CSP_OrderSync.substvar file showed no WMS.HttpTimeout property. Inspecting the Invoke_WMS_GetInventory activity in the BW application confirmed the Timeout field was blank. The BW HTTP Client Shared Resource showed no connect or read timeout values. The diagnosis was complete at 14:20: WMS NIC partial failure, BW HTTP Request with no timeout, all 8 threads blocked indefinitely.

The fix: HTTP Request timeout and fault handler with dead-letter queue routing

The fix had two parts: set a timeout on the HTTP Request activity to prevent indefinite thread blocking, and add a fault handler to route failed orders to a dead-letter queue rather than silently losing them when the WMS API was unavailable.

Part 1: HTTP timeout via application substvar. In TIBCO BW 6.x, the recommended pattern for configurable timeouts is to define the timeout value as a module property in the application’s substvar file and bind the HTTP Request activity’s Timeout property to the module property. This allows the timeout to be changed per environment without redeploying the BW application EAR:

-- CSP_OrderSync.substvar (after fix — WMS timeout added):
WMS.HttpTimeout = 30000
# 30,000 milliseconds = 30 seconds
# Chosen: WMS API normal response <200ms; 30s provides margin for legitimate
# slow responses while ensuring threads are freed within ~30s if WMS is unresponsive.

-- Invoke_WMS_GetInventory HTTP Request activity (after fix):
# Timeout: %WMS.HttpTimeout%
# BW 6.x module property substitution syntax: %PropertyName%
# At runtime: activity timeout = 30,000ms (30 seconds)
# If no HTTP response within 30s: throws com.tibco.bw.palette.restjson.runtime.RESTException

-- BW HTTP Client Shared Resource (after fix):
# Connection Timeout: 10000  (10 seconds — fail fast if TCP connect cannot establish)
# Read Timeout:       (left blank — controlled at activity level via %WMS.HttpTimeout%)
# Note: activity-level Timeout takes precedence over Shared Resource read timeout in BW 6.x

Part 2: Fault handler with dead-letter queue routing. Setting a timeout causes the HTTP Request activity to throw a RESTException after 30 seconds. Without a fault handler, BW would propagate the exception up to the process fault handler (if one exists) or terminate the process instance with a fault. The unprocessed order message would be lost — not retried, not recoverable. The fix added an Error Boundary Event on the Invoke_WMS_GetInventory activity to catch com.tibco.bw.palette.restjson.runtime.RESTException and route the failed order to a dead-letter queue:

-- OrderSyncHTTP.process fault handler design (after fix):

Main flow:
  [JMS Receive: ReadOrder_INBOUND]           -- reads from CSP.ORDERS.INBOUND
          |
  [Mapper: MapOrderToWMSRequest]             -- map order fields to WMS JSON payload
          |
  [HTTP Request: Invoke_WMS_GetInventory]    -- POST to wms-internal.csp.local:8080
          |  \
          |   [Error Boundary Event]         -- catches RESTException (timeout + HTTP errors)
          |           |
          |   [Call Process: ErrorHandler]   -- subprocess invoked on timeout/error
          |
  [Mapper: MapWMSResponseToConfirmation]
          |
  [JMS Send: WriteOrder_CONFIRMED]           -- writes to CSP.ORDERS.CONFIRMED

-- ErrorHandler subprocess:
  [JMS Send: WriteOrder_DLQ]
  # Destination: CSP.ORDERS.DLQ
  # Message: original order payload (passed as subprocess parameter)
  # Delivery Mode: PERSISTENT
          |
  [Hawk Alert: bw.order.wms.failure]
  # Alert name: bw.order.wms.failure
  # Severity: WARN
  # Message: "WMS inventory check failed for order {orderId}: {errorMessage}"
  # Allows Hawk rules to notify on-call or trigger remediation workflow

-- Error catch: com.tibco.bw.palette.restjson.runtime.RESTException
# Covers: connection timeout, read timeout, HTTP 4xx/5xx, connection refused
# BW 6.x: RESTException is the top-level exception for all HTTP/REST palette errors

The updated application was packaged as CSP_OrderSync 2.3.2 and deployed to both BW engines. To clear the 16 stalled process instances, both BW engines were restarted with a graceful shutdown: the BW Engine graceful shutdown signals all active process instances to abort at their current activity, frees the worker threads, and shuts down the engine. The restart was issued at 14:52. Both engines came back online within 90 seconds. At 14:54, the engines began processing the 4,847 queued orders in the EMS queue. All 4,847 orders were processed in 23 minutes. The WMS NIC was replaced by the network team at 15:14; WMS API response times returned to under 200 milliseconds.

A QA test was performed before the production deployment: a WireMock stub was configured to simulate WMS timeout by accepting the HTTP connection but never sending a response. The updated BW process, with WMS.HttpTimeout=30000, threw a RESTException after 30 seconds, the Error Boundary Event caught the exception, the ErrorHandler subprocess ran, and the failed order was routed to CSP.ORDERS.DLQ. The Hawk alert bw.order.wms.failure fired. Worker threads were freed after 30 seconds. The test confirmed the fix behaved correctly before any production order was processed.

Why HTTP timeout exhaustion is a recurring TIBCO BW retainer problem

The TIBCO BusinessWorks 6.x HTTP Request activity is designed to be stateless by default: it makes an HTTP call and waits for the response. The platform does not impose a timeout because the appropriate timeout varies by use case — a WMS inventory check may warrant 10 seconds, a batch data extract from a reporting database may warrant 120 seconds. Leaving the Timeout property blank is not a configuration warning in BW Studio; it is simply the absence of a value in the activity properties panel. In a development or QA environment where the target API is responsive, the absent timeout is invisible — the activity completes in milliseconds regardless. The consequence of the missing timeout only materializes when the target API experiences a partial failure mode where TCP connections are established but HTTP responses are never delivered. That failure mode is rare enough that most BW deployments go months or years before encountering it, which means the missing timeout is often discovered only during an incident.

The compounding factor is the fixed thread pool architecture. BW engines are configured with a specific tibco.bw.engine.threadcount value. That value is chosen for normal operation: it should be large enough to handle peak message concurrency without excessive context-switching overhead. It is not designed to absorb indefinitely blocking threads. An HTTP timeout that blocks 1 thread is a minor resource issue; an HTTP timeout that blocks all N threads is a complete processing outage, with no difference in the BW log between the two states other than the thread count metric.

A TIBCO BW developer retainer who performs a quarterly timeout audit — reviewing all HTTP Request activities, all JDBC Query and JDBC Update activities, and all external call activities in every deployed BW application to confirm that timeout properties are set — catches this class of problem before it causes a production thread pool exhaustion. The audit also confirms that fault handlers exist for all activities that can throw checked exceptions (HTTP Request, JDBC, JMS Send) and that dead-letter queue destinations exist for each message type. For BW deployments with more than a few HTTP Request activities, the audit takes 2–4 hours per application and is a standard quarterly retainer deliverable.

How HourTab helps TIBCO BusinessWorks developer retainers

The Coastal Shipping Partners engagement totaled 4.0 hours: diagnosis (1.0h — 0.25h connecting to BW Engine admin console on both engines and reviewing the 16 active process instances all blocked in Invoke_WMS_GetInventory since 09:47–10:23, confirming zero completions in 3+ hours; 0.25h reviewing TIBCO Hawk console for activeThreads=8/maxThreads=8 and queueDepth=4847 confirming complete thread pool saturation; 0.25h coordinating with the network team to run ICMP, telnet, and curl diagnostics from bw-engine-01 against wms-internal.csp.local:8080, confirming TCP connect succeeds but HTTP response never arrives, WMS NIC partial failure confirmed at 14:20; 0.25h reviewing CSP_OrderSync.substvar for missing WMS.HttpTimeout property and confirming blank Timeout field in Invoke_WMS_GetInventory activity and no timeout on BW HTTP Client Shared Resource), fix (2.0h — 0.5h adding WMS.HttpTimeout=30000 to substvar and binding the HTTP Request activity Timeout property to %WMS.HttpTimeout%; 1.0h developing the Error Boundary Event fault handler on Invoke_WMS_GetInventory, implementing the ErrorHandler subprocess with JMS Send to CSP.ORDERS.DLQ (PERSISTENT delivery mode, original order payload preserved) and Hawk alert bw.order.wms.failure; 0.5h packaging CSP_OrderSync 2.3.2 EAR and deploying to both BW engines, issuing graceful shutdown to drain the 16 stalled instances at 14:52, confirming engine restart at 14:54 and queue drain start), QA (0.5h — WireMock stub configuration simulating WMS no-response; BW process test confirming 30s timeout fires, RESTException caught by Error Boundary Event, failed order payload written to CSP.ORDERS.DLQ, Hawk alert bw.order.wms.failure fires, worker thread freed after 30s), and documentation (0.5h — post-incident write-up, substvar timeout audit SOP added to retainer runbook, quarterly HTTP/JDBC timeout audit added to retainer calendar for all CSP_OrderSync applications).

In a HourTab retainer log, the incident session reads: “TIBCO ActiveMatrix BW 6.7.2; CSP_OrderSync v2.3.1; OrderSyncHTTP.process; Invoke_WMS_GetInventory HTTP Request activity; no timeout property → indefinite thread block; WMS NIC partial failure Sep 12 09:47 — TCP connect succeeds, HTTP response never delivered (~15% TCP egress packet drop on wms-app-01 NIC); 8 worker threads consumed (4 per engine × 2 engines) between 09:47 and 10:23; EMS queue CSP.ORDERS.INBOUND depth: 0 → 4,847 over 4h; BW Engine admin console: 16 active instances all blocked in Invoke_WMS_GetInventory, zero completed; Hawk: activeThreads=8/maxThreads=8, queueDepth=4847; root cause: BW HTTP Request no-timeout default + WMS NIC partial failure (rare TCP connect success + no response); fix: substvar WMS.HttpTimeout=30000ms + HTTP Request Timeout bound to %WMS.HttpTimeout% + Error Boundary Event fault handler → ErrorHandler subprocess → CSP.ORDERS.DLQ (PERSISTENT) + Hawk alert bw.order.wms.failure; CSP_OrderSync 2.3.2 deployed both engines; BW engine graceful shutdown 14:52, restart 14:54; 4,847 orders processed in 23min; WMS NIC replaced 15:14; QA: WireMock stub WMS no-response → 30s timeout confirmed, DLQ routing confirmed, Hawk alert confirmed; 4.0h.” The CSP operations director sees the exact failure mode (NIC partial failure + no-timeout default), the complete scope of the outage (8 threads, 4,847 orders, 4 hours), the two-part fix (timeout configuration + fault handler DLQ routing), and the QA evidence — with no ambiguity about where each retainer hour was spent.

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: TIBCO BusinessWorks developer retainers

See also: MuleSoft developer retainer · DataStage developer retainer · Informatica developer retainer