Blog › ICP guides

JCL developer on retainer: z/OS ENQUEUE contention on shared VSAM dataset DISP=OLD, IBM JCL developer, mainframe JCL developer on monthly retainer

October 10, 2026 · ~15 min read

An IBM JCL developer was maintaining the nightly premium billing batch job stream for a major health insurance carrier running on an IBM z15 mainframe (z/OS 2.5, JES2). The batch stream processed premium billing for 2.4 million policies in a nightly window between midnight and 06:00 AM. The stream consisted of three jobs with a time-ordered dependency: PREMBIL1 (premium billing calculation, JOBID JOB14872, scheduled 01:00 AM) ran first and produced the billing output that PREMRPT1 (premium reporting summary, JOBID JOB14891, scheduled 02:00 AM) consumed to produce management reports, whose output PREMFEED1 (billing clearinghouse feed, JOBID JOB14912, scheduled 04:00 AM) consumed to transmit the nightly billing feed to the company’s external clearinghouse. The PREMBIL1 job contained four steps: STEP010 (sort and validate incoming billing transactions, 8 minutes), STEP020 (process billing master — the main COBOL batch program PREMBIL2, 63 minutes), STEP030 (update master file totals, 12 minutes), and STEP040 (produce billing output sequential files, 6 minutes). Total PREMBIL1 expected runtime: 89 minutes.

PREMBIL1 STEP020 contained the following DD statement for the billing master dataset: //BILMASTIN DD DSN=PREM.BILLING.MASTER.FILE,DISP=OLD. The DISP parameter in z/OS JCL specifies the dataset’s allocation disposition: DISP=OLD tells z/OS that the dataset must already exist and requests exclusive use. When z/OS processes a DISP=OLD dataset allocation at job step initiation (before the program begins executing), it issues an ENQUEUE (ENQ) on the dataset name with EXCLUSIVE scope via the Global Resource Serialization (GRS) component. An EXCLUSIVE ENQ on a dataset name PREM.BILLING.MASTER.FILE prevents any other job or task on the z/OS system (or sysplex) from allocating the same dataset in any disposition — SHR, OLD, or MOD — for the entire duration of the step that holds the ENQ. No concurrent access, even read-only (DISP=SHR), is possible while an EXCLUSIVE ENQ is held. DISP=SHR, in contrast, requests a shared allocation: z/OS issues a SHARE ENQ, which is compatible with other SHARE ENQs on the same dataset (multiple jobs can hold SHR simultaneously), but incompatible with an EXCLUSIVE ENQ (a job requesting SHARE must wait if any job holds EXCLUSIVE, and a job requesting EXCLUSIVE must wait if any job holds SHARE).

On the night of the incident, a batch SLA miss caused PREMBIL1 to start 85 minutes late (02:25 AM instead of 01:00 AM) due to an upstream data feed from the enrollment system arriving 90 minutes behind schedule. PREMBIL1 STEP010 ran from 02:25 to 02:33. PREMBIL1 STEP020 started at 02:47 and immediately issued an EXCLUSIVE ENQ on PREM.BILLING.MASTER.FILE as part of dataset allocation. PREMRPT1 had been submitted for execution at 02:00 AM as scheduled. PREMRPT1 STEP010 ran its first two steps normally (steps that processed sequential datasets not shared with PREMBIL1). When PREMRPT1 STEP010 reached its DD statement //BILMASTIN DD DSN=PREM.BILLING.MASTER.FILE,DISP=SHR at 02:49 AM, z/OS attempted to allocate the dataset with a SHARE ENQ. The allocation was blocked — PREM.BILLING.MASTER.FILE was held exclusively by PREMBIL1 STEP020. z/OS displayed in the PREMRPT1 job log: IEF095I WAITING FOR DATASET PREM.BILLING.MASTER.FILE; HELD BY JOB PREMBIL1 STEP020. PREMRPT1 entered an allocation wait state.

The SDSF diagnostic was unambiguous. The overnight batch operator ran the SDSF DA (Display Active) panel at 03:15 AM when the batch monitoring console showed PREMRPT1 still in a waiting state. SDSF DA showed PREMBIL1 executing STEP020 (26 minutes into the 63-minute step), PREMRPT1 in JOB WAIT status. The operator issued the z/OS system console command D GRS,RES=(SYSDSN,PREM.BILLING.MASTER.FILE) — the Display GRS Resource command — which returned: ISG343I 03.15.22 GRS STATUS 417; SYSDSN PREM.BILLING.MASTER.FILE; OWNER PREMBIL1 ST=EX SYS=SYSB JOBNAME=PREMBIL1 STEPNAME=STEP020 (ENQUEUED); WAITER PREMRPT1 ST=SH SYS=SYSB JOBNAME=PREMRPT1 STEPNAME=STEP010 (WAITING). The output confirmed the ENQ owner (PREMBIL1 STEP020, EXCLUSIVE) and the waiter (PREMRPT1 STEP010, SHARE). PREMBIL2, the COBOL program executing in STEP020, was actively reading the 2.4-million-record PREM.BILLING.MASTER.FILE VSAM KSDS, processing each premium billing record against current rate tables, validating coverage, and writing billing transactions to the PREM.BILLING.OUTPUT.FILE. The EXCLUSIVE ENQ would be held until STEP020’s PREMBIL2 program completed and the step ended, releasing all allocated datasets including PREM.BILLING.MASTER.FILE.

PREMBIL1 STEP020 completed at 03:50 AM (63 minutes as expected). The EXCLUSIVE ENQ on PREM.BILLING.MASTER.FILE was released. PREMRPT1 STEP010 immediately acquired the SHARE ENQ (waited 61 minutes at allocation). PREMRPT1 ran its steps and completed at 05:22 AM — 3 hours and 22 minutes past its 02:00 AM SLA completion target. Meanwhile, PREMFEED1 had started at 04:00 AM as scheduled. PREMFEED1 STEP010 contained //RPTOUT DD DSN=PREM.BILLING.REPORT.OUTPUT.GDG(0),DISP=SHR — it expected PREMRPT1’s output file PREM.BILLING.REPORT.OUTPUT.GDG(0) to be cataloged (available in the ICF catalog with a valid extent on DASD). Since PREMRPT1 had not completed at 04:00 AM, the dataset did not exist in the ICF catalog — PREMRPT1’s STEP030 creates it with DISP=(NEW,CATLG,DELETE) during its execution. PREMFEED1 STEP010 received IEF459I PREM.BILLING.REPORT.OUTPUT.GDG(0) NOT CATALOGED and ABEND A03 — the allocation failure return code indicating a required dataset could not be found or allocated. The clearinghouse feed file was not transmitted by the 04:30 AM contract SLA. The overnight operations team had to manually restart PREMFEED1 at 05:35 AM after PREMRPT1 completed. A03 ABEND frequency: 1–2 per month (each occurrence correlated with a batch SLA miss that delayed PREMBIL1 past 02:47 AM).

IBM JCL, z/OS, and the mainframe batch stream in 2026: the operational backbone of insurance, banking, and government

IBM System z mainframes (IBM z13, z14, z15, z16) running IBM z/OS are the operational backbone of major U.S. and European health insurance carriers, commercial and investment banks, property-casualty insurance companies, federal and state government agencies, and large utilities. IBM z/OS 2.4 entered General Availability September 2020 and end of IBM service September 2024. IBM z/OS 2.5 entered General Availability September 2022 and is the current widely deployed production version. IBM z/OS 3.1 entered General Availability September 2024 and is the current release for organizations doing new z/OS deployments. Most large insurance and banking mainframe sites have migrated from z/OS 2.3 and 2.4 to z/OS 2.5 and are beginning z/OS 3.1 evaluation for 2026–2028 migration windows.

JCL (Job Control Language) is the scripting language for IBM z/OS batch job submission: every batch program execution on z/OS is initiated by a JCL job, and every JCL job specifies the programs to run (via EXEC PGM= or EXEC PROC=), the datasets to allocate (via DD statements with DISP, DSN, DCB, SPACE, UNIT, VOLUME parameters), and the execution conditions (COND or IF/THEN/ELSE for step conditional execution). The JCL developer’s role covers the authoring and maintenance of these job definitions, which are stored as members in Partitioned Data Sets (PDS or PDSE) called JCL libraries or PROCLIB libraries on DASD. In a large insurance carrier’s batch environment, the JCL library may contain thousands of job members representing the nightly, weekly, monthly, and quarterly batch cycles. A JCL developer on monthly retainer for an insurance carrier’s premium billing, claims payment, or actuarial batch stream covers: maintaining and optimizing the JCL job members, diagnosing ABEND codes (S0C7 data exception, B37 extent overflow, A03 allocation failure, S878 insufficient virtual storage, S322 CPU time limit exceeded), diagnosing ENQ contention between concurrent batch jobs, maintaining GDG (Generation Data Group) base entries and generations, DFSORT step tuning (sort record selection, output splitting, join operations), and coordinating with the COBOL and PL/I application developers who write the programs that the JCL executes.

The DISP parameter is the most commonly misspecified JCL parameter in JCL that has been maintained by multiple developers over decades, because its correct specification depends on knowing how the executing program uses the dataset — specifically, whether the program opens the file for INPUT (read-only), OUTPUT (write, create new), I-O (read and write update), or EXTEND (append). DISP=OLD is correct when the program opens the dataset for I-O (update in place) or when the step must guarantee exclusive access for correctness reasons. DISP=SHR is correct when the program opens the dataset for INPUT only. A JCL change that modifies the program to use INPUT (read-only) without updating the corresponding DD statement from DISP=OLD to DISP=SHR leaves the JCL over-specifying the disposition: the EXCLUSIVE ENQ is held unnecessarily, blocking concurrent jobs from accessing the same dataset in SHARE mode. In a batch environment where jobs are expected to run concurrently (different jobs reading the same master file simultaneously), an unnecessary EXCLUSIVE ENQ serializes the batch window and causes downstream SLA misses.

z/OS DISP parameter, GRS ENQUEUE/DEQUEUE, and SDSF diagnostic commands

The z/OS DISP parameter controls three aspects of dataset allocation: the status (OLD = dataset must exist; NEW = create dataset; MOD = extend existing or create if not found; SHR = dataset must exist, share with others); the normal-end disposition (KEEP, CATLG, UNCATLG, DELETE, PASS); and the abnormal-end disposition (KEEP, CATLG, UNCATLG, DELETE). For ENQ scope, only the status portion matters: DISP=OLD issues ENQ EXCLUSIVE; DISP=SHR issues ENQ SHARE; DISP=(NEW,CATLG,DELETE) issues ENQ EXCLUSIVE (new datasets are allocated exclusively). The ENQ is issued at dataset allocation time — before the program begins executing for most allocations, or at dynamic allocation time during program execution (via DYNALLOC) for dynamically allocated datasets. The ENQ is held for the duration of the job step (from step initiation to step termination) for statically allocated (JCL-specified) datasets.

The D GRS console command (Display GRS Resource) is the primary diagnostic for z/OS ENQ contention. D GRS,RES=(SYSDSN,dataset.name) displays all current ENQ holders and waiters for a specific dataset name in the SYSDSN major resource. The output identifies the holding job and step (OWNER), the ENQ type (EX for exclusive, SH for share), the system in the sysplex (SYS=), and all waiting jobs (WAITERs) with their ENQ type requests. D GRS,ANALYZE shows all ENQ contention situations across the system. In SDSF, the ENQ panel (SDSF ENQ) provides a full-screen view of all GRS enqueue holders and waiters, filterable by resource name or major name. The SDSF LOG panel shows the z/OS system log messages including IEF095I WAITING FOR DATASET messages generated when a job step cannot complete dataset allocation due to ENQ conflict.

The root cause analysis for the PREMBIL1/PREMRPT1 ENQ contention required reviewing the COBOL source for the PREMBIL2 program executing in STEP020. The JCL developer retrieved the COBOL source from the source library (COBL.SOURCE.LIBRARY member PREMBIL2) and located the FILE-CONTROL section and PROCEDURE DIVISION file handling. The relevant COBOL statements were: FD BILMASTIN RECORD CONTAINS 800 CHARACTERS BLOCK CONTAINS 0 RECORDS RECORDING MODE IS F. and OPEN INPUT BILMASTIN. — OPEN INPUT, not OPEN I-O. The PREMBIL2 program only reads PREM.BILLING.MASTER.FILE; it does not write to it. The EXCLUSIVE ENQ from DISP=OLD was unnecessary for correctness: PREMBIL2 would function identically with DISP=SHR, because multiple jobs reading the same VSAM KSDS in INPUT mode simultaneously is fully supported by z/OS and VSAM. The DISP=OLD was a legacy specification from an earlier version of the PREMBIL2 program (three years ago) that performed both read and write operations on BILMASTIN in the same step; a refactoring separated the read and update into STEP020 (read-only) and STEP030 (write), but the COBOL programmer who made the refactoring documented the change in the COBOL source code (a comment at the top of the file: “STEP020 refactored 2023-04 to read-only; update moved to STEP030”) but did not update the JCL. The change was never tested in the concurrent scenario that would expose the ENQ contention, because the test environment ran PREMBIL1 and PREMRPT1 sequentially (PREMRPT1 was manually submitted after PREMBIL1 completed), not in the concurrent overnight schedule configuration.

The fix was a one-line JCL change: in the PREMBIL1 JCLLIB member (JCLLIB.PRDLIB member PREMBIL1), STEP020’s BILMASTIN DD statement was changed from //BILMASTIN DD DSN=PREM.BILLING.MASTER.FILE,DISP=OLD to //BILMASTIN DD DSN=PREM.BILLING.MASTER.FILE,DISP=SHR. The change was documented in the organization’s ServiceNow change management system (change record CHG0087341: PREMBIL1 STEP020 BILMASTIN DISP corrected from OLD to SHR after COBOL source review confirming OPEN INPUT only; root cause: JCL not updated during March 2023 PREMBIL2 refactoring; risk: low — changing DISP from OLD to SHR for a read-only OPEN INPUT VSAM KSDS removes unnecessary EXCLUSIVE ENQ and allows concurrent DISP=SHR allocation). The change was tested in the pre-production environment by submitting PREMBIL1 and PREMRPT1 concurrently and confirming that both jobs allocated PREM.BILLING.MASTER.FILE successfully (PREMRPT1 no longer shows IEF095I WAITING FOR DATASET in the LOG). After deployment: PREMBIL1 and PREMRPT1 run concurrently; PREMRPT1 completes by 03:45 AM; PREMFEED1 allocates its input file successfully at 04:00 AM; the clearinghouse feed meets its 04:30 AM SLA. A03 ABEND rate: 1–2 per month to 0.

Typical IBM JCL developer retainer work and what it looks like in a work log

DISP=OLD EXCLUSIVE ENQ on a read-only DD statement after JCL/program refactoring is the canonical invisible z/OS batch production contention bug for any mainframe batch stream maintained by multiple developers over years. It appears in the work logs of JCL retainer engagements for health insurance carrier nightly premium billing cycles (billing master dataset read by both the billing calculation job and the premium reporting job), commercial bank nightly posting cycles (account master dataset read by both the DDA posting job and the account statement generation job), claims payment processing (claims database export dataset read by both the check print job and the remittance advice reporting job), and government agency benefit payment cycles (benefit master dataset read by both the payment generation job and the audit trail reporting job). The work log entry: “IBM z/OS 2.5; JES2; PREMBIL1 (JOBID JOB14872) STEP020 DD BILMASTIN DSN=PREM.BILLING.MASTER.FILE DISP=OLD — ENQ EXCLUSIVE; PREMBIL2 COBOL OPEN INPUT BILMASTIN (STEP020 refactored read-only 2023-04; DISP=OLD not updated); PREMRPT1 (JOBID JOB14891) STEP010 BILMASTIN DISP=SHR blocked at allocation; IEF095I WAITING FOR DATASET PREM.BILLING.MASTER.FILE HELD BY JOB PREMBIL1 STEP020 in PREMRPT1 LOG; D GRS RES=(SYSDSN,PREM.BILLING.MASTER.FILE): OWNER PREMBIL1 STEP020 EX WAITER PREMRPT1 STEP010 SH; PREMBIL1 STEP020 runtime 63 min; PREMRPT1 blocked 61 min; PREMRPT1 SLA miss: 05:22 AM actual vs. 02:00 AM target; PREMFEED1 ABEND A03 at 04:00 AM: PREM.BILLING.REPORT.OUTPUT.GDG(0) not yet cataloged; manual PREMFEED1 restart 05:35 AM; A03 rate 1–2/month; fix: PREMBIL1 STEP020 BILMASTIN DISP=OLD changed to DISP=SHR; change CHG0087341; PREMRPT1 SLA met; A03 rate: 0; hours: 3h.”

ABEND A03 downstream allocation failure from SLA-missed upstream job is the most common JCL retainer ABEND diagnosis when a downstream job’s input dataset does not exist because the upstream job that produces it has not yet completed. The A03 ABEND (IEABENDx code, issued by the z/OS Allocation and Recovery routine) means a required dataset could not be found or allocated: either it is not cataloged (NEW/CATLG disposition from the creating job has not executed yet), not present on the specified volume, the extent cannot be found, or the catalog entry is corrupt. For time-based scheduling (where PREMFEED1 is submitted at 04:00 AM based on the expected completion of PREMRPT1 at 02:00 AM), any SLA miss that delays PREMRPT1 past 04:00 AM will cause PREMFEED1 to ABEND A03 because its input GDG generation does not yet exist. The permanent fix is workload automation dependency (IBM Workload Automation or CA7: PREMFEED1 is not submitted until PREMRPT1 has completed successfully with RC=0), rather than time-based scheduling. Work log: “z/OS 2.5; PREMFEED1 ABEND A03 STEP010 DSN=PREM.BILLING.REPORT.OUTPUT.GDG(0) DISP=SHR — IEF459I NOT CATALOGED; root cause: PREMRPT1 not completed at PREMFEED1 04:00 AM scheduled start; PREMRPT1 delayed to 05:22 AM by ENQ contention with PREMBIL1 STEP020; immediate fix: PREMBIL1 STEP020 DISP=OLD changed to DISP=SHR (see CHG0087341); structural fix: IBM Workload Automation (TWS) dependency added PREMFEED1 predecessor = PREMRPT1 RC=0; PREMFEED1 now held in TWS WAITING PREDECESSOR state until PREMRPT1 completes and posts RC=0; A03 ABEND rate: 1–2/month → 0; hours: 2h TWS dependency configuration.”

DFSORT step optimization for large VSAM KSDS batch preprocessing is the most common planned JCL retainer work when the batch window is running over schedule due to inefficient sort steps rather than ENQ contention. A DFSORT step reading a 2.4-million-record VSAM KSDS with a complex INCLUDE statement and writing to a sequential output file can be reduced from 18 minutes to 4 minutes by adding a BUFND specification in the DD statement (VSAM buffer count for the input KSDS), specifying SORT FIELDS=(1,10,CH,A) with SKIPREC and STOPAFT to limit the sort record scope, using OUTFIL with INCLUDE to filter in DFSORT rather than filtering in the COBOL program, and adjusting SORTWORK DD space allocation to eliminate extent overflow. Work log: “z/OS 2.5; PREMBIL1 STEP005 DFSORT step preprocessing PREM.BILLING.MASTER.FILE (2.4M records, 800-byte records, VSAM KSDS); original sort time: 22 minutes; SDSF STEP005 elapsed time from SMF Type 30 records; tuning: BILMASTIN DD added BUFND=20 (20 VSAM I/O buffers vs. default 3); SORT statement changed from FIELDS=(1,800,CH,A) to FIELDS=(1,10,CH,A) (key only, not full record); INCLUDE COND=(801,3,CH,EQ,C'ACT') added to filter inactive records in sort; SORTWORK01-03 SPACE=(CYL,(200,50),RLSE) specified explicitly to eliminate secondary extent overflow; sort time after tuning: 5 min 20 sec (vs. 22 min original); hours: 2h.”

How HourTab tracks IBM JCL developer retainer hours

z/OS DISP=OLD EXCLUSIVE ENQ on a read-only dataset OPEN INPUT bugs are invisible in sequential batch testing by the same mechanism that makes all concurrency bugs invisible when only one job runs at a time. The developer who wrote the PREMBIL1 JCL originally specified DISP=OLD because PREMBIL2 at the time performed both reads and in-place updates on PREM.BILLING.MASTER.FILE (requiring exclusive access for correctness). The test environment ran PREMBIL1 alone, verified correct output, and never tested the concurrent scenario with PREMRPT1 running simultaneously. Three years later, the COBOL programmer who refactored PREMBIL2 to read-only (separating the update to STEP030) tested the refactored PREMBIL2 in isolation: ran PREMBIL1 in the test environment, confirmed STEP020 completed successfully, confirmed the read-only COBOL logic was correct. The test did not run PREMRPT1 concurrently. The DISP=OLD had no visible effect in isolated testing because there were no competing ENQ requests. The ENQ contention only becomes visible in the production overnight schedule when PREMBIL1 and PREMRPT1 overlap.

The work log entry that makes z/OS JCL ENQ contention work auditable must name every element of the causal chain: the z/OS version (2.5), the JES2 environment, the holding job and step (PREMBIL1 STEP020), the dataset name (PREM.BILLING.MASTER.FILE), the ENQ type (EXCLUSIVE from DISP=OLD), the COBOL program name (PREMBIL2), the COBOL OPEN verb (OPEN INPUT, not OPEN I-O — the critical evidence for why DISP=SHR is correct), the blocked job and step (PREMRPT1 STEP010, DISP=SHR), the D GRS output confirming OWNER and WAITER, the SLA miss duration (PREMRPT1 completion at 05:22 AM vs. 02:00 AM target), the downstream ABEND (PREMFEED1 ABEND A03 at 04:00 AM), and the A03 frequency before and after the fix. A log that says “fixed JCL contention issue, 3h” is not auditable to a health insurance carrier’s batch operations manager who is trying to explain to the clearinghouse why the premium billing feed arrived 96 minutes past SLA. A log that names the DISP=OLD DD statement, the D GRS output, the COBOL OPEN INPUT evidence, the one-line JCL change, and the zero-ABEND outcome after the fix is auditable and demonstrates the value of a long-term JCL retainer relationship for an insurance carrier whose entire nightly batch cycle depends on JCL job stream maintained by a mainframe specialist on monthly retainer. HourTab gives IBM JCL developers a public retainer-hours URL they send to clients — health insurance carriers, banks, claims payment processors, government agencies, and utilities that run critical nightly batch cycles on IBM z/OS mainframes maintained by a JCL specialist on monthly retainer.

The broader context for IBM JCL retainer billing is that the DISP=OLD EXCLUSIVE ENQ held on a read-only dataset is the z/OS instantiation of the universal principle that any system allowing long-lived exclusive locks demands disposition (and lock type) accuracy at the code level. Oracle PL/SQL developer retainers cover the equivalent in Oracle Database: a stored procedure’s SELECT ... FOR UPDATE NOWAIT at line 14 holds an Oracle TX LMODE=6 exclusive lock for 47 minutes when the procedure returns early without ROLLBACK — different database platform, same class of exclusive lock held longer than necessary. Ingres developer retainers cover the equivalent in Ingres IPL stored procedures: a database procedure’s UPDATE acquires an Ingres exclusive row lock held for the calling connection’s transaction lifetime when the procedure RETURN exits without ROLLBACK, serializing concurrent benefits processors behind the holding session. Microsoft Access developer retainers cover the equivalent on the desktop database platform: the ACE engine’s Edited Record pessimistic locking holds a record-level lock for the Access session lifetime when the estimator minimizes the form without saving — different platform, same class of exclusive lock held indefinitely without explicit release.

FAQ: IBM JCL developer retainers