Blog › ICP guides

COBOL developer on retainer: z/OS, VSAM, CICS, DB2, and mainframe batch on monthly retainer

September 20, 2026 · ~23 min read

A regional bank had a core demand deposit accounting system written in COBOL on z/OS that had been in production for twenty-two years. The batch processing window for overnight statement runs had grown from four hours to eight hours as account volumes doubled, and the CICS-based teller transaction system was dropping screens for agents during peak morning hours when the regional branches opened simultaneously. They hired a COBOL developer on retainer to diagnose the batch window expansion and stabilize the CICS transaction system. The batch problem was traced to VSAM KSDS free space exhaustion: the account master file had been originally defined with FREESPACE(5 2) — five percent free space per control interval and two percent per control area — and three years of account record growth had driven the record format from 240 bytes to 310 bytes due to added address fields, exhausting the reserved CI space and causing cascade control-interval and control-area splits on every batch update run. The LISTCAT output showed SPLITS-CI counts increasing by more than 800 per batch window.

The CICS problem was architectural: the teller screen map flow had been written in conversational mode in 1998 and never redesigned. Each active teller session held a CICS TCB from the moment the account inquiry screen was sent until the transaction was committed — including the 30 to 90 seconds the teller spent typing account information and verifying the customer. With 180 teller stations opening simultaneously at 9:00 AM, the CICS region's 200-TCB high-water-mark was reached within four minutes of branch opening, and subsequent screen transactions waited for a TCB to free. Neither problem produced a visible code defect. The VSAM issue required reading LISTCAT output and understanding FREESPACE interaction with record size growth over a three-year period. The CICS issue required recognizing a 1998 conversational design pattern in a system where the original architects were no longer available. The retainer work that resolved both problems took 44 hours and produced one VSAM cluster redefinition and one pseudoconversational redesign of the teller map flow. A COBOL developer on monthly retainer does this category of work continuously: diagnosing VSAM space management issues before they expand the batch window, redesigning CICS transaction flows before TCB exhaustion affects teller throughput, and tuning DB2 cursor isolation before lock contention degrades overnight reporting.

COBOL language, data division design, and the PROCEDURE DIVISION

COBOL programs are organized into four divisions that must appear in order: IDENTIFICATION (program metadata — PROGRAM-ID, AUTHOR, DATE-WRITTEN), ENVIRONMENT (file assignment and special-names — INPUT-OUTPUT SECTION with SELECT/ASSIGN statements mapping internal file names to external DD names), DATA (storage declarations — FILE SECTION for record layouts, WORKING-STORAGE SECTION for permanent fields, LINKAGE SECTION for passed parameters), and PROCEDURE DIVISION (executable statements, optionally with USING for passed parameters). The DATA DIVISION is where most retainer diagnostic work begins: PIC clause declarations define field storage types and sizes. PIC X(n) declares n bytes of character storage; PIC 9(n) declares n-digit display numeric; PIC S9(n) COMP-3 declares signed packed-decimal (BCD) storage in ⌈(n+1)/2⌉ bytes — the most common type for currency amounts and counters in financial COBOL because it is more compact than display and more portable than binary; PIC S9(n) COMP or COMP-4 declares binary integer storage in 2 or 4 bytes depending on digit count. A field declared PIC S9(7) COMP-3 occupies 4 bytes and stores values from -9999999 to 9999999, while the same field as PIC S9(7) COMP occupies 4 bytes in binary, but PIC S9(8) COMP also occupies 4 bytes — the digit boundary for COMP storage size is 4 digits (2 bytes), 9 digits (4 bytes), 18 digits (8 bytes). Retainer work reviewing data layouts catches storage mismatch errors: a PIC S9(4) COMP field receiving a value from a MOVE statement that can exceed 9,999 silently truncates without an OVERFLOW condition unless the receiving field is sized correctly.

Level numbers structure the DATA DIVISION record hierarchy. Level 01 is a top-level record or group item. Level 02 through 49 are subordinate fields within a group — by convention, 05, 10, 15 are used for nesting. Level 77 is a standalone non-group item (equivalent to a level-01 elementary field, avoided in modern COBOL in favor of explicit level-01 declarations). Level 88 is a condition-name: 88 ACCOUNT-TYPE-CHECKING VALUE 'C' defines a named boolean that is true when the parent field contains 'C', enabling IF ACCOUNT-TYPE-CHECKING rather than IF ACCOUNT-TYPE = 'C'. Level 66 is a RENAMES clause for aliasing a range of fields. The REDEFINES clause — 05 AMOUNT-BINARY REDEFINES AMOUNT-DISPLAY PIC S9(7) COMP-3 — overlays two field definitions on the same storage location, enabling the same bytes to be interpreted as either display numeric or packed-decimal without a MOVE. Retainer work diagnosing data corruption in record sharing between programs often traces back to REDEFINES clauses where one program wrote via the display overlay and another read via the binary overlay, producing garbage when the packed-decimal bit pattern was not a valid EBCDIC character sequence.

The PROCEDURE DIVISION's most important structured constructs in production COBOL are PERFORM VARYING, EVALUATE, and STRING/UNSTRING. PERFORM VARYING IDX FROM 1 BY 1 UNTIL IDX > TABLE-LENGTH iterates with an explicit counter; inline PERFORM with END-PERFORM is preferred over paragraph-name PERFORM in modern COBOL for clarity. The EVALUATE statement implements case logic: EVALUATE TRUE WHEN COND-A ... WHEN COND-B ... WHEN OTHER ... END-EVALUATE avoids nested IF chains and documents the intent more clearly than a sequence of IF/ELSE IF. The STRING verb concatenates multiple fields: STRING FIRST-NAME DELIMITED BY SPACE ' ' DELIMITED BY SIZE LAST-NAME DELIMITED BY SPACE INTO FULL-NAME WITH POINTER PTR — DELIMITED BY SPACE stops at the first trailing space (trimming the field), DELIMITED BY SIZE uses the full declared length. UNSTRING parses delimited input: UNSTRING INPUT-RECORD DELIMITED BY ',' INTO FIELD-1 FIELD-2 FIELD-3 WITH POINTER PTR TALLYING IN COUNT splits a comma-separated string and counts the number of tokens parsed. The COMPUTE verb performs arithmetic with standard operator precedence: COMPUTE NET-AMOUNT = GROSS-AMOUNT - (GROSS-AMOUNT * TAX-RATE / 100) — parentheses required for correct evaluation order when mixing multiplication and subtraction.

VSAM file organization, KSDS design, and JCL DD parameters

VSAM (Virtual Storage Access Method) is the primary file organization for production COBOL data on z/OS. Three VSAM organization types cover the main access patterns: KSDS (Key-Sequenced Dataset) stores records in key order and supports random access by key, sequential access by key range, and direct UPDATE/DELETE by key — the standard choice for account master files, customer records, and any dataset requiring both random lookup and sequential reporting access. ESDS (Entry-Sequenced Dataset) stores records in insertion order with no key, supports sequential access and direct access by relative byte address (RBA), and allows WRITE to add records at end and READ forward but not DELETE or keyed READ — the standard choice for audit logs and sequential event streams. RRDS (Relative Record Dataset) stores fixed-length records addressed by relative record number (RRN) starting at 1, supports direct READ/WRITE/DELETE by RRN and sequential access — the standard choice for arrays of fixed-size objects requiring direct indexed access without key ordering overhead.

VSAM KSDS cluster definition via IDCAMS DEFINE CLUSTER determines the physical storage layout: DEFINE CLUSTER (NAME(PROD.ACCOUNT.MASTER) INDEXED KEYS(8 0) RECORDSIZE(240 400) FREESPACE(15 10) VOLUMES(DSKPRD) CYLINDERS(500 100)) DATA(NAME(PROD.ACCOUNT.MASTER.DATA) CONTROLINTERVALSIZE(4096)) INDEX(NAME(PROD.ACCOUNT.MASTER.INDEX)). The KEYS(8 0) specifies an 8-byte key at offset 0. RECORDSIZE(240 400) gives the average and maximum record length for variable-length records. FREESPACE(15 10) reserves 15% of each control interval and 10% of each control area for record insertions and size growth — the critical parameter for preventing CI and CA splits. CONTROLINTERVALSIZE(4096) sets the CI size, which determines I/O transfer granularity; larger CI sizes improve sequential throughput but increase the cost of random single-record access. Reading LISTCAT output after production batch runs is the primary diagnostic: IDCAMS LISTCAT ENTRIES(PROD.ACCOUNT.MASTER) ALL shows SPLITS-CI and SPLITS-CA counters — CI splits indicate free space exhausted within a control interval (VSAM creates a new CI and redistributes records), CA splits indicate all CIs in a control area are full (VSAM allocates a new CA). Sustained CI split rates above 50 per batch window indicate insufficient FREESPACE for the record growth pattern, requiring a REPRO to a redesigned cluster. IDCAMS REPRO INFILE(OLDCLUST) OUTFILE(NEWCLUST) copies all records from the old cluster to the newly defined one with corrected FREESPACE settings.

VSAM record access in COBOL requires SELECT/ASSIGN, FD, and ACCESS MODE declarations. SELECT ACCOUNT-MASTER ASSIGN TO AS-ACCMST ORGANIZATION IS INDEXED ACCESS MODE IS DYNAMIC RECORD KEY IS ACCT-KEY FILE STATUS IS ACCT-STATUS declares dynamic access mode (both sequential and random access within the same OPEN) with a file status code in ACCT-STATUS checked after each file operation. The OPEN statement must match the access intent: OPEN I-O ACCOUNT-MASTER for update access, OPEN INPUT ACCOUNT-MASTER for read-only. The START verb positions within the KSDS by key comparison: START ACCOUNT-MASTER KEY IS NOT LESS THAN ACCT-KEY positions at the first record with key greater than or equal to the search key, after which READ NEXT fetches records sequentially. INVALID KEY in READ, WRITE, REWRITE, and DELETE statements catches key errors: FILE STATUS 23 indicates the record was not found on READ or DELETE, 22 indicates a duplicate key on WRITE, 02 indicates a duplicate alternate key, and 00 indicates success. JCL DD statements define the physical dataset associated with each ASSIGN: //AS-ACCMST DD DSN=PROD.ACCOUNT.MASTER,DISP=SHR for shared read access, DISP=OLD for exclusive update access. The SPACE parameter for new sequential datasets — SPACE=(CYL,(10,5),RLSE) for 10 cylinders primary, 5 secondary, release unused space at close — must be sized to accommodate the full expected output to avoid B37 (out-of-space) abends during batch production runs.

JCL batch, DFSORT, and checkpoint/restart

JCL (Job Control Language) structures batch execution as a sequence of steps, each an invocation of a program with its DD statements. The COND parameter on EXEC PGM statements controls conditional step execution based on prior step return codes: COND=(4,LT) skips the step when 4 is less than the previous highest return code (i.e., when any prior step returned 5 or higher); COND=(0,NE,SORT01) skips the step when step SORT01 returned nonzero; COND=EVEN runs the step even if a prior step abended (required for cleanup steps); COND=ONLY runs the step only when a prior step abended (for error recovery). The COND logic is inverted from natural language — "skip when condition is true" — which requires care in multi-step jobs: COND=(4,LT) means "skip this step if 4 < highest prior return code", so the step runs only when all prior steps returned 4 or less. Return code conventions: 0 is success, 4 is warning (step succeeded with advisory messages), 8 is error (step failed partially), 12 is severe error, 16 is abend-equivalent failure.

DFSORT (IBM's sort utility on z/OS) processes sequential input records with SORT, MERGE, COPY, and OUTFIL statements in SYSIN DD. The fundamental sort control statement — SORT FIELDS=(1,8,CH,A,25,4,ZD,D) — sorts ascending by character field at offset 1 length 8, then descending by zoned-decimal field at offset 25 length 4. Format codes: CH for character, ZD for zoned-decimal display, PD for packed-decimal COMP-3, BI for binary, FS for floating-point sign. INCLUDE and OMIT filter records before sorting: INCLUDE COND=(10,2,CH,EQ,C'NY') passes only records where bytes 10-11 equal 'NY'; OMIT COND=(1,1,CH,EQ,C'D') drops records with a 'D' delete flag. OUTFIL generates multiple output datasets from a single sort pass: OUTFIL FNAMES=NYONLY,INCLUDE=(10,2,CH,EQ,C'NY') and OUTFIL FNAMES=OTHER,OMIT=(10,2,CH,EQ,C'NY') create two distinct output files without requiring a second sort pass. DFSORT SYNCSORT behavioral difference: SYNCSORT's INCLUDE/OMIT syntax is compatible but its INREC/OUTREC reformatting field reference syntax uses different column numbering conventions — a DFSORT control deck that references field positions via INREC BUILD=(1,80,SQN,8,ZD,CHANGE=(8,C'00000000',SEQNUM)) will not run under SYNCSORT without modification.

Checkpoint/restart allows a long-running batch job to resume from a mid-job savepoint after an abnormal termination rather than restarting from the beginning. IBM checkpoint support uses the CHKPT macro in the program — CHKPT DCB,SYNCH writes a checkpoint record to the SYSCHK DD dataset — combined with the RESTART parameter on the JOB card: // JOB (acct),'name',RESTART=(STEP05.step05a) to restart execution at a specific step. The RDJFCB macro reads the JFCB (Job File Control Block) for a DD statement to determine the current file position, enabling the restart step to skip records already processed before the checkpoint. For VSAM-based checkpoints, the application saves the last processed key in the COMMAREA or a checkpoint record, and the restart logic uses START with that key to resume KSDS processing from where it left off. Retainer work implementing checkpoint/restart focuses on identifying the natural checkpoint granularity (every 10,000 records for a 2-million-record file is a common starting point), designing the checkpoint record format to capture all in-flight accumulators and the last processed key, and testing the restart path by simulating abends at checkpoint boundaries to verify that the restart produces identical output to the uninterrupted run.

CICS online transactions, BMS maps, and pseudoconversational design

CICS (Customer Information Control System) executes COBOL programs as transactions in response to terminal screen input. BMS (Basic Mapping Support) defines screen layouts as map definitions in assembler macro syntax — DFHMSD for the mapset, DFHMDI for individual maps, DFHMDF for each field on the map with ATTRB (attribute byte: PROT for protected, UNPROT for enterable, ASKIP for autoskip, BRT/NORM/DRK for intensity, IC for initial cursor position, FSET for modified data tag presetting), LENGTH, and INITIAL (default value). The COBOL program interacts with the BMS map through EXEC CICS RECEIVE MAP and EXEC CICS SEND MAP: EXEC CICS RECEIVE MAP('ACCTMAP') MAPSET('ACCTMAPS') INTO(ACCT-MAP-AREA) END-EXEC reads terminal-entered data into the working-storage map structure; EXEC CICS SEND MAP('ACCTMAP') MAPSET('ACCTMAPS') FROM(ACCT-MAP-AREA) ERASE END-EXEC sends the map to the terminal with the ERASE option to clear the previous screen contents. The EIB (Execute Interface Block) provided by CICS contains EIBTRNID (current transaction ID), EIBDATE and EIBTIME (transaction start date/time in YYYDDD and HHMMSS format), EIBRESP and EIBRESP2 (response codes from the last CICS command — EIBRESP value 13 is NOTFND, 16 is DUPREC, 26 is INVREQ), and EIBCALEN (length of the COMMAREA passed to this invocation, zero on a cold start).

Pseudoconversational design is the architectural pattern that prevents CICS TCB exhaustion under load. In a conversational design, the program issues EXEC CICS SEND MAP, then waits in the program for the operator's keystroke response before continuing — this holds the CICS task and its TCB for the entire think time between sends. In a pseudoconversational design, the program issues EXEC CICS SEND MAP followed immediately by EXEC CICS RETURN TRANSID('ACCT') COMMAREA(SAVE-AREA) LENGTH(500) END-EXEC, which terminates the current task and returns the TCB to the pool. When the operator presses Enter, CICS re-invokes the transaction with the saved COMMAREA passed back to the new invocation's LINKAGE SECTION. The COMMAREA size limit of 32,767 bytes constrains how much state can be carried across pseudoconversational interactions; for applications requiring larger state (multi-page inquiry results, complex workflow context), EXEC CICS BTS (Business Transaction Services) provides named containers with no size limit. The redesign from conversational to pseudoconversational requires identifying every RECEIVE MAP that follows a SEND MAP in the same task, extracting the program state at the SEND point into a COMMAREA layout, and adding EIBCALEN-checking logic at program entry to distinguish a cold start (EIBCALEN = 0, initialize COMMAREA) from a return (EIBCALEN > 0, restore COMMAREA state).

EXEC CICS HANDLE ABEND provides structured abend handling within CICS tasks: EXEC CICS HANDLE ABEND PROGRAM('ABNDPROG') END-EXEC transfers control to an error-handling program on any unhandled abend within the task. EXEC CICS PUSH HANDLE and EXEC CICS POP HANDLE allow saving and restoring the current condition/abend handler set around a section of code with different handling requirements. Dynamic storage within a CICS task — storage that persists for the task duration but is released at task end — is obtained with EXEC CICS GETMAIN SET(PTR-FIELD) LENGTH(WS-LEN) FLENGTH(WS-FLENGTH) END-EXEC where PTR-FIELD is a pointer receiving the address of the allocated storage; the corresponding EXEC CICS FREEMAIN DATA(PTR-FIELD) releases it explicitly. The TWA (Transaction Work Area) is a fixed-size block of storage defined in the TCT (Terminal Control Table) entry for the terminal, available to all programs in the same CICS task via EXEC CICS ADDRESS TWA(TWA-POINTER) END-EXEC — it is reused across tasks on the same terminal and must be initialized at transaction start. The CWA (Common Work Area) is a single block shared across all CICS tasks in the region, accessed via EXEC CICS ADDRESS CWA(CWA-POINTER) END-EXEC — updates require serialization via EXEC CICS ENQ/DEQ or a flag-based protocol to prevent concurrent modification from parallel tasks.

DB2/Db2 embedded SQL, cursor design, and BIND isolation tuning

DB2 (now branded Db2 for z/OS) embeds SQL in COBOL programs using EXEC SQL ... END-EXEC delimiters. The precompiler (DB2PRECOM or DSNHPC) translates embedded SQL into COBOL CALL statements and produces a DBRM (Database Request Module) that is bound into a package or plan via the BIND command before the program can execute. Single-row retrieval uses SELECT INTO: EXEC SQL SELECT ACCT-BALANCE, ACCT-TYPE INTO :WS-BALANCE, :WS-ACCT-TYPE FROM ACCOUNT_MASTER WHERE ACCOUNT_ID = :WS-ACCOUNT-ID END-EXEC — the host variable names prefixed with colons are COBOL working-storage fields. SQLCODE in SQLCA must be checked after every EXEC SQL statement: SQLCODE 0 is success, SQLCODE 100 is not-found (no rows matched), positive SQLCODE values are warnings, negative values are errors. The most common production error is failing to check SQLCODE before using the INTO variables — a SQLCODE 100 on a SELECT INTO leaves the host variables unchanged from their prior values, silently producing incorrect output if not checked.

Multi-row retrieval uses cursors. The four-step cursor lifecycle: EXEC SQL DECLARE C1 CURSOR FOR SELECT ACCOUNT_ID, BALANCE FROM ACCOUNT_MASTER WHERE BRANCH_CODE = :WS-BRANCH ORDER BY ACCOUNT_ID END-EXEC declares the cursor at program load (this statement is not executed, only registered). EXEC SQL OPEN C1 END-EXEC positions the cursor before the first row and executes the SELECT against the database. The FETCH loop: EXEC SQL FETCH C1 INTO :WS-ACCOUNT-ID, :WS-BALANCE END-EXEC advances to the next row and populates the host variables — loop until SQLCODE = 100 (end of result set). EXEC SQL CLOSE C1 END-EXEC releases the cursor's resources. Null indicator variables — a paired COMP field declared alongside each nullable host variable: 01 WS-BALANCE PIC S9(13)V99 COMP-3. 01 WS-BALANCE-IND PIC S9(4) COMP — are populated by DB2 with -1 when the column is null, 0 when not null. Using a nullable column's host variable without checking the indicator field is a common source of incorrect calculation results when null values represent legitimately missing data.

BIND options control how DB2 optimizes and executes packages. ISOLATION determines the locking behavior: ISOLATION(CS) (cursor stability, the default) acquires a lock on the current row and releases it when the cursor advances — appropriate for update cursors where you need to hold the row while updating; ISOLATION(UR) (uncommitted read) acquires no locks and reads whatever data is in the buffer — appropriate for reporting cursors on tables with no concurrent updaters, eliminating all lock acquisition overhead; ISOLATION(RS) (read stability) holds locks on all rows examined by the predicate for the transaction duration — appropriate for queries where the result set must remain stable across multiple fetch calls in the same unit of work. EXPLAIN populates the PLAN_TABLE after EXPLAIN PLAN FOR SELECT ... or after BIND with EXPLAIN(YES): the ACCESSTYPE column shows whether DB2 chose index access (I for index scan, I1 for one-fetch index-only, M for multiple index access) or R (tablespace scan). RUNSTATS — RUNSTATS TABLESPACE dbname.tsname TABLE(ALL) INDEX(ALL) — updates the statistics in the DB2 catalog used by the optimizer. After major batch loads or deletes, the catalog statistics can diverge significantly from actual data distribution, causing the optimizer to choose suboptimal access paths (tablespace scan where an index was available because RUNSTATS showed a uniform key distribution that no longer holds after a skewed batch insert). Scheduling RUNSTATS after batch runs that change more than 20% of a table's rows is standard retainer advisory practice.

How HourTab tracks COBOL developer retainer hours

COBOL developer retainers produce some of the least visible work-to-deliverable ratios of any platform retainer. A session that resolved the overnight batch window expansion produced a single IDCAMS DEFINE CLUSTER statement with different FREESPACE parameters. The session involved running LISTCAT on the account master cluster before and after two batch windows to establish that SPLITS-CI was increasing by 847 per window, reading the original DEFINE CLUSTER JCL to find the FREESPACE(5 2) definition from 2001, calculating the expected CI occupancy given the current average record size of 310 bytes against the 240-byte design-time estimate and the 4096-byte CI size, determining that the 5% FREESPACE left only 204 bytes per CI for expansion — less than a single record update, sizing the replacement FREESPACE(20 10) to leave 819 bytes per CI based on measured record size variance, and scheduling the IDCAMS REPRO job in the weekend batch window to reload the cluster. The log entry “fixed VSAM splits, 28h” gives the client no path from 28 hours to the cluster redefinition that halved the batch window — because nothing in one DEFINE CLUSTER parameter change communicates the LISTCAT analysis and free-space sizing calculation that produced it.

HourTab gives COBOL developers a public retainer-hours URL they share with each client at the start of the engagement. The client opens the URL and sees the current burn-down without logging in. For mainframe retainers specifically, the work log format carries the weight: each entry should name the LISTCAT counter and the delta it showed (SPLITS-CI: 847 per batch window before FREESPACE redesign; 14 per window after DEFINE CLUSTER with FREESPACE(20 10)), the CICS statistic and the TCB count improvement (concurrent TCBs at peak: 198 of 200 high-water-mark reduced to 22 after pseudoconversational redesign — EXEC CICS RETURN TRANSID COMMAREA carries state across 500-byte COMMAREA), the DB2 EXPLAIN access type and the elapsed time delta (TABLESPACE SCAN on ACCOUNT_MASTER 40M rows → index range scan after RUNSTATS and BIND ISOLATION(UR); report step elapsed: 14 min → 47 sec), the COND parameter and the step it protected (COND=(4,LT) on REPORT01 step — skips report generation when SORT01 step returns RC>4; prevents reading partially-written SORTOUT dataset), and the DFSORT control statement and the record count filtered (INCLUDE COND=(10,2,CH,EQ,C'NY') on REGSORT step — 1.2M of 3.4M input records passed to report; eliminated two downstream batch steps processing the full dataset). That entry takes five minutes to write and turns the client status call from a thirty-minute explanation of what VSAM CI splits are and why the batch window grows into a two-sentence acknowledgment that the overnight run is completing in four hours again.

The retainer model fits COBOL mainframe engineering because the platform evolves on a different timescale than the application — z/OS release upgrades change storage manager behavior and CICS service definitions, DB2 for z/OS releases introduce new optimizer statistics tables that require RUNSTATS schedule review, and IBM service APARs occasionally change DFSORT control statement syntax or VSAM catalog record formats in ways that require existing JCL to be audited. A project contract closes when the current batch window or CICS TCB problem is resolved. A COBOL retainer stays open for the next VSAM dataset that outgrows its FREESPACE definition, the next DB2 statistics aging that causes an access path regression after a major batch load, and the next CICS pseudoconversational conversion that prevents a TCB exhaustion event during a peak business period.

FAQ: COBOL developer retainers