Blog › ICP guides

Natural developer on retainer: READ HISTOGRAM vs READ LOGICAL, Adabas ISN conflicts, and Software AG Natural on monthly retainer

October 3, 2026 · ~14 min read

A Natural developer was maintaining a Natural/Adabas reporting and update application at a German manufacturing company running Software AG Natural 6.3 on z/OS. The application processed open purchase orders from an Adabas file named PURCHORDER, updating the order status field (STATUS-CD descriptor) from “OPEN” to one of several terminal states — “PROCESSED”, “CANCELLED”, or “ONHOLD” — based on business rules evaluated for each order. The developer had written the update loop using READ HISTOGRAM to iterate through the orders grouped by their STATUS-CD descriptor value, then called a Natural subprogram to evaluate each order’s business rules and applied UPDATE to write the new status when the rules indicated a state change was required.

The bug was invisible in testing because it only appeared when the same batch job was run twice against a dataset that already contained “PROCESSED” orders. Adabas READ HISTOGRAM iterates the inverted list — the descriptor index that Adabas maintains for fast lookup — in sorted order of the descriptor value. When the Natural UPDATE statement changed a PURCHORDER record’s STATUS-CD from “OPEN” to “PROCESSED”, Adabas immediately updated the inverted list: it removed the record’s entry from the “OPEN” position in the inverted list and inserted a new entry at the “PROCESSED” position. Because “PROCESSED” sorts after “OPEN” in the descriptor index, the record’s new inverted list entry appeared later in the histogram iteration sequence. READ HISTOGRAM subsequently encountered the record again at its “PROCESSED” position, processed it a second time through the business rules evaluation subprogram, and — because the business rules subprogram did not expect to receive an already-PROCESSED record — applied a second UPDATE that wrote a wrong state into the record.

The Adabas READ HISTOGRAM statement is documented (Software AG Adabas documentation, section on READ HISTOGRAM) as an iteration over the inverted list, not over the physical records. When records are modified in a way that changes their descriptor value during the iteration, the inverted list entries for those records move to new positions. If those new positions are later in the iteration sequence, the records are encountered again. This is a known and documented property of READ HISTOGRAM: it is designed for read-only histogram traversal, counting occurrences of descriptor values, not for update operations that change descriptor values. The correct approach for update iterations in Natural/Adabas is FIND: FIND PURCHORDER WITH STATUS-CD = "OPEN" retrieves the ISN (Internal Sequence Number) list for all matching records at the moment of the FIND call, before any updates are applied; subsequent READ ISN loops over those ISNs in a stable sequence that is not affected by descriptor updates during the loop. Three records that had been processed twice with wrong state transitions were corrected by switching from READ HISTOGRAM to FIND and verifying the FIND-based loop against a test dataset that covered end-of-batch double-run scenarios. The investigation — identifying that certain orders had received two state transitions, tracing the Adabas call trace to the second READ HISTOGRAM encounter, and understanding the inverted list mutation mechanism — was four hours.

The reason this class of bug is systematically invisible is that READ HISTOGRAM mutation works correctly when the descriptor value change moves the record to a position earlier in the inverted list, or when the batch job runs only once against a clean dataset. In both cases, the records are processed exactly once and the output is correct. The bug only appears when descriptor value changes move records later in the sorted index (which is true for “OPEN” → “PROCESSED” because “P” sorts after “O”), and only when the batch dataset contains records whose initial descriptor value puts them early enough in the iteration that they are still encountered before the iterator reaches their new descriptor position. On the test environment, the developer had always run the batch job against a fresh dataset with only “OPEN” records — the test dataset never contained pre-existing “PROCESSED” records, so the double-processing of re-encountered records was never triggered. The production failure only appeared when the batch job was re-run mid-month to catch late-arriving purchase orders alongside an already-partially-processed dataset.

Software AG Natural and Adabas: architecture, data model, and descriptor semantics

Software AG Natural was developed by Software AG in the late 1960s as a fourth-generation language designed primarily for use with Adabas (Adaptable Database System), Software AG’s own non-relational database engine. Natural for Mainframe runs on IBM z/OS, Siemens BS2000, and IBM VSE; Natural for UNIX and Natural for Windows extend the platform reach to open systems environments; NaturalONE is Software AG’s Eclipse-based web IDE that supports Natural development and web service wrapper generation under the Natural for Ajax framework. Current production versions range from Natural 8.x to Natural 9.x, though Natural 6.3 (as in the scenario above) remains in active production at many European enterprises. Natural is used predominantly in German and European enterprises, banking and insurance institutions, telecommunications companies, and government agencies — organizations that deployed Adabas and Natural in the 1970s and 1980s and have maintained these applications continuously since. The language’s 4GL character, its tight integration with Adabas, and the business-critical nature of the applications running on it make Natural retainer work high-stakes and detail-intensive.

The Adabas data model organizes data into files (analogous to relational tables, but with a flat record structure rather than a normalized schema). Each field in an Adabas file has a defined format: A for alphanumeric, N for numeric, P for packed decimal (BCD), F for floating point, and L for logical (boolean). Fields declared with DE=Y in the FDT (Field Definition Table) become descriptors: Adabas maintains an inverted list for each descriptor, enabling fast lookup by that field’s value without a full-file scan. Superdescriptors combine multiple fields into a single inverted list entry for compound-key lookups. Phonetic descriptors use a Soundex-like algorithm for approximate string matching. Collation descriptors support locale-aware sorting for languages with special character ordering requirements. The inverted list is the core data structure behind all descriptor-based access, and its behavior under concurrent updates — or under updates within the same iteration — is the source of the READ HISTOGRAM mutation bug class.

Each Adabas record has a unique ISN (Internal Sequence Number) assigned at record creation time. The ISN is a stable identifier: it does not change when field values are updated, when the record is modified, or when other records in the same file are added or deleted. FIND returns an ISN list — all ISNs matching the selection criterion, evaluated at the moment the FIND is executed. READ ISN uses an ISN to retrieve a specific record directly. READ HISTOGRAM iterates the inverted list (the descriptor index) in sorted descriptor value order, returning descriptor statistics (the count of records at each unique descriptor value) rather than the record data itself; it does not iterate ISNs directly. READ LOGICAL reads records in descriptor order, one full record per inverted list entry, and locks the record if an UPDATE follows; it is the correct statement for reading records in sorted descriptor order when the loop does not modify the descriptor being iterated. READ PHYSICAL reads records in physical storage order (ISN sequence), which is the fastest access pattern for sequential full-file processing because it avoids the indirection of the inverted list entirely.

Natural statements, data areas, and the FIND vs READ HISTOGRAM distinction

The FIND statement is the correct tool for update iterations over a selected subset of Adabas records. FIND file WITH descriptor = value evaluates the selection criterion against the current inverted list at FIND time, before any updates are applied, and returns a stable ISN list. That ISN list does not change even if records within the loop are subsequently updated in ways that change their descriptor values during the iteration. The ISN list was captured at FIND time; it is a snapshot. FIND with the MULTI-FETCH clause retrieves multiple ISNs per Adabas network round-trip, significantly reducing round-trip overhead for large result sets on Natural for Mainframe where each Adabas call crosses the Natural/Adabas interface. FIND FIRST retrieves a single ISN for the first matching record; FIND NUMBER retrieves only the count of matching records without retrieving any record data, useful for pre-check and reporting. READ ISN follows FIND to retrieve each record in the ISN list in sequence, providing the full record data for processing.

Natural data areas define variable scope and sharing. An LDA (Local Data Area) is used within a single program or subprogram; variables defined in the LDA are not visible outside the program that defines it. A GDA (Global Data Area) is shared across a Natural program and all subprograms it calls: both the calling program and all called subprograms declare USING GDA <gda-name> in their program header, and all GDA variables are shared by reference across the entire call stack. A PDA (Parameter Data Area) defines the parameter interface between a calling program and a subprogram, analogous to a function signature: the calling program and the called subprogram each declare the same PDA, and the calling program passes values through it at CALLNAT time. The distinction between GDA (implicitly shared, always visible to all called subprograms) and PDA (explicitly passed at each CALLNAT) is the source of the most common Natural side-effect bugs: a subprogram that modifies a GDA variable modifies it for the entire call stack, not just locally.

The CALLNAT statement calls a Natural subprogram by name: CALLNAT 'subprogram-name' USING pda-variable. Parameters are passed in a PDA. Subprograms have their own LDA and receive GDA by reference implicitly. CALLNAT parameters passed through the PDA are passed by reference, not by value — the called subprogram can modify the caller’s variables directly through the PDA, which is a common source of unintended side effects when the calling convention is not carefully documented. Natural error handling centers on the *ERROR-NR system variable, which contains the most recent error number after any Natural or Adabas statement. An ON ERROR block catches errors; ESCAPE ROUTINE exits a subprogram and returns control to the caller. Adabas response codes are returned in the Adabas Control Block and are accessible via *ISN, *NUMBER, and *COUNTER system variables after database statements. Transaction management in Natural/Adabas uses END TRANSACTION to commit all uncommitted changes since the last END TRANSACTION, and BACKOUT TRANSACTION to reverse all changes since the last END TRANSACTION. Changes are uncommitted and visible only to the current session until END TRANSACTION is issued; READ HISTOGRAM followed by UPDATE without END TRANSACTION leaves records in an uncommitted state. Proper transaction management is critical for batch jobs that process large numbers of records, where restart/recovery logic depends on knowing exactly which records were committed before an abnormal termination.

The READ LOGICAL statement reads records in descriptor order, one record per inverted list entry, and applies a record lock when followed by UPDATE. READ LOGICAL is safe for update iterations provided the UPDATE does not change the descriptor being iterated — if the iterated descriptor is not the one being modified, the inverted list positions do not shift and the iteration is stable. READ HISTOGRAM reads descriptor value statistics — the count of records at each unique descriptor value, without retrieving the record data — and is designed for read-only reporting and aggregation: how many records have STATUS-CD = “OPEN”, how many have STATUS-CD = “PROCESSED”, and so on. Using READ HISTOGRAM as the outer loop for an update operation that changes the iterated descriptor combines two incompatible semantics: histogram iteration (which assumes the inverted list is stable) with descriptor update (which modifies the inverted list during iteration). The result is the mutation bug described in the opening paragraphs.

Typical Natural retainer work and what it looks like in a work log

The READ HISTOGRAM mutation bug is the most invisible category of Natural retainer work. The pattern: a developer writes a READ HISTOGRAM loop to iterate through records by descriptor value, applies UPDATE within the loop to change that same descriptor value, and the changed records reappear later in the iteration because the Adabas inverted list update moves their entries to a later sorted position. The bug does not appear on clean datasets; it only appears when the descriptor value change moves the record later in the sorted index and the batch dataset already contains records at those later positions. Work log entry: “PURCHUPD: READ HISTOGRAM on PURCHORDER STATUS-CD descriptor; UPDATE within loop changed STATUS-CD from ‘OPEN’ to ‘PROCESSED’; Adabas inverted list mutation — ‘PROCESSED’ sorts after ‘OPEN’, records re-encountered at new inverted list position; ISNs 4521, 8903, 12447 received two state transitions; switched to FIND WITH STATUS-CD = ‘OPEN’ + READ ISN loop; added re-run test against partially-processed dataset; wrong-state records: 3 → 0; 4h.”

CALLNAT GDA side-effect bugs are the second most common Natural retainer pattern. A subprogram modifies a GDA variable — for example, a status flag or a running accumulator such as #PROC-COUNT — that the calling program also reads after CALLNAT returns. Because GDA variables are shared by reference across the entire call stack, the subprogram’s modification is visible in the calling program immediately after CALLNAT returns, even if the calling program did not intend the subprogram to modify that variable. The fix is to use an LDA counter within the subprogram for its local processing state, and return the value the caller needs through the PDA explicitly rather than writing it to a shared GDA variable. Work log entry: “CALLNAT ORDER-STATUS: subprogram modified GDA #PROC-COUNT accumulator; caller read stale count after return; added LDA counter in subprogram, returned via PDA; wrong accumulation values: 4 → 0; 2.5h.”

READ LOGICAL termination and Adabas response code handling round out the common Natural retainer work categories. Natural does not have a SQL-style IN clause; selecting records matching multiple values requires FIND WITH descriptor = value1 OR descriptor = value2, or a value table with a multi-value descriptor. A common mistake is using READ LOGICAL with a compound criterion and assuming the loop terminates when no more records match — READ LOGICAL iterates the inverted list from the starting descriptor value to the end of the file unless an ESCAPE TOP or ESCAPE BOTTOM is placed correctly; a missing ESCAPE BOTTOM causes the loop to scan the entire Adabas file rather than stopping at the last matching record, producing batch timeouts on large files. Work log entry: “READ LOGICAL loop: missing ESCAPE BOTTOM after last matching record; loop continued to end of file (1.2M records); batch timeout; added ESCAPE BOTTOM on first non-matching STATUS-CD; timeout: 1 → 0; 2h.” For Adabas response code handling: in a batch program updating 50,000 records, one record failed with Adabas response code 113 (record locked by another session); because *ERROR-NR was not checked after UPDATE, the program continued and counted the locked record as successfully updated; end-of-day reconciliation found 1 record in the wrong state; work log entry: “PURCHORDER UPDATE: Adabas response 113 (record locked) not checked; 1 wrong-state record; added ON ERROR / *ERROR-NR check after UPDATE; ESCAPE with error log entry on non-zero response; wrong-state count: 1 → 0; 1.5h.”

How HourTab tracks Natural developer retainer hours

Natural retainer work is invisible by the same mechanism that makes READ HISTOGRAM mutation dangerous: the bug only appears when specific runtime conditions coincide (descriptor changes that sort later in the inverted list, against a dataset that already contains records at those later positions). On the test environment, the batch job ran against a clean dataset; the double-processing never occurred. In production, the batch job ran against a partially-processed dataset from a prior run. The three doubly-processed records looked like normal state transitions in the Adabas change log — each UPDATE was individually valid; the problem was that two UPDATEs occurred for the same record in the same batch run.

The work log needs to name the mechanism: which Natural program, which Adabas file and descriptor, which READ HISTOGRAM statement and which UPDATE changed the descriptor being iterated, what the Adabas inverted list mutation mechanism means for that change direction (sorted later → encountered again), and what the wrong-state record count was before and after the fix. A log entry that says “fixed batch update loop, 4h” is not auditable. A log entry that names the program, the READ HISTOGRAM on STATUS-CD, the UPDATE that changed STATUS-CD from “OPEN” to “PROCESSED”, the Adabas inverted list behavior, the ISNs of the three doubly-processed records, and the fix (FIND + READ ISN) is auditable and defensible.

HourTab gives Natural developers a public retainer-hours URL they send to clients — German and European enterprises, banking and insurance institutions running z/OS, telecommunications companies, and government agencies maintaining Software AG Natural and Adabas production applications. For Natural retainers, each work log entry should name the mechanism: which Natural statement (READ HISTOGRAM, FIND, READ LOGICAL, CALLNAT), which Adabas file and descriptor or ISN, which Adabas semantic property applies (inverted list mutation, ISN stability, GDA reference passing), and what record count change confirmed the fix. Comparative context: Natural retainer work has structural overlap with adjacent 4GL languages targeting legacy enterprise data platforms. ABAP retainers cover the SAP-specific 4GL where FOR ALL ENTRIES IN empty-table semantics (returning all rows rather than no rows when the selection table is empty) is the dominant class of invisible batch processing bug — structurally analogous to Natural’s READ HISTOGRAM mutation in that both bugs only manifest under a specific runtime condition (empty table at period-end for ABAP; partially-processed dataset for Natural READ HISTOGRAM). MUMPS retainers cover a language where naked reference semantics (bare ^ inheriting runtime global context rather than lexical scope) produce silent wrong-global-node writes in Epic and VistA healthcare maintenance code.

FAQ: Natural developer retainers