Enscribe developer on retainer: FARESVR PATHWAY serverclass acquires FARECARD record lock but error-exit path skips UNLOCKRECORD — lock held by idle server instance causes FE 40 during morning boarding rush — HP NonStop Enscribe developer on monthly retainer

HPE NonStop NS-X • NonStop OS J06.20 • TAL • PATHWAY • FARECARD KSAM LOCKREQ 1 • READUPDATELOCK • File Error 40 lock timeout • retainer billing

At Coastal Ferry Authority, card readers at the four main terminal gates began intermittently refusing boardings every weekday morning between 07:15 and 08:45. The error displayed on each gate terminal: “Card system busy — please retry.” The retry succeeded about 80% of the time; 20% of retries failed again and the rider was directed to a staffed booth. For 14 consecutive business days, an average of 31 card errors per terminal were logged during the 90-minute morning rush window before the HP NonStop Enscribe developer retainer was engaged to diagnose the pattern.

The fare card system ran on an HPE NonStop NS-X series server with NonStop OS J06.20. The central database was an Enscribe key-sequenced file at \FERRY1.$FARES.CARDDB.FARECARD, holding 1.47 million active card records. Each record was 512 bytes, keyed on CARD-ID (16 bytes, unique). The file was created with LOCKREQ 1 — record-level locking enabled — so that concurrent updates to a fare card record from multiple PATHWAY server instances would not collide.

A PATHWAY serverclass, FARESVR, handled boarding requests from the gate terminals. PATHWAY spawned three FARESVR server instances to meet peak throughput. Each instance was a persistent TAL process that opened FARECARD once at startup and held the file open for the duration of the serverclass lifetime. When a gate terminal submitted a boarding request, PATHWAY routed it to an available FARESVR server instance.

The FARESVR boarding flow and the READUPDATELOCK call

The standard boarding flow in FARESVR had five steps:

1. FARESVR receives the CARD-ID from the gate terminal via a PATHWAY request message.
2. FARESVR calls READUPDATELOCK on FARECARD with the CARD-ID key. This reads the card record into the program’s working buffer AND acquires an Enscribe exclusive record lock on that record. While the lock is held, no other FARESVR server instance can call READUPDATELOCK on the same CARD-ID — it will block for the LOCKWAIT duration (8 seconds, set at file open) and then receive File Error 40 (lock-wait timeout) if the lock is not released.
3. FARESVR validates the card record against four criteria: card-not-found, card-suspended, balance-zero, daily-limit-reached.
4. If the card passes validation, FARESVR decrements the trip count and calls WRITE to write the updated record back to FARECARD. The Enscribe record lock is released implicitly as part of the WRITE operation.
5. FARESVR calls PATHWAY$REPLY to send the approval response to the terminal and returns to PATHWAY. PATHWAY marks the server instance as available for the next request.

This five-step sequence was correct for the normal boarding path. The bug was in the error-handling logic for the four validation failure cases.

The missing UNLOCKRECORD on error-exit paths

For each of the four validation failures, FARESVR’s error path sent an error response to the gate terminal and returned to PATHWAY — without ever writing back to the FARECARD file. Since the record was never updated, FARESVR skipped the WRITE call. But it also skipped UNLOCKRECORD. The TAL structure for each error exit followed this pattern:

! Read the record and acquire exclusive lock
CALL READUPDATELOCK(farecard_file, buf, 512, card_id, 16) -> fe;
IF fe <> 0 THEN BEGIN
    ! Card not found
    CALL PATHWAY$REPLY(err_buf, err_len);  ! BUG: no UNLOCKRECORD before reply
    RETURN;
END;

! Validate: card suspended?
IF buf.status_code = SUSPENDED THEN BEGIN
    CALL PATHWAY$REPLY(suspend_buf, suspend_len);  ! BUG: lock still held
    RETURN;
END;

! Validate: balance zero?
IF buf.balance = 0 THEN BEGIN
    CALL PATHWAY$REPLY(empty_buf, empty_len);  ! BUG: lock still held
    RETURN;
END;

! Validate: daily limit reached?
IF buf.daily_trips >= MAX_DAILY_TRIPS THEN BEGIN
    CALL PATHWAY$REPLY(limit_buf, limit_len);  ! BUG: lock still held
    RETURN;
END;

! Normal boarding path
buf.daily_trips := buf.daily_trips + 1;
CALL WRITE(farecard_file, buf, 512);  ! Correct: WRITE releases the lock
CALL PATHWAY$REPLY(ok_buf, ok_len);

In each of the four error exit paths, PATHWAY$REPLY was called to send the response and RETURN was called to release the FARESVR server instance back to PATHWAY. The server instance was now available to process the next PATHWAY request. However, the Enscribe record lock acquired by READUPDATELOCK was not released by PATHWAY$REPLY or RETURN — those are PATHWAY-level operations that mark the server as available for messaging, but they have no effect on Enscribe file system locks. The lock on the error-case CARD-ID remained active on the FARESVR server process, persisting across subsequent PATHWAY request cycles on that server instance.

How stale locks accumulated during the morning rush

During the 07:15–08:45 peak, the four terminals collectively processed 3,000–4,500 PATHWAY requests. Approximately 2–4% of requests triggered a validation failure — mostly daily-limit-reached for multi-leg commuters boarding their second or third ferry of the morning, with a smaller fraction of balance-zero for cards that had not been topped up. Each validation failure left one stale Enscribe record lock on the FARESVR instance that processed it.

After 30 minutes of rush traffic, each of the three FARESVR server instances was holding between 8 and 15 stale Enscribe record locks on distinct CARD-IDs. If a gate terminal submitted a boarding request for one of those locked CARD-IDs — which happened regularly, since daily-limit commuters sometimes tapped again attempting a second entry — the FARESVR instance handling the new request called READUPDATELOCK on an already-locked record. That call blocked for 8 seconds (the LOCKWAIT duration). After 8 seconds with no lock release (the holding server instance was between requests and would never release the lock voluntarily), the READUPDATELOCK returned File Error 40 (lock-wait timeout). FARESVR returned the error to PATHWAY, PATHWAY delivered it to the gate terminal, and the terminal displayed “Please retry.”

The 14-day accumulation pattern: on day 1, each rush period added roughly 60–90 stale locks across the three instances. Overnight, FARESVR instances were not restarted — PATHWAY kept them running to avoid startup latency. By day 14, each instance was holding 840–1,260 stale locks. At that lock count, even non-repeating CARD-IDs had a meaningful collision probability during the rush — not because the same card was retried, but because the lock accumulation had grown large enough that a new, first-boarding card occasionally shared a key with one of the stale locks from earlier days.

Diagnosis: PATHCOM, FILEINFO, and process ID matching

The Enscribe developer retainer began diagnosis with the PATHCOM utility to check serverclass state. PATHCOM STATUS SERVER FARESVR showed three instances, all in RECV state (idle, waiting for the next message) — consistent with normal operation. No server instance was actively stuck processing a request. The PATHWAY layer appeared healthy.

The next step was examining FARECARD for active locks. A diagnostic TAL program used FILEINFO calls to enumerate lock owners on FARECARD. Running this tool during non-rush hours (when no boarding requests were in flight) confirmed 29 locked records on FARECARD, all with lock owner process IDs matching the three FARESVR instances identified by PATHCOM. No current PATHWAY request was in progress for any of those records — the locks were residual from prior requests that had exited through an error path.

Source code review of FARESVR.tal confirmed the pattern. All four validation error paths — card-not-found (tested against the READUPDATELOCK file error return), card-suspended, balance-zero, daily-limit-reached — called PATHWAY$REPLY and RETURN without a preceding UNLOCKRECORD call. The normal (approved boarding) path was correct: it called WRITE before PATHWAY$REPLY, and WRITE released the Enscribe lock implicitly. Only the error paths were missing the explicit unlock.

The fix: UNLOCKRECORD on all error exit paths

The corrected FARESVR TAL structure for the error paths:

CALL READUPDATELOCK(farecard_file, buf, 512, card_id, 16) -> fe;
IF fe <> 0 THEN BEGIN
    ! Card not found — no lock was acquired (READUPDATELOCK failed)
    CALL PATHWAY$REPLY(err_buf, err_len);
    RETURN;
END;

IF buf.status_code = SUSPENDED THEN BEGIN
    CALL UNLOCKRECORD(farecard_file);  ! Release lock before reply
    CALL PATHWAY$REPLY(suspend_buf, suspend_len);
    RETURN;
END;

IF buf.balance = 0 THEN BEGIN
    CALL UNLOCKRECORD(farecard_file);  ! Release lock before reply
    CALL PATHWAY$REPLY(empty_buf, empty_len);
    RETURN;
END;

IF buf.daily_trips >= MAX_DAILY_TRIPS THEN BEGIN
    CALL UNLOCKRECORD(farecard_file);  ! Release lock before reply
    CALL PATHWAY$REPLY(limit_buf, limit_len);
    RETURN;
END;

! Normal path: WRITE releases the lock implicitly
buf.daily_trips := buf.daily_trips + 1;
CALL WRITE(farecard_file, buf, 512);
CALL PATHWAY$REPLY(ok_buf, ok_len);

Note on the card-not-found path: when READUPDATELOCK itself returns a non-zero file error, the record was not found and no lock was acquired. UNLOCKRECORD on a file with no held lock is a no-op but harmless. The correction for the other three paths added explicit UNLOCKRECORD calls immediately before PATHWAY$REPLY — after the error condition is determined but before sending the reply and returning to PATHWAY.

After the corrected FARESVR binary was deployed and the PATHWAY serverclass was restarted (clearing all 29 stale locks in one operation), FILEINFO enumeration during off-hours showed 0 locked records. Morning rush FE 40 errors: 0 across the next 5 business days.

A secondary hardening was added to the FARESVR startup procedure: on initialization, the server now calls FILEINFO to enumerate any existing Enscribe locks on FARECARD held by its process ID and calls UNLOCKRECORD for each one. This defends against a crash-restart scenario in which a server instance terminates while holding locks — if the replacement server instance is assigned the same process ID, the defensive cleanup at startup will clear any orphaned locks before the instance begins processing requests.

How HourTab helps HP NonStop Enscribe developer retainers

The Coastal Ferry Authority engagement totaled 5.5 hours across two sessions on the same day: initial triage (1.0h, reviewing PATHCOM output and gate terminal error logs to confirm the FE 40 pattern and isolate it to FARESVR), lock diagnosis (1.5h, writing and running the FILEINFO diagnostic tool, matching process IDs, confirming 29 stale locks during off-hours), source review (1.0h, reading FARESVR.tal and confirming all four error-exit paths), fix (0.5h, adding three UNLOCKRECORD calls and confirming the card-not-found path needed no change), staging test (1.0h, simulating each of the four validation failure conditions on the staging NonStop system, confirming no stale locks post-fix), and startup-procedure hardening (0.5h, adding FILEINFO + UNLOCKRECORD sweep to FARESVR initialization).

In a HourTab retainer log, the diagnostic session reads: “FARESVR Enscribe LOCKREQ-1 lock not released on error exits: READUPDATELOCK acquires record lock; four error paths (card-not-found, card-suspended, balance-zero, daily-limit-reached) call PATHWAY$REPLY + RETURN without UNLOCKRECORD; lock persists on idle server instance across request cycles; PATHCOM shows 3 FARESVR instances in RECV state; FILEINFO diagnostic: 29 stale FARECARD locks, all held by FARESVR process IDs; source confirmed: only WRITE on the normal boarding path released locks; fix: UNLOCKRECORD added before PATHWAY$REPLY on 3 error paths (card-not-found path needed no change — READUPDATELOCK file error means no lock was acquired); FARESVR restarted, 29 locks cleared, FILEINFO post-fix: 0 stale locks; morning rush FE 40 errors: 0 (5 business days confirmed); added startup FILEINFO sweep for defensive cleanup; 5.5h.” Coastal Ferry Authority can see the 8-second LOCKWAIT penalty, the 14-day accumulation root cause, and the zero-error result of the fix.

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: HP NonStop Enscribe developer retainers

See also: Tandem developer retainer · Btrieve developer retainer · IDMS developer retainer