Blog › ICP guides

InterSystems Caché developer on retainer: global lock not released on failure, ObjectScript developer, Caché MUMPS developer on monthly retainer

October 9, 2026 · ~15 min read

A Caché ObjectScript developer was maintaining a clinical data processing routine in a hospital information system running InterSystems Caché 2018.1 on a Windows Server 2016 host. The Caché instance served a cardiology department’s clinical data system with 60 concurrent clinical staff sessions across three nursing floors and two physician workrooms. A routine named UpdatePatientRecord processed patient record updates submitted from clinical workstations: it acquired an exclusive global lock on the patient record with LOCK +^PatientData(patientId) to prevent concurrent modifications, validated the submitted data against required field constraints, then updated the global if all validations passed. When a required field (attending physician ID) was missing from the submitted data, the validation check at line 22 called QUIT to exit the routine with an error code, returning control to the calling process. The developer who wrote UpdatePatientRecord placed the corresponding LOCK -^PatientData(patientId) release only on the success path, after the global update completed. The QUIT on the validation failure path exited the routine without calling LOCK -. In Caché’s global locking model, LOCK +^GlobalNode acquires a lock that is held in the current job’s lock table until an explicit LOCK -^GlobalNode releases it, the job exits, or a LOCK command with no arguments releases all locks for the job. With the missing physician ID validation failure, the global lock on ^PatientData(patientId) was held for the lifetime of the clinical workstation’s Caché job. Other clinical staff attempting to update the same patient record from different workstations called LOCK +^PatientData(patientId):30 (with a 30-second timeout), received a $T = 0 result indicating the lock could not be acquired within the timeout, and displayed an error message that the patient record was unavailable. Patient record lock hang incidents per week: 3–5 before fix; 0 after LOCK -^PatientData(patientId) was added before every QUIT and THROW on the failure paths.

The root cause was the behavior of Caché’s incremental locking commands and their interaction with routine control flow. Caché provides two distinct locking syntaxes with fundamentally different semantics. Non-incremental locking (LOCK ^GlobalNode without a + or - prefix) acquires the specified lock and simultaneously releases all other locks currently held by the job — a “lock set replacement” semantics that simplifies lock management for simple single-lock use cases at the cost of releasing any other locks the job holds. Incremental locking (LOCK +^GlobalNode and LOCK -^GlobalNode) adds or removes a specific lock from the job’s current lock set without affecting other locks the job holds, enabling multiple concurrent global locks to be held simultaneously. The UpdatePatientRecord routine used LOCK +^PatientData(patientId) (incremental acquisition) to preserve any other locks the calling process might hold. Incremental locks have no implicit release mechanism: they are not released at a code block boundary, at a transaction boundary, or when a local variable goes out of scope. The only ways an incremental lock is released are: an explicit LOCK -^PatientData(patientId) call, a LOCK command with no arguments (releases all job locks), or job termination. The QUIT statement in ObjectScript is a routine exit command — it exits the current routine label block and returns control to the caller, but it has no effect on any locks held by the job. When the validation failure QUIT at line 22 exited the routine, the LOCK +^PatientData(patientId) lock acquired at line 14 remained in the job’s lock table, held indefinitely.

The invisibility of the bug in development testing follows the same pattern as all multi-user lock bugs that are invisible in single-process tests. The developer who wrote UpdatePatientRecord tested the required-field validation failure on a Caché development instance with a single process. They called UpdatePatientRecord with a test submission missing the attending physician ID, verified that the routine returned the correct error code, and confirmed that no update was written to ^PatientData. The global lock was held by the developer’s own test process — the only process running in the development instance. No other process attempted to acquire LOCK +^PatientData(testPatientId) during or after the test, so no lock contention occurred, and no diagnostic showed that the lock was still held after the routine exited. In production, the lock contention pattern became visible only when two conditions were true simultaneously: a clinical workstation submitted a patient update with a missing physician ID, leaving the patient record global locked by that workstation’s Caché job, and another clinical workstation attempted to update the same patient record within the same session. The second workstation’s 30-second lock timeout expired with no indication to the clinical staff of what was holding the lock or why. The error message (“Patient record unavailable, try again in a few minutes”) did not name the routine, the lock holder, or the global node involved.

Caché: InterSystems’ commercialized MUMPS and the foundation of healthcare IT

InterSystems Corporation was founded in 1978 in Cambridge, Massachusetts by Phil Ragon, with the original mission of commercializing MUMPS (Massachusetts General Hospital Utility Multi-Programming System) for the healthcare and financial services sectors. MUMPS was developed at Massachusetts General Hospital in 1966 by Neil Pappalardo, Curt Marble, and Robert Greenes as a high-performance database programming language designed for the data access patterns of hospital information systems: sparse, hierarchical, alphanumeric key-value storage with high read/write throughput for thousands of concurrent clinical processes. MUMPS’s fundamental data structure is the global variable: a persistent, multi-dimensional sparse array stored on disk, accessible by any process in the system using the same global name. A global is declared and set simply by assigning a value to a subscripted global name: SET ^PatientData(12847,"name") = "Smith, John" creates the global node ^PatientData(12847,"name") with the value “Smith, John”, regardless of whether the parent nodes ^PatientData or ^PatientData(12847) exist. The MUMPS global model — schema-free, hierarchical, universally accessible within the instance — was ideally suited to the evolving, patient-centric data models of clinical systems in the 1970s through 1990s, where new data types were added continuously and the patient was the natural root of all data hierarchies.

InterSystems released Caché 1.0 in 1997 as a commercialized evolution of their DSM (Digital Standard MUMPS) implementation, adding three distinct access models layered over the core MUMPS global storage engine: the native ObjectScript language (a modernized, object-oriented MUMPS derivative), a persistent object model (Caché Objects, with class definitions, inheritance, and automatic global storage mapping), and a relational SQL interface (Caché SQL, which maps class properties to SQL columns and generates SQL-compatible query plans over the underlying global storage). The three-layer architecture allowed Caché to serve both the existing MUMPS developer community (who wrote ObjectScript routines accessing globals directly) and the new generation of enterprise developers who preferred object-oriented or SQL programming models. Healthcare became Caché’s dominant market: Epic Systems, the largest electronic health record (EHR) vendor in the United States (and the world by deployed patient population), built its entire Epic Chronicles clinical database on Caché in the 1990s and continues to run it today. The consequence is that a majority of large US hospitals and health systems — over 350 million patient records as of 2024 — have their primary clinical data stored in Caché (or InterSystems IRIS, Caché’s successor) globals accessed via Epic’s ObjectScript codebase. Retainer developers who can maintain the ObjectScript routines and Caché class methods that integrate with, extend, or process data alongside the Epic Chronicles database are among the most specialized and highest-compensated in the healthcare IT sector.

InterSystems introduced the InterSystems IRIS Data Platform in 2018 as the successor to Caché, adding cloud-native container deployment, distributed data management, analytics integration, and AI/ML inference capabilities to the core Caché architecture. InterSystems IRIS is backward-compatible with Caché ObjectScript: MUMPS globals, LOCK commands, DO and QUIT statements, and %ExecDirect SQL calls work identically in IRIS. The migration path from Caché to IRIS is primarily a configuration and infrastructure change rather than a code rewrite, though IRIS introduces new features (horizontal sharding, distributed globals, cloud-native deployment) that require ObjectScript code changes to exploit. The global lock model is identical in both Caché and IRIS: incremental LOCK + / LOCK - semantics, job-level lock tables, the same LOCK +^GlobalNode:timeout syntax, and the same requirement that every LOCK + must be matched by an explicit LOCK - on every exit path. Retainer developers migrating a hospital’s Caché 2018 instance to IRIS 2022 must audit every routine and class method containing LOCK + calls for missing release patterns before migration — the migration is the correct moment to perform the lock audit and fix all instances of the pattern.

Caché’s global locking model differs structurally from both SQL transactional locking and POSIX file locking in ways that are non-obvious to developers who come to ObjectScript from a SQL or file I/O background. A Caché global lock does not implicitly interact with Caché’s transaction model (TSTART / TCOMMIT / TROLLBACK): a TROLLBACK that rolls back all global changes made since the matching TSTART does not release any LOCK + locks acquired during the transaction. Conversely, a LOCK -^GlobalNode that releases a lock does not roll back any global changes made while the lock was held. Locks and transactions are entirely orthogonal mechanisms in Caché: locks prevent concurrent access; transactions provide atomicity and rollback for global data changes; neither implies the other. This means that a routine which uses both TSTART and LOCK + must manage both the transaction (commit or rollback on every exit path) and the lock (release with LOCK - on every exit path) independently. A TROLLBACK in an error handler that forgets to also call LOCK -^GlobalNode leaves the lock held even though the transaction is rolled back. The retainer developer must verify both the transaction and the lock are handled correctly on every exit path from every routine that uses both mechanisms — a more complex audit requirement than either mechanism alone.

Caché ObjectScript LOCK command: incremental locking, timeouts, and job lifetime holds

Caché’s LOCK command syntax has precise semantics that determine the scope and lifetime of the lock. The incremental LOCK +^GlobalNode syntax acquires an exclusive lock on the specified global node and adds it to the current job’s lock table without affecting other locks in the table. The LOCK +^GlobalNode:timeout syntax adds a timeout argument: the job waits up to timeout seconds for the lock to become available; after the timeout, the $T (truth value) system variable is set to 0 (false) to indicate the lock was not acquired. The correct pattern for timed incremental lock acquisition with error handling is: LOCK +^PatientData(patientId):30 on its own line; check $T immediately after (IF '$T DO ErrorHandler QUIT); proceed with the protected operation; call LOCK -^PatientData(patientId) at the end of the protected region on the success path and on every failure path. The LOCK command with a list of global nodes in a single call (LOCK +(^Node1,^Node2)) acquires all specified locks atomically — either all are acquired or none are, preventing partial acquisition deadlocks when a routine needs to lock multiple globals before proceeding. The corresponding release is LOCK -(^Node1,^Node2).

Diagnosing a Caché global lock hold requires access to the Caché system’s lock table and job list. The primary diagnostic command is ^%JOBEXAM — a Caché system utility that displays information about all running jobs, including their current status, the routine they are executing, and the globals they have locked. From the Caché terminal prompt (in the appropriate namespace), running DO ^%JOBEXAM and selecting the lock display option shows all current global locks held by each job, with the global node (e.g., ^PatientData(12847)), the lock type (exclusive or shared), and the job number of the lock holder. The Caché System Management Portal (accessible via web browser) provides the same information through the “Active Processes” and “Lock Manager” pages under the System Operation section. In the UpdatePatientRecord incident, the ^%JOBEXAM output during a hang incident shows: ^PatientData(12847) locked exclusively by job 3421; jobs 3422 and 3423 waiting for LOCK +^PatientData(12847) with timeout 30 seconds (their $T will return 0 after 30 seconds). Cross-referencing job 3421 with the ^%JOBEXAM job detail shows the holding job’s last executed line: UpdatePatientRecord+8^ClinicalDB (routine label UpdatePatientRecord, line 8 relative to the label, in the ClinicalDB routine file), combined with the job’s user name and workstation IP address, which identifies the specific clinician whose session is holding the lock. The diagnosis: job 3421 executed UpdatePatientRecord, reached the validation failure path at line 22, called QUIT without LOCK -^PatientData(12847), and returned to the workstation’s idle state while the lock persisted in the job’s lock table.

The fix for UpdatePatientRecord requires adding LOCK -^PatientData(patientId) before every QUIT and every THROW on any path that exits the locked region without having called the release on the success path. For a routine with multiple exit points, the most maintainable pattern is to restructure the routine to use Caché’s TRY/CATCH error handling, placing the LOCK - in the CATCH block to ensure it executes on all error paths, and placing a second LOCK - at the end of the TRY block for the success path. The restructured UpdatePatientRecord: LOCK +^PatientData(patientId):30; IF '$T THROW ##class(%Exception.General).%New("LockTimeout",,"Patient record locked"); enter TRY block; validate required fields — on failure, THROW ##class(%Exception.General).%New("ValidationError",,"Missing physician ID") (the CATCH block will release the lock); update ^PatientData; LOCK -^PatientData(patientId) (success path release); end TRY; CATCH e: LOCK -^PatientData(patientId); re-throw or set error return value; end CATCH. The TRY/CATCH pattern guarantees that the LOCK - in the CATCH block executes on any exception thrown within the TRY block — whether from an explicit THROW for validation failures, an implicit %Exception from a Caché runtime error, or a re-thrown exception from a called routine. One caveat: if the timeout expired and the lock was never acquired ($T = 0), calling LOCK -^PatientData(patientId) in the CATCH block releases a lock that was never held, which is safe in Caché (releasing a non-held lock is a no-op). Guarding with IF $T LOCK -^PatientData(patientId) in the CATCH block avoids the unnecessary release call but requires tracking the $T result in a local variable for use inside the CATCH block.

Typical Caché developer retainer work and what it looks like in a work log

Global lock not released on validation failure QUIT is the canonical Caché ObjectScript invisible production lock bug, and it appears in the work logs of nearly every Caché retainer engagement for hospital information systems, clinical data processing routines, and Epic EHR customizations with concurrent clinical staff access. The pattern is consistent across Caché 2015, 2016, 2017, 2018, and InterSystems IRIS: a routine acquires a global lock with LOCK +^GlobalNode to protect a patient or clinical record update; the developer tests the lock acquisition and the update logic on a single-process development instance; a validation failure path added to the routine for production data quality enforcement uses QUIT without the corresponding LOCK - release; in production with 40–80 concurrent clinical sessions, a validation failure leaves the patient record global locked for the job lifetime; other workstations attempting to update the same patient hang for 30 seconds (if using timeout syntax) and receive a “Patient record unavailable” error. The work log entry that makes this diagnosable and auditable: “HospitalCache instance (Caché 2018.1), CLINICAL namespace; routine UpdatePatientRecord; LOCK call site: LOCK +^PatientData(patientId):30 at line 14; $T checked at line 15 (success: continue; failure: QUIT with error code); validation failure path that lacked LOCK -: required physician ID check at line 22: IF ValidPhysician(PhysicianId) = 0 QUIT without LOCK -^PatientData(patientId); global node locked: ^PatientData(12847) (patient ID 12847); holding job: job 3421 (identified via ^%JOBEXAM: routine UpdatePatientRecord+8, user CLINICIAN01, workstation 192.168.10.47); blocked jobs: 3422 (CLINICIAN03, workstation 192.168.10.52) and 3423 (CLINICIAN07, workstation 192.168.10.61) both waiting on LOCK +^PatientData(12847):30; hang incidents per week before fix: 3–5 (each resolved by the Caché SA killing job 3421 or waiting for the workstation idle timeout); fix: LOCK -^PatientData(patientId) added before QUIT on validation failure at line 22, and routine refactored to TRY/CATCH with LOCK -^PatientData(patientId) in CATCH block; hang incidents per week after fix: 0; 3.5h.”

Caché class method and persistent object maintenance is the routine day-to-day Caché retainer task for clinical systems that use Caché’s object model rather than direct global access. A common engagement: the clinical informatics team reports that a Caché class method Clinical.PatientRecord:AddObservation() returns a “concurrency conflict” error intermittently when two nurses try to add observations to the same patient chart simultaneously. The retainer developer examines the class definition in Caché Studio or the VS Code Caché ObjectScript extension: Clinical.PatientRecord is a %Persistent class that stores each patient record as a global node via Caché’s automatic object-to-global mapping. The AddObservation() method calls ..%OpenId(patientId, 4) — the 4 argument is the concurrency control parameter specifying an exclusive lock (%OpenId concurrency constants: 1 = shared lock, 4 = exclusive lock). The method updates the object and calls ..%Save(). The concurrency conflict error occurs because two sessions call %OpenId(patientId, 4) for the same patient simultaneously: both sessions wait for the exclusive lock; the first acquires it, but the second is being served by a background daemon process with a very short lock timeout configured at the Caché server level. The fix involves reviewing the server-level LockTimeout configuration, changing the second session’s retry logic in the calling CSP page or web application, and ensuring that the %OpenId return value is checked for error status before proceeding. Work log: “HospitalCache (Caché 2018.1); class Clinical.PatientRecord; AddObservation() method; %OpenId(patientId, 4) exclusive lock concurrency conflict; LockTimeout server setting: 5 seconds (insufficient for concurrent clinical workflows); fix: increased LockTimeout to 30 seconds, added retry logic (3 retries with 2-second sleep) in the calling CSP page before displaying 'Record temporarily locked' to the user; concurrency conflict errors per 100 observation saves: 0.8% → 0.02%; 2h.”

InterSystems IRIS migration from Caché 2018 is the highest-investment Caché retainer engagement category for healthcare clients who need continued security patches, cloud deployment support, and SAP certification for the modern electronic medical record regulatory environment. The migration assessment begins with a namespace and routine inventory: count the ObjectScript routines and Caché class definitions in each namespace, identify routines using deprecated Caché 2015 syntax that IRIS’s updated compiler rejects, and audit all routines containing LOCK + calls for missing release patterns. InterSystems provides a Caché-to-IRIS compatibility checker tool that identifies syntax incompatibilities; the retainer developer runs the checker against the full routine set and reviews each flagged item. The global lock audit is performed independently: grep the routine text for all LOCK + occurrences and verify each call site has matching LOCK - releases on all exit paths. The migration also requires validating that IRIS’s updated SQL engine produces identical query results for any %ExecDirect SQL calls in the codebase, particularly for queries using Caché-specific SQL extensions (the %STARTSWITH operator, Caché temporal data functions, and %ROWID in subqueries). Work log: “HospitalCache Caché 2018.1 → InterSystems IRIS 2022.1 migration; 4 namespaces: CLINICAL, BILLING, PHARMACY, ADMIN; routine inventory: 847 routines, 312 class definitions; compatibility checker: 23 routines with deprecated syntax (17 using old-style DO block syntax, 6 using deprecated %Library class references updated to %SYS); LOCK + audit: 94 routines containing LOCK + calls; 6 routines missing LOCK - on failure paths identified and fixed (including UpdatePatientRecord from the prior incident); IRIS SQL compatibility: 3 queries using Caché-specific %STARTSWITH replaced with LIKE predicate; IRIS test deployment: all 4 namespaces validated against production data snapshot; hours: 22h across 3 sessions.”

How HourTab tracks Caché developer retainer hours

Caché global lock bugs caused by LOCK - missing on validation failure QUIT paths are invisible in single-process development testing by exactly the mechanism that makes all multi-process lock bugs invisible when only one process exercises the code. The developer who writes UpdatePatientRecord tests the validation failure path on a single-process Caché development instance. They call the routine with a test patient ID and missing physician ID data, verify that the routine exits with the expected error code, and confirm that no update was written to ^PatientData. The global lock acquired at line 14 is held by the developer’s own Caché job — the only job in the development instance. No other job attempts to acquire LOCK +^PatientData(testPatientId) during or after the test, so no hang or timeout occurs. The ^%JOBEXAM output would show the lock still held by the developer’s job after the test, but the developer never checks ^%JOBEXAM because there is no visible consequence of holding the lock in a single-process test. In production with 60 concurrent clinical sessions, the pattern becomes visible only when two specific clinical workstations interact with the same patient record within the same session, with one of them submitting incomplete data that triggers the validation failure path. The mean time to first production incident depends on the frequency of concurrent accesses to the same patient record and the frequency of validation failures — in a high-volume cardiology unit with rapid nurse shift handoffs, it can occur within the first week of the routine going live.

The work log entry that makes Caché global lock work auditable must name every element of the causal chain. The Caché instance name and namespace (HospitalCache, CLINICAL). The Caché version (2018.1). The routine name (UpdatePatientRecord). The LOCK + call site (line 14: LOCK +^PatientData(patientId):30). The validation failure path that lacked LOCK - (required physician ID check at line 22: IF ValidPhysician(PhysicianId) = 0 QUIT). The global node locked (^PatientData(12847)). The holding job (job 3421, identified via ^%JOBEXAM: routine UpdatePatientRecord+8, user CLINICIAN01, workstation 192.168.10.47). The blocked jobs (jobs 3422 and 3423 waiting on LOCK +^PatientData(12847):30). The hang duration before fix (30 seconds per blocked workstation timeout, each incident requiring the Caché SA to kill the holding job). The hang incident frequency before fix (3–5 per week). The fix (LOCK -^PatientData(patientId) added before QUIT on validation failure at line 22; routine refactored to TRY/CATCH with LOCK - in CATCH block). The hang incident frequency after fix (0). Hours (3.5h). A log that says “fixed Caché locking issue, 3.5h” is not auditable to a hospital IT director, a clinical informatics team lead, or the HIPAA compliance officer reviewing the change log. A log that names the routine, the LOCK call sites, the ^%JOBEXAM evidence, and the before/after hang rate is auditable, defensible, and builds the trust that sustains a long-term Caché retainer relationship. HourTab gives Caché developers a public retainer-hours URL they send to clients — hospitals, health systems, clinical informatics teams, and healthcare IT departments that run core patient care operations on InterSystems Caché or IRIS and maintain them with a Caché specialist on retainer.

The broader context for Caché retainer billing is that the global-lock-not-released-on-failure-path pattern is not unique to Caché — it is the universal consequence of explicit lock management in any platform where the developer is responsible for pairing every lock acquisition with a matching release on every code path. Sybase ASE developer retainers cover the Sybase ASE open transaction lock hold caused by ROLLBACK TRANSACTION missing on RAISERROR + RETURN failure paths — structurally identical to Caché’s LOCK + not released on QUIT failure paths: in both cases, a lock acquired before the validation check persists after the failure path exits because the developer placed the release only on the success path. Progress OpenEdge ABL developer retainers cover the Progress ABL FIND FIRST EXCLUSIVE-LOCK without RELEASE pattern — the same lock acquired before a validation operation and not released on the failure branch, the same invisible single-user test / multi-user production contention gap, the same discovery mechanism (concurrent users receiving lock-wait errors while the developer’s test never triggered the problem). RPGLE developer retainers cover IBM i record lock patterns — the same enterprise-scale lock management discipline required for multi-user clinical, financial, or manufacturing system maintenance, where the lock not released on a failure branch is the most reliably invisible class of production concurrency bug across all platforms that require explicit lock management.

FAQ: Caché ObjectScript developer retainers