Blog › ICP guides

Chapel developer on retainer: distributed domain maps, locale-aware parallelism, forall loops, coforall tasks, and Chapel HPC programming on monthly retainer

December 7, 2026 · ~15 min read

A Chapel program computing a distributed numerical simulation declared a Block-distributed domain const dom = {1..N} dmapped Block(boundingBox={1..N}) and a distributed array var arr: [dom] real. The developer wrote forall i in dom { arr[i] = compute(i); } to fill the array in parallel. On a single-locale run the program was fast. On a 4-locale run, all iterations ran on locale 0 by default — because the forall loop over the domain dispatches loop iterations to the locale that owns each index, but compute(i) inside the loop body called a helper function that read from a separate non-distributed array params declared on locale 0. Every iteration for indices owned by locales 1, 2, and 3 issued a remote get to locale 0 to read from params. The 8 remote communication events per iteration (2 reads × 4 locales × each iteration batch) slowed the simulation to single-locale speed. The developer restructured using coforall loc in Locales { on loc { const localDom = dom.localSubdomain(); forall i in localDom { arr[i] = compute(i, localParams); } } } where localParams was replicated per-locale with var localParams: [Locales.domain] ParamRecord. The on loc { } block executes the enclosed code on the target locale; localSubdomain() returns the subset of dom owned by that locale; forall i in localDom iterates over only the locale-local indices. Remote gets: 8/iteration → 0. The Chapel developer on retainer diagnosed the locale affinity mismatch: the forall loop was locale-aware for the distributed array writes but the helper function read from non-distributed locale-0 data, creating implicit remote gets that are invisible in the source code but account for most of the runtime on multi-locale runs.

The work log entry read “optimized parallel loop, 8h.” It names the result and duration. It cannot explain why the forall loop issued remote gets even though the writes went to arr correctly — Chapel’s forall over a distributed domain dispatches iterations to the owning locale for the distributed array indexed by the loop variable, but any other data access inside the loop body is subject to the locale of the loop variable, not the locale of the calling thread; params[i] inside a forall on locale 1 reads params from wherever params lives, which is locale 0 if it was declared on locale 0 without distribution. It cannot explain when to use coforall loc in Locales { on loc { } } versus forall i in dom { } — forall over a distributed domain is locale-aware for that domain’s data but not for any non-distributed data the loop body touches; coforall loc in Locales { on loc { } } explicitly migrates execution to each locale and expresses that the body should run with locale-local data; the restructuring pattern replaces implicit remote gets with explicit locale migration and data replication. It cannot explain the difference between Block and Cyclic distributions — Block assigns contiguous chunks of the index space to each locale (good for spatial locality: indices 1..N/4 on locale 0, N/4+1..N/2 on locale 1, etc.), while Cyclic assigns indices round-robin (good for load balance: indices 1,5,9,... on locale 0, 2,6,10,... on locale 1, etc.); the choice depends on whether the computation has spatial locality that benefits from contiguous chunks or irregular work distribution that benefits from interleaving. The 8 hours of locality analysis, distribution selection, replication design, and communication profile verification are invisible in the diff.

Chapel distributed domain maps: Block, Cyclic, localSubdomain(), and locale-affine execution with on loc {}

Chapel’s distributed domain maps are the core abstraction for expressing how array data is distributed across locales. The Block distribution assigns contiguous index ranges to locales: dmapped Block(boundingBox={1..N}) partitions {1..N} into numLocales contiguous chunks, one per locale. The Cyclic distribution assigns indices round-robin: dmapped Cyclic(startIdx=1) assigns index 1 to locale 0, index 2 to locale 1, ..., index numLocales to locale numLocales-1, then wraps. BlockCyclic combines both by assigning fixed-size blocks in round-robin order. Stencil is a Block distribution with a configurable halo for stencil computations — it prefetches boundary elements from neighboring locales and caches them locally, eliminating the most common remote get pattern in stencil codes. The distribution choice is a significant source of retainer work: teams transitioning from single-locale to multi-locale Chapel programs frequently choose Block for its spatial locality but fail to replicate read-only global data, causing the pattern described above — distributed writes go to the correct locale, but reads from non-distributed data incur remote gets on every iteration.

dom.localSubdomain() returns the subset of a distributed domain dom that is owned by the current locale. It is the key primitive for writing locale-aware loops: coforall loc in Locales { on loc { const localDom = dom.localSubdomain(); forall i in localDom { } } } explicitly migrates computation to each locale and iterates over only the locale-local indices. This pattern has three parts: coforall loc in Locales spawns one task per locale concurrently (coforall = concurrent forall, all tasks run in parallel); on loc { } migrates execution to locale loc (the enclosed code runs on that locale’s threads, with that locale’s memory as local memory); dom.localSubdomain() called inside on loc { } returns the indices owned by loc. Any array access arr[i] inside this pattern where i is from localSubdomain() and arr is distributed with the same domain map is guaranteed to be a local access — no remote gets. The alternative, forall i in dom { } at the top level, is a shorthand that achieves the same distribution of work but requires that every array access inside the loop body is also data-parallel and locale-affine; as soon as any non-distributed data is read, implicit remote gets appear.

The on clause is Chapel’s explicit locale migration primitive. on loc { code } executes code on locale loc. on Locales[2] { writeln(here.id); } prints 2 — here is the current locale, and inside the on clause it is Locales[2]. on arr[i].locale { } migrates execution to the locale that owns arr[i], which is the standard pattern for data-driven locale migration in irregular computations. Chapel also provides here (current locale), here.id (locale index), here.numPUs() (hardware thread count), and numLocales (total locale count). Locale migration has overhead: the on clause involves inter-locale task spawning, which is more expensive than intra-locale task spawning. The general principle for high-performance Chapel programs is to minimize the number of on transitions: one on loc per locale per phase is fine; one on arr[i].locale per array element is too many. Chapel was designed by Bradford Chamberlain at Cray (later HPE), with the Cascade project at University of Washington as the academic foundation. Its retainer work is primarily in distributed HPC systems, data analytics at scale, and scientific computing — teams that have working single-locale Chapel programs and need to scale them to multi-locale execution without rewriting the computation logic. Its closest retainer neighbors are Fortran (HPC numerical) and Julia (numerical, parallel), but Chapel’s distributed domain maps, locale-affine execution, and PGAS (partitioned global address space) memory model make the retainer work distinct in distribution selection, locale affinity diagnosis, and communication profile analysis.

Chapel task parallelism: begin, cobegin, coforall, sync/single variables, and atomic operations

Chapel’s task parallelism model provides three primitives for spawning concurrent tasks. begin { code } spawns a single asynchronous task and returns immediately; the parent task continues in parallel with the spawned task. cobegin { stmt1; stmt2; stmt3; } spawns one task per statement and waits for all to complete; it is the structured parallel composition primitive. coforall i in 0..N-1 { code(i) } spawns one task per iteration value and waits for all to complete; coforall loc in Locales { on loc { } } is the canonical multi-locale pattern. The key difference between forall and coforall is the number of tasks: forall creates a task pool and dispatches iterations to available threads (many iterations per task), while coforall creates exactly one task per iteration value (one task per locale, in the multi-locale pattern). coforall is appropriate when exactly N tasks are needed with N small (like numLocales); forall is appropriate for parallel loops over large iteration spaces.

Chapel’s synchronization primitives are sync and single variables, and atomic types. A sync variable has two states: full and empty. Writing to a sync variable sets it to full; reading from a full sync variable returns the value and sets it back to empty; reading from an empty sync variable blocks until it is written. This is a producer-consumer synchronization: the producer writes, the consumer reads, and the consumer blocks until data is available. A single variable is a write-once sync: it can be written once and read many times; subsequent reads return the same value without blocking. atomic variables support lock-free operations: atomic.add(1), atomic.compareExchange(expected, desired), atomic.fetchAdd(1). These are the primitives for lock-free counters, barriers, and coordination. Chapel retainer work involving synchronization typically involves diagnosing races in begin-spawned tasks (tasks share the enclosing scope by default; variables captured from the enclosing scope are shared references, not copies), restructuring with sync variables for producer-consumer coordination, and replacing shared-counter patterns with atomic variables for lock-free increment.

How HourTab tracks Chapel developer retainer hours

Chapel retainer work carries the invisible-hours problem specific to distributed memory parallel programming: the computation logic may be correct on a single locale, and the distributed version may appear correct (producing the right results) while being slow due to implicit remote gets that are invisible in the source code but dominate runtime on multi-locale runs. The remote get pattern described above — forall loop over distributed domain, reads from non-distributed locale-0 data — is the single most common performance issue in Chapel programs transitioning from single-locale to multi-locale execution. The retainer work is the communication profile analysis (how many remote gets per iteration, which data accesses are non-local), the replication design (which read-only data needs to be replicated per locale, what the replication overhead is), the distribution selection (Block vs Cyclic vs Stencil based on access pattern), and the on loc { localSubdomain() } restructuring that makes the locale affinity explicit. None of these appear in the diff as visible changes to the computation logic — the numbers come out the same, just faster.

HourTab gives Chapel developers a public retainer-hours URL they send to clients — typically organizations running Chapel HPC programs that require ongoing distributed systems engineering, teams scaling numerical simulations from single-locale to multi-locale execution, and projects applying Chapel’s PGAS model to data analytics at scale where remote memory access is the dominant performance cost. For Chapel retainers, each work log entry should name the mechanism (locality: forall loop over distributed domain reading non-distributed data; distribution: Block vs Cyclic selection based on access pattern; replication: read-only global data replicated per locale with var localParams: [Locales.domain] ParamRecord; synchronization: sync variable or atomic or coforall task design), the specific array, distribution, and before/after remote-get count, and the locale-affinity strategy rationale. Chapel retainers are often compared to Fortran developer retainers for the shared HPC numerical computing context, and to Julia developer retainers for the shared parallel and numerical emphasis, but Chapel’s PGAS memory model, distributed domain maps, and locale-affine execution model make the retainer work distinct in distribution selection, locale affinity diagnosis, and communication profile analysis. HourTab’s work log makes the locality analysis, distribution selection, and data replication design visible to clients who would otherwise see only the symptom — multi-locale performance regression — and not understand why the fix required understanding that forall i in dom { arr[i] = compute(i); } is not the same as coforall loc in Locales { on loc { forall i in dom.localSubdomain() { arr[i] = compute(i, localParams); } } } in terms of remote communication, and why choosing the right locale-affinity pattern at each parallel loop site is the work that ensures the Chapel program performs correctly at scale across multiple locales.

FAQ: Chapel developer retainers