Blog › ICP guides

Erlang developer on retainer: OTP gen_server, supervisor trees, BEAM distribution, and Dialyzer on monthly retainer

September 4, 2026 · ~22 min read

A telco infrastructure team had a memory growth problem. An Erlang node that processed 2,000 events per second was growing its heap by 1.2 GB per hour and crashing once every six hours, restarting the entire application tree including services that had nothing to do with event processing. The crash was reported as a supervisor hitting its restart intensity limit — MaxRestarts=10 in MaxTime=60 — which caused the supervisor itself to terminate, propagating the failure up the tree. The root cause was in the event processor gen_server: handle_cast for each incoming event performed a synchronous mnesia:transaction/1 write. At 2,000 events per second against a write throughput of 500 transactions per second, the gen_server’s message queue grew without bound. erlang:process_info(event_processor, message_queue_len) in the observer console returned 47,000.

The fix required two coordinated changes. First, the event processor was redesigned to buffer events in its state as an accumulator list, write to Mnesia in batches on a 100-millisecond timer (erlang:send_after(100, self(), flush_buffer) set in init/1 and reset in each handle_info({flush_buffer, _}, State) clause), and batch all accumulated events in a single mnesia:transaction/1 per flush cycle. At 2,000 events per second, each flush cycle processes 200 events in a single transaction instead of 200 separate transactions. Message queue length stabilized at under 20. Second, the event processor was isolated under its own dedicated supervisor with restart intensity separate from the top-level supervisor, so that event processor crashes triggered only its sub-supervisor rather than the entire application tree.

No new feature shipped across the two sessions. Event processing throughput remained at 2,000 events per second and the output was the same Mnesia records. What changed structurally was the ownership of the backpressure problem: the synchronous transaction-per-event design pushed the backpressure into the gen_server’s mailbox (unbounded, growing to 1.2 GB/hour); the timer-based batch design moved it into the accumulator list (bounded by 200 events per 100ms flush cycle, stable at 8 MB). An Erlang developer on monthly retainer identified the message_queue_len symptom in the observer, traced it to the synchronous write pattern in handle_cast, and redesigned the gen_server’s buffering architecture across eleven hours.

The OTP process design work that retainers fund

Supervisor tree design is the category of Erlang retainer work that generates the most hours with the least visible output. The Erlang OTP supervisor behavior implements the let-it-crash philosophy: a process that encounters an unrecoverable error crashes instead of trying to handle the error internally, and its supervisor restarts it. The supervisor’s restart strategy determines what happens when one child crashes: one_for_one restarts only the crashed child; one_for_all restarts all siblings when any child crashes (appropriate when children share state that becomes inconsistent after one crashes); rest_for_one restarts the crashed child and all children started after it (appropriate for pipeline architectures where downstream workers depend on upstream state).

A supervisor tree with the wrong restart strategy produces cascading failures. A system with a single top-level supervisor using one_for_all where one child is a database connection pool and another is a stateless HTTP handler will restart the HTTP handler on every database connection error — even though the HTTP handler has no shared state with the database pool and could continue processing non-database requests during the outage. A retainer engagement covering supervisor tree redesign begins with a failure mode analysis: for each child in the tree, what happens to siblings if this child crashes? Which siblings share state with the crashed child and must restart? Which are independent and should continue? The answer determines the correct restart strategy and whether the tree needs to be restructured into multiple sub-supervisors, each with isolated restart policies.

gen_server callback design is the second major OTP retainer category. A gen_server with incorrect callback structure accumulates subtle problems over time. A handle_info clause that matches only one message pattern and has no catch-all handle_info(_Msg, State) -> {noreply, State} clause produces a function_clause error when an unexpected message arrives — a DOWN monitor message from a process that terminated, a timeout message from a stale timer, or a message from a caller that crashed before receiving the reply. Dialyzer flags the missing catch-all if the gen_server is instrumented with -spec handle_info(any(), state()) -> gen_server:handle_info_result(state()). Without Dialyzer coverage, the missing clause waits until the unexpected message arrives in production. A retainer engagement covering gen_server callback completeness reviews each callback signature, adds the catch-all clauses, and adds Dialyzer -spec declarations that surface missing clauses as type errors rather than runtime crashes.

gen_statem state machines replace ad-hoc state tracking in gen_server state tuples. A gen_server that has a mode field in its state record and routes messages differently based on mode is implementing a state machine manually: the mode transitions are implicit in handle_call clauses that pattern-match on #state{mode = connecting} versus #state{mode = connected}. gen_statem makes the state explicit — each state is an atom or tuple that appears in the function clause head, transitions are declared with {next_state, NewState, NewData}, and guards can prevent invalid transitions at the gen_statem layer rather than in ad-hoc if expressions. A retainer engagement covering gen_statem migration reads the existing gen_server, maps the implicit state transitions, and rewrites as a gen_statem with the handle_event_function callback mode that centralizes all state-event combinations in one function or the state_functions callback mode that gives each state its own function clause.

BEAM distribution, ETS/Mnesia, and hot code reloading

BEAM distribution is the feature that makes Erlang architecturally distinct from most server-side runtimes. Distributed Erlang connects multiple BEAM nodes into a cluster where processes can send messages to remote processes using the same Pid ! Message syntax used for local processes — the BEAM runtime transparently serializes and routes the message to the remote node. A retainer engagement covering BEAM distribution design begins with node topology: how many nodes, whether they all connect to each other (fully connected mesh with net_kernel:start([NodeName, longnames]) and shared cookie) or form a hub-and-spoke topology where worker nodes connect only to coordinator nodes. The mesh topology requires that every node know about every other node, which works for clusters of 10 to 50 nodes but becomes a coordination bottleneck at hundreds of nodes because the global name registry (global:register_name) must synchronize across all nodes. Hub-and-spoke with local registration and a custom routing layer scales better but requires more design work.

ETS is the BEAM’s in-process in-memory table store. An ETS table is created with ets:new(TableName, Options) where options include the table type (set for unique keys, ordered_set for sorted unique keys, bag for duplicate values under the same key, duplicate_bag for duplicate key-value pairs), access control (public, protected, private), and concurrency tuning (read_concurrency which uses a read-write lock that favors concurrent reads, and write_concurrency which uses fine-grained locking on table segments). A retainer engagement covering ETS design audits whether ets:lookup is being used for filtered searches that should use ets:select with a match specification — ets:lookup(Table, Key) returns all records matching a single key; ets:select(Table, MatchSpec) filters the entire table based on a match specification compiled with ets:fun2ms/1 that avoids copying non-matching records to the caller process. For large ETS tables where lookup patterns are complex, the difference in memory allocation between repeated ets:lookup and ets:select is measurable with erlang:memory(processes).

Hot code upgrades are the mechanism by which Erlang systems apply code changes to a running production node without restarting the VM or dropping connections. A retainer engagement covering hot code upgrades involves writing the .appup file that declares which module versions can be upgraded from (a list of version patterns and their upgrade and downgrade instructions), running systools:make_relup/3 to generate the relup file from the current and new release specifications, and testing the upgrade path in a staging cluster with release_handler:install_release/1. The critical path in a hot code upgrade is the gen_server:code_change/3 callback: this function is called on the running gen_server process when its module is upgraded, receiving the old state term and a OldVsn identifier. If the state record added a field between versions, code_change/3 must construct the new state record from the old one, handling the missing field with a default value. A missing or incorrect code_change/3 callback causes the gen_server to crash during the upgrade window — which is acceptable only if the supervisor will restart it cleanly with the new module loaded.

Dialyzer, typespecs, and property-based testing

Dialyzer is a static analysis tool that performs success typing analysis on Erlang code. Unlike traditional type checkers that require explicit type annotations to typecheck, Dialyzer infers types from actual usage and reports discrepancies where a function is called with arguments that the function can never successfully handle based on its implementation. A retainer engagement covering Dialyzer adoption begins with building the PLT (persistent lookup table) for the OTP standard library and project dependencies: dialyzer --build_plt --apps kernel stdlib erts mnesia ssl crypto creates the baseline PLT that Dialyzer uses to infer types for OTP functions. Adding -spec declarations to exported functions improves Dialyzer’s analysis: a function declared as -spec handle_event(event(), state()) -> {ok, state()} | {error, term()} gives Dialyzer the declared return type so that callers that only handle {ok, State} and ignore {error, _} are flagged as having a missing pattern match.

PropEr property-based testing complements Dialyzer by finding inputs that violate behavioral invariants. A property test generates random inputs using PropEr’s type generator combinators (proper_types:list/1, proper_types:integer/0, proper_types:oneof/1, proper_types:frequency/1) and verifies that the property holds for each generated input. When a property fails, PropEr shrinks the failing input to the smallest case that still fails — a failing list of 500 elements shrinks to the specific 2-element list that demonstrates the violation. A retainer engagement covering PropEr for an OTP system typically writes properties for: gen_server invariants (after any sequence of valid handle_call messages, the state satisfies a validity predicate); ETS table consistency (after a series of inserts and deletes, the table size is always equal to the number of unique keys inserted minus the number of deleted keys); and Mnesia transaction properties (a transaction that reads a value, transforms it, and writes it back always produces a result consistent with the initial value, even when the transaction retries due to conflict).

How HourTab tracks Erlang developer retainer hours

Erlang and OTP retainers present the hour-visibility problem in its purest form. A gen_server redesign session that eliminates message queue growth involves reading the observer dashboard, querying erlang:process_info(Pid, message_queue_len), tracing the queue growth to the synchronous mnesia:transaction call in handle_cast, designing the timer-based flush buffer, updating the gen_server init/1, handle_cast, and handle_info callbacks, and running a load test to verify the queue length stabilizes. The visible output of that session is changes to three callback functions in one Erlang module. A time log entry that says “gen_server refactor, 6h” accurately describes the session duration and tells the client nothing about why six hours were needed or what was eliminated.

HourTab gives Erlang developers a public retainer-hours URL they paste into the first message of every client Slack thread. The client opens the URL and sees the current burn-down without asking. For Erlang retainers specifically, the work log format is the evidence of value: each entry should name the gen_server or supervisor module, the observer or erlang:process_info metric that identified the problem, the OTP design pattern applied (timer-based flush buffer for message batching; dedicated sub-supervisor for fault isolation; gen_statem state machine for explicit state transitions; Dialyzer -spec for type coverage), and the before/after metric (message_queue_len: 47,000 → <20; heap memory: growing at 1.2 GB/hr → stable at 8 MB; Dialyzer warnings: 23 → 0; supervisor crash frequency: 3/hr → 0). That entry is four structured fields that take four minutes to write and make the next client check-in a reading of the metrics rather than an explanation of the BEAM scheduler.

The retainer model fits Erlang platform engineering because OTP system correctness is not a one-time achievement — it is maintained against ongoing changes. Mnesia schema evolution requires transform_table operations that must be designed before the schema change is merged. Hot code upgrade paths must be written and tested before each production release. Dialyzer PLT updates are needed when OTP versions upgrade. Supervisor tree design must be revisited when new services are added. A project contract closes when the feature ships. An Erlang retainer stays open for the next Mnesia migration, the next .appup file, and the next message queue growth that only erlang:process_info can diagnose.

FAQ: Erlang developer retainers