Blog › ICP guides

LFE developer on retainer: OTP gen_server pattern matching, LFE macro system, BEAM concurrency, Erlang interop, and Lisp Flavoured Erlang on monthly retainer

November 21, 2026 · ~15 min read

An LFE system implementing a distributed key-value cache was returning undefined for four keys per request batch. The system used an OTP gen_server behaviour to manage a process-local ETS table. The handle_call/3 clause responsible for key lookup was written in LFE as (defun handle_call ([(tuple 'get key) _ state] ...)). In Erlang and LFE, the tuple pattern (tuple 'get key) where 'get is the atom get and key is a free pattern variable matches any two-element tuple whose first element is the atom get and second element is anything. The developer expected the server to respond to lookup requests {get, Key} for any value of Key, including both atoms and lists (strings in Erlang/LFE). The request sender was encoding keys as lists: (tuple 'get "user_session_token") where "user_session_token" is a list of character integers, not an atom. The handle_call clause matched these requests correctly because key was a free variable that bound to any term. The bug was subtler: the ETS table lookup was performed with (ets:lookup table-name key), and the ETS keys stored in the table were atoms of the form user_session_token (without quotes, the atom form). The lookup with a list key against an atom key in ETS returned an empty list (no match), and the handle_call clause returned (tuple 'reply 'undefined state) on empty lookup. Four different keys per batch were affected — four calls passing list keys against atom ETS keys, four undefined responses. The LFE developer on retainer diagnosed the type mismatch between request key representation (lists) and ETS table key representation (atoms), audited the full pipeline from key encoding at the request site to key storage at the table insertion site, and restructured the server to normalize incoming keys to atoms using (list_to_atom key) at the handle_call entry point before ETS lookup. Missing keys per request batch: 4 → 0.

The work log entry read “fixed cache miss for string keys, 11h.” It names the symptom and duration. It cannot explain to a client why Erlang and LFE use two distinct representations for string-like data — atoms are unique global constants interned in an atom table (the BEAM VM limits atoms to approximately one million by default; atoms are never garbage collected, so dynamic atom creation from untrusted input is a resource exhaustion risk); lists are ordinary linked lists of character integer codes, not pointers to the atom table; and the two representations are never equal under == or pattern matching even when they spell the same characters. It cannot explain why the gen_server handle_call pattern matched successfully (because the pattern variable key in the clause head bound to the list without needing a type constraint) but the ETS lookup failed (because ETS comparison uses Erlang’s structural equality, where user_session_token the atom and "user_session_token" the list are structurally different terms). It cannot explain the retainer’s key decision: normalizing at the server boundary rather than at each call site, and why normalizing to atoms requires a guard on the incoming value to avoid the atom table exhaustion risk from untrusted input (normalizing bounded internal keys to atoms is safe; normalizing arbitrary user-supplied strings to atoms is a DoS vulnerability). The 11 hours of key representation audit, pipeline trace from call site to ETS table, and normalization strategy selection — atom vs binary vs explicit type dispatch — are invisible beyond the added normalization call.

LFE OTP integration: gen_server pattern matching, clause guards, and handle_call design

LFE runs on the BEAM virtual machine and compiles to the same bytecode format as Erlang. Every OTP behaviour available in Erlang — gen_server, gen_statem, gen_event, supervisor — is available in LFE with identical semantics. The difference is syntax: LFE uses a Lisp-2 syntax derived from Common Lisp, with S-expressions, quasiquotation, and a hygienic macro system layered on top of the BEAM. A gen_server module in LFE declares its behaviour with (behaviour 'gen_server) and implements the required callbacks using LFE’s defun with pattern-matching clause heads.

LFE’s defun supports multiple clause heads with different patterns. The handle_call/3 callback dispatches on the request message, the caller reference (a PID/Tag pair), and the server state. Each clause head pattern matches against the request tuple, and LFE uses first-match semantics: the first clause whose pattern matches the incoming message executes. Guards in LFE clause heads use the (when ...) form: (defun handle_call ([(tuple 'get key) _from state] (when (is_atom key)) ...)) restricts this clause to requests where the key is an atom. Without a guard, the free pattern variable key matches any term, including lists, binaries, tuples, and integers. Retainer pattern: when a gen_server exhibits inconsistent response behavior for requests that appear structurally identical, add explicit type guards to each clause head that constrain the types of free pattern variables; the type guard documents the expected protocol and prevents silent fall-through to a default clause on type-mismatched messages.

The default handle_call clause — the catch-all that executes when no earlier clause matches — is a common source of silent failures. An LFE gen_server that returns (tuple 'reply 'undefined state) from its default clause will respond to every unmatched message with undefined rather than crashing. This is sometimes correct (graceful degradation for unknown messages), but it masks type mismatch bugs: a key lookup that silently returns undefined because the key type didn’t match the clause guard looks identical at the caller to a successful lookup for a key that genuinely maps to the value undefined. Retainer pattern: prefer a default clause that logs the unmatched message and returns an error tuple (tuple 'reply (tuple 'error 'unmatched_request) state) during development; this converts silent type-mismatch falls into detectable protocol errors.

LFE macro system: defmacro, quasiquote, hygiene, and compile-time code generation

LFE’s macro system is its most distinctive feature relative to Erlang. LFE macros are compile-time transformations: a defmacro form defines a function from syntax (S-expressions) to syntax, executed at compile time before the resulting code is compiled to BEAM bytecode. This enables building embedded domain-specific languages in LFE code: a macro can generate complex gen_server boilerplate, synthesize OTP supervision tree declarations, or produce repetitive pattern matching logic from a concise declarative input.

Quasiquotation is the primary tool for constructing macro output: the backquote character ` starts a quasiquoted expression in which most subforms are treated as literal data; a comma , unquotes a subform (inserting the value of the macro argument at that position); and ,@ splices a list into the surrounding list. For example, a macro that generates a gen_server:call wrapper for a specific request type can be written as (defmacro lookup-key (server key) `(gen_server:call ,server (tuple 'get ,key))). At every call site where (lookup-key my-server my-key) appears, the macro expands to the full gen_server call expression before compilation.

Macro hygiene is the source of a class of subtle LFE retainer bugs. An unhygienic macro introduces a binding (a (let ...)) or a (defun ...) inside its expansion) that uses a name which could collide with a binding in the calling code. In LFE, macros are not automatically hygienic (unlike in Racket or Scheme’s syntax-rules): if a macro introduces a local binding named result and the call site also has a binding named result, the macro’s expansion may shadow or be shadowed by the call-site binding, producing wrong values. The Erlang/LFE runtime will not report a type error or exception; the wrong binding will simply evaluate to a different value. The retainer pattern: use gensyms (generated unique symbols) for all internal bindings introduced by macros; LFE provides (gensym) for this purpose. Any macro that introduces a binding that is not intended to be visible at the call site should use a gensym name.

BEAM concurrency, process design, and LFE-Erlang interoperability

LFE shares the BEAM VM’s concurrency model with Erlang, Elixir, and Gleam: processes are lightweight (stack starts at approximately 2 KB), communication is by message passing with no shared memory, and each process has an independent mailbox. LFE processes are spawned with (spawn #'my-module:entry-function/0) or the equivalent form, and message passing uses (! pid message) (the send operator). A process receives messages with a (receive ...) form that pattern-matches against the mailbox contents and selects the first matching message.

LFE modules compile to Erlang modules at the BEAM level. Calling an Erlang function from LFE uses the same module:function syntax as in Erlang: (erlang:now), (ets:lookup table key), (lists:foldl fun acc list). Calling an LFE function from Erlang requires the LFE module to be compiled first, after which it is a standard BEAM module and callable as lfe_module:function(args) from Erlang code. The only LFE-specific consideration for Erlang callers is that LFE macros are expanded at LFE compile time and are invisible at the Erlang call boundary: an Erlang module calling into an LFE module calls the compiled function, not the macro, and the macro expansion is not re-run.

ETS (Erlang Term Storage) tables are used from LFE exactly as from Erlang: (ets:new table-name options), (ets:insert table-name (tuple key value)), (ets:lookup table-name key). The critical property for LFE retainer work: ETS key lookup uses Erlang structural equality, which distinguishes atoms, lists, binaries, integers, and tuples as distinct types with distinct equality semantics. A retainer audit of an ETS-backed gen_server should always verify that the type of keys at insertion and the type of keys at lookup are identical — this is the most common source of ETS-based silent misses in LFE codebases where the key pipeline crosses a serialization or parsing boundary (network input, file read, configuration load) that may introduce type coercions.

Robert Virding designed LFE starting around 2007 as an experiment in embedding a Lisp on top of the BEAM VM without modifying the VM itself. LFE is available on GitHub and installable via rebar3 or mix (in the Elixir/Mix ecosystem). LFE’s closest relatives in the BEAM ecosystem are Erlang (same semantics, different syntax), Elixir (similar BEAM foundation with a Ruby-influenced syntax and a more developed macro system via Elixir’s defmacro), and Gleam (static typing on BEAM). LFE retainer work is distinct from all three: the Lisp syntax and macro system make LFE attractive for teams who want compile-time code generation beyond what Erlang’s parse transforms or Elixir’s macros provide, but the BEAM concurrency model and OTP behaviour semantics are shared across all.

How HourTab tracks LFE developer retainer hours

LFE retainer work shares the invisible-work problem of all BEAM-ecosystem retainers, compounded by the gap between the Lisp syntax and the Erlang semantics that drives most LFE retainer bugs. The gen_server pattern-match miss described above is a diff with a single (list_to_atom key) normalization call added at the handle_call entry point; the value is correct ETS lookups for all key types that the server is expected to receive, a key-representation audit documenting where in the pipeline the type divergence occurred, and a normalization strategy that handles the atom table exhaustion risk for externally-sourced keys. The 11 hours to diagnose, audit, and engineer the normalization correctly are invisible in the diff.

HourTab gives LFE developers a public retainer-hours URL they send to clients — typically distributed systems teams using LFE for its macro DSL capabilities on BEAM infrastructure, organizations running Erlang/OTP systems that adopted LFE for new service components, and teams maintaining LFE codebases where OTP upgrade cycles require gen_server state shape migrations and code_change/3 implementations. For LFE retainers, each work log entry should name the mechanism (gen_server handle_call pattern match repair; gen_server state shape migration with code_change; gen_statem state machine clause design; supervisor strategy selection; LFE macro quasiquote and hygiene design; gensym usage for internal binding names; Erlang module interop; ETS key type normalization; BEAM process spawn and mailbox design; receive clause pattern design), the specific behaviours, clause patterns, message types, macro expansions, and ETS tables involved in the bug, and the before/after metric. LFE retainers are often compared to Erlang developer retainers for the shared BEAM and OTP infrastructure and to Elixir developer retainers for the shared macro system patterns, but LFE’s Lisp-2 syntax and compile-time macro evaluation discipline make the retainer work distinct in DSL design and hygiene engineering. HourTab’s work log makes the key-type audit, normalization strategy selection, and ETS key pipeline trace visible to clients who would otherwise see only the symptom — four undefined responses per request batch — and not understand why the fix required understanding Erlang’s atom/list distinction, auditing every key encoding and storage site in the pipeline, and choosing a normalization boundary that avoided the atom table exhaustion risk.

FAQ: LFE developer retainers