Blog › ICP guides
Elixir developer on retainer: OTP architecture, Phoenix LiveView, Ecto, and distributed systems on monthly retainer
August 21, 2026 · ~22 min read
A real-time collaboration platform built on Phoenix had a GenServer leak. The application created a SessionServer GenServer for every WebSocket connection through a DynamicSupervisor, registered each process in a Registry keyed by socket ID, and stored per-session state in each process’s heap. Over 72 hours in production, the BEAM VM process count climbed from approximately 12,000 at restart to 847,000 — each process holding 18 KB of heap — and total VM memory reached 14 GB on a node provisioned for 4 GB. The supervision tree child spec had restart: :permanent, so a crashing session process was immediately restarted rather than terminated. Socket disconnects sent a :DOWN monitor message to a coordinator that was missing the handle_info/2 clause, so the message was silently dropped into the process mailbox and session processes accumulated indefinitely.
A fractional Elixir architect on monthly retainer connected to the production node with bin/app rpc and ran :recon.proc_count(:memory, 20) to identify the top 20 processes by memory consumption. The top 847,000 entries were all SessionServer processes. A follow-up :recon_trace.calls({SessionServer, :handle_info, 2}, 100) trace showed the :DOWN clause missing from the match list. The fix required changing the child spec to restart: :temporary, adding the missing handle_info({:DOWN, ref, :process, _pid, _reason}, state) clause, and implementing terminate/2 to flush pending ETS writes before shutdown. Process count at steady state dropped to 12,400. VM memory dropped to 1.8 GB. No user-visible feature changed.
The second month’s work focused on Phoenix LiveView performance: a document list LiveView was calling Enum.map/2 inside the HEEx template to render 2,000 documents, re-rendering the entire list DOM on every event — a collaborator joining, a cursor position update, a document rename. The fix migrated the list to LiveView streams (stream/4, phx-update="stream"), reducing the diff payload from a full 2,000-item re-render to a single-item insert or update on each event. The third month addressed Ecto N+1 elimination: a GraphQL resolver was calling Repo.get/2 once per document to load the document’s owner profile, executing 2,000 individual queries for a single page request. Integrating Dataloader with Dataloader.Ecto.new/2 collapsed those 2,000 queries into one batched IN clause.
Elixir developers, OTP architects, and Phoenix consultants on monthly retainer — fractional Elixir engineers, BEAM platform advisors, and Phoenix LiveView performance consultants — do their highest-value work in the OTP supervision architecture, LiveView stream optimization, Ecto query design, and distributed node coordination that produces the reliable, scalable platform the engineering director defends to the CTO. This guide covers OTP architecture in depth, Elixir language features, Phoenix LiveView, Ecto patterns, distributed Elixir with libcluster, and production observability — and how to structure an Elixir developer retainer that makes the hours behind each optimization visible.
OTP architecture
OTP — the Open Telecom Platform — is the set of Erlang/Elixir libraries, design principles, and supervision behaviours that make BEAM applications fault-tolerant. An Elixir architect on retainer designs and maintains the supervision tree: the hierarchy of supervisors and workers that determines how the system restarts when individual processes crash, how processes find each other by name, and how transient work is supervised without leaking processes.
GenServer lifecycle
GenServer is the foundational OTP behaviour. Every GenServer module implements a set of callbacks that the OTP framework calls at well-defined lifecycle events. Getting these callbacks right — especially terminate/2, handle_info/2, and format_status/2 — is the work that prevents process leaks, runaway mailboxes, and sensitive state appearing in :observer.
defmodule MyApp.SessionServer do
use GenServer, restart: :temporary # do not restart on crash — supervisor does not revive
# --- Lifecycle callbacks ---
# init/1 return variants:
# {:ok, state} — start normally
# {:ok, state, timeout} — send :timeout to handle_info after N ms of inactivity
# {:ok, state, :hibernate} — hibernate immediately (reduces memory if infrequently used)
# {:stop, reason} — abort startup, supervisor considers it failed
def init({socket_id, user_id}) do
# Monitor the calling process (usually a LiveView or Channel pid):
Process.monitor(self())
# Trap exits so terminate/2 is called on linked process death:
Process.flag(:trap_exit, true)
state = %{socket_id: socket_id, user_id: user_id, buffer: [], timer: nil}
{:ok, state, {:continue, :load_initial_state}}
end
# handle_continue/2 — runs immediately after init, before any other messages:
def handle_continue(:load_initial_state, state) do
initial = MyApp.Repo.get(MyApp.Session, state.socket_id)
{:noreply, %{state | buffer: initial && initial.pending_ops || []}}
end
# handle_call/3 — synchronous request; caller blocks until {:reply, result, state}:
def handle_call({:get_buffer}, _from, state) do
{:reply, state.buffer, state}
end
def handle_call({:push_op, op}, _from, state) do
new_buffer = [op | state.buffer]
{:reply, :ok, %{state | buffer: new_buffer}}
# Other return forms:
# {:reply, result, state, timeout} — set inactivity timeout
# {:noreply, state} — reply later with GenServer.reply/2
# {:stop, reason, reply, state} — reply then stop
end
# handle_cast/2 — async, fire-and-forget; no reply:
def handle_cast({:flush}, state) do
Enum.each(state.buffer, &persist_op/1)
{:noreply, %{state | buffer: []}}
# {:noreply, state, :hibernate} — hibernate after processing
# {:stop, :normal, state} — stop gracefully
end
# handle_info/2 — non-OTP messages: :EXIT, monitor :DOWN, timer ticks, raw sends:
def handle_info({:DOWN, _ref, :process, _pid, _reason}, state) do
# The monitored process died — clean up and stop:
{:stop, :normal, state}
end
def handle_info(:timeout, state) do
# Inactivity timeout fired — flush buffer and hibernate:
Enum.each(state.buffer, &persist_op/1)
{:noreply, %{state | buffer: []}, :hibernate}
end
def handle_info({:EXIT, _pid, reason}, state) do
# Linked process exited — propagate or handle:
{:stop, reason, state}
end
# terminate/2 — called when the GenServer stops; flush state before exit:
# Guaranteed only when Process.flag(:trap_exit, true) is set, or when
# the supervisor shuts down the process (not on :kill).
def terminate(_reason, state) do
Enum.each(state.buffer, &persist_op/1)
:ok
end
# code_change/3 — called during hot code upgrades via :sys.change_code:
def code_change(_old_vsn, state, _extra) do
# Migrate state format between versions:
{:ok, state}
end
# format_status/2 — called by :observer and :sys.get_status; hide sensitive fields:
def format_status(_reason, [_pdict, state]) do
{:ok, %{state | buffer: "[#{length(state.buffer)} ops, redacted]"}}
end
defp persist_op(op), do: MyApp.Repo.insert(op, on_conflict: :nothing)
end
Supervisor strategies
Supervisor restart strategies determine how the supervisor responds when a child process crashes. Choosing the wrong strategy is the most common OTP architecture mistake: one_for_all applied to loosely-coupled workers causes cascading restarts on any single worker crash.
defmodule MyApp.Application do
use Application
def start(_type, _args) do
children = [
MyApp.Repo,
{Phoenix.PubSub, name: MyApp.PubSub},
MyAppWeb.Endpoint,
# DynamicSupervisor for per-session processes (see below):
{DynamicSupervisor, name: MyApp.SessionSupervisor, strategy: :one_for_one},
# Registry for named process lookup:
{Registry, keys: :unique, name: MyApp.SessionRegistry}
]
# Supervisor strategies:
# :one_for_one — restart only the failed child (default; use for independent workers)
# :one_for_all — restart ALL children when any one fails (use for tightly coupled)
# :rest_for_one — restart failed child + all children started after it (dependency order)
opts = [
strategy: :one_for_one,
# max_restarts: 3 children may restart within max_seconds: 5 seconds.
# If exceeded, the supervisor itself crashes (propagating up the tree):
max_restarts: 3,
max_seconds: 5
]
Supervisor.start_link(children, opts)
end
end
# child_spec/1 — defines how the supervisor starts, restarts, and shuts down a child.
# The full child spec map:
%{
id: MyApp.Worker, # unique identifier within the supervisor
start: {MyApp.Worker, :start_link, [arg]}, # {module, function, args}
restart: :permanent, # :permanent (always restart), :temporary (never), :transient (on abnormal exit)
shutdown: 5_000, # ms to wait for terminate/2 before sending :kill; or :brutal_kill; or :infinity
type: :worker # :worker or :supervisor
}
# Most modules implement child_spec/1 via `use GenServer` or `use Supervisor`.
# Override with a custom spec:
def child_spec(arg) do
%{
id: {__MODULE__, arg}, # unique ID per instance (important for DynamicSupervisor)
start: {__MODULE__, :start_link, [arg]},
restart: :temporary,
shutdown: 10_000
}
end
DynamicSupervisor and Registry
DynamicSupervisor starts children at runtime rather than at application boot — the right tool for per-session, per-request, or per-tenant processes. Combined with Registry, it provides named process lookup without a global atom table and enables broadcasting to all processes registered under a shared key.
# In Application.start/2, add to children list:
{DynamicSupervisor, name: MyApp.SessionSupervisor, strategy: :one_for_one,
max_children: 50_000} # hard limit on concurrent sessions
{Registry, keys: :unique, name: MyApp.SessionRegistry}
# Start a named child process at runtime:
defmodule MyApp.Sessions do
alias MyApp.{SessionSupervisor, SessionRegistry, SessionServer}
def start_session(socket_id, user_id) do
child_spec = {SessionServer, {socket_id, user_id}}
DynamicSupervisor.start_child(SessionSupervisor, child_spec)
# Returns {:ok, pid} | {:error, {:already_started, pid}} | {:error, :max_children}
end
def stop_session(socket_id) do
case Registry.lookup(SessionRegistry, socket_id) do
[{pid, _meta}] -> DynamicSupervisor.terminate_child(SessionSupervisor, pid)
[] -> {:error, :not_found}
end
end
def get_session_pid(socket_id) do
case Registry.lookup(SessionRegistry, socket_id) do
[{pid, _meta}] -> {:ok, pid}
[] -> {:error, :not_found}
end
end
end
# In SessionServer, register on start using {:via, Registry, ...}:
defmodule MyApp.SessionServer do
use GenServer, restart: :temporary
def start_link({socket_id, user_id}) do
# {:via, Registry, {registry_name, key}} — GenServer registers itself in the Registry.
# Name lookup via Registry.lookup(MyApp.SessionRegistry, socket_id).
GenServer.start_link(__MODULE__, {socket_id, user_id},
name: {:via, Registry, {MyApp.SessionRegistry, socket_id}})
end
# ...callbacks as above...
end
# Registry.dispatch/3 — broadcast a message to ALL processes registered under a key.
# Useful when multiple processes share a room/tenant key (Registry keys: :duplicate):
{Registry, keys: :duplicate, name: MyApp.RoomRegistry}
# Register multiple processes under the same room key:
Registry.register(MyApp.RoomRegistry, "room:#{room_id}", %{user_id: user_id})
# Broadcast to all members of a room:
Registry.dispatch(MyApp.RoomRegistry, "room:#{room_id}", fn entries ->
for {pid, _meta} <- entries do
send(pid, {:room_event, event})
end
end)
Task and Task.Supervisor
# Task.async/1 + Task.await/2 — parallel work with result collection:
def fetch_dashboard_data(user_id) do
orders_task = Task.async(fn -> MyApp.Orders.list_for_user(user_id) end)
invoices_task = Task.async(fn -> MyApp.Invoices.list_for_user(user_id) end)
activity_task = Task.async(fn -> MyApp.Activity.recent(user_id, limit: 20) end)
# Task.await/2 — blocks until result, raises after timeout_ms (default 5000):
%{
orders: Task.await(orders_task, 3_000),
invoices: Task.await(invoices_task, 3_000),
activity: Task.await(activity_task, 3_000)
}
end
# Task.Supervisor.async_nolink/3 — supervised fire-and-forget; crash does not kill caller:
{Task.Supervisor, name: MyApp.TaskSupervisor}
Task.Supervisor.async_nolink(MyApp.TaskSupervisor, fn ->
MyApp.Mailer.send_welcome_email(user)
end)
# The Task.Supervisor handles restart and isolation — caller is unaffected if task crashes.
# Task.Supervisor.async_stream/4 — parallel enumeration with back-pressure:
# max_concurrency limits concurrent tasks; ordered: false returns results as they complete.
def process_batch(ids) do
ids
|> Task.Supervisor.async_stream(
MyApp.TaskSupervisor,
fn id -> MyApp.Processor.run(id) end,
max_concurrency: 10, # at most 10 concurrent tasks
timeout: 30_000, # 30 s per task
ordered: false, # yield results as completed (faster for variable-duration tasks)
on_timeout: :kill_task # kill timed-out tasks; :exit kills the stream process
)
|> Enum.reduce({[], []}, fn
{:ok, result}, {ok, err} -> {[result | ok], err}
{:exit, reason}, {ok, err} -> {ok, [{:error, reason} | err]}
end)
end
Elixir language depth
Elixir’s functional core — immutable data, pattern matching, and pipe-oriented composition — is what makes the OTP concurrency model tractable. An Elixir architect on retainer spends time on language-level design: replacing nested case expressions with with macros, defining protocols for polymorphic dispatch, writing macros to eliminate cross-cutting boilerplate, and designing guard-based function heads that make control flow explicit and exhaustively matched.
Pattern matching and guards
# Multiple function heads — each clause matched in definition order:
defmodule MyApp.Pricing do
# Guard clauses restrict when a clause matches:
def discount(%{tier: :premium, spend: spend}) when spend >= 1_000, do: 0.20
def discount(%{tier: :premium}), do: 0.10
def discount(%{tier: :standard, spend: spend}) when spend >= 500, do: 0.05
def discount(_customer), do: 0.0
# defguard — reusable guard that can be used in guard expressions:
defguard is_high_value(spend) when is_number(spend) and spend >= 10_000
def priority(%{spend: spend}) when is_high_value(spend), do: :vip
def priority(_), do: :standard
end
# Pattern matching on maps, lists, structs, binaries:
defmodule MyApp.EventParser do
# Map pattern — matches any map containing these keys (extras are ignored):
def parse(%{"type" => "click", "x" => x, "y" => y}) when is_number(x) and is_number(y) do
{:click, x, y}
end
# Struct pattern — only matches %User{} structs:
def display(%MyApp.User{name: name, role: :admin}), do: "Admin: #{name}"
def display(%MyApp.User{name: name}), do: "User: #{name}"
# List head/tail pattern:
def first([head | _tail]), do: {:ok, head}
def first([]), do: {:error, :empty}
# Binary/bitstring pattern for protocol parsing:
def parse_frame(<<0x01, length::16, payload::binary-size(length), _rest::binary>>) do
{:ok, payload}
end
def parse_frame(<<0x02, _::binary>>), do: {:error, :unsupported_frame_type}
def parse_frame(_), do: {:error, :malformed}
end
with macro and composable error handling
# with chains <- pattern matches; stops on first mismatch and executes else:
defmodule MyApp.Orders do
def create(params, user) do
with {:ok, validated} <- validate_params(params),
{:ok, authorized} <- authorize(user, :create_order),
{:ok, inventory} <- check_inventory(validated.items),
{:ok, order} <- insert_order(validated, inventory),
{:ok, _event} <- publish_event(:order_created, order) do
{:ok, order}
else
{:error, %Ecto.Changeset{} = cs} -> {:error, format_changeset_errors(cs)}
{:error, :unauthorized} -> {:error, "Insufficient permissions"}
{:error, {:out_of_stock, item}} -> {:error, "#{item.name} is out of stock"}
{:error, reason} -> {:error, inspect(reason)}
end
end
# Replacing nested case with with — before:
def create_nested(params, user) do
case validate_params(params) do
{:ok, validated} ->
case authorize(user, :create_order) do
{:ok, _} ->
case insert_order(validated) do
{:ok, order} -> {:ok, order}
{:error, reason} -> {:error, reason}
end
{:error, reason} -> {:error, reason}
end
{:error, reason} -> {:error, reason}
end
end
# The with version is flat, readable, and handles all error paths in one else block.
# Custom guard used in with:
defguardp is_valid_id(id) when is_binary(id) and byte_size(id) == 36
def get(id) when is_valid_id(id) do
with {:ok, order} <- fetch_from_db(id),
true <- order.active? || {:error, :archived} do
{:ok, order}
end
end
def get(_), do: {:error, :invalid_id}
end
Pipe operator, comprehensions, and protocols
# Pipe operator — threads value as first argument:
def process_orders(user_id) do
user_id
|> MyApp.Orders.list_for_user()
|> Enum.filter(&(&1.status == :pending))
|> Enum.map(&enrich_order/1)
|> Enum.sort_by(& &1.created_at, :desc)
|> Enum.take(20)
end
# & capture operator — anonymous function shorthand:
# &Module.fun/arity creates a capture; &(&1 + 1) is an inline anonymous function.
# Partial application with fixed second arg:
format_with_prefix = &String.pad_leading(&1, 10, "0")
transform = &MyApp.Formatter.format(&1, timezone: "UTC")
# Comprehensions — generators, filters, into:, reduce:
def active_user_emails(users) do
for user <- users,
user.active?, # filter: skip inactive users
{:ok, email} = parse_email(user), # pattern match in generator
uniq: true, # deduplicate results
into: MapSet.new() do # collect into a MapSet instead of list
email
end
end
# reduce: option for accumulation (replaces Enum.reduce in many cases):
def group_by_status(orders) do
for order <- orders, reduce: %{pending: [], shipped: [], delivered: []} do
acc -> Map.update!(acc, order.status, &[order | &1])
end
end
# Protocols — polymorphic dispatch on data type:
defprotocol MyApp.Serializable do
@doc "Serialize a value to a map suitable for JSON encoding"
def to_map(value)
end
defimpl MyApp.Serializable, for: MyApp.Order do
def to_map(%MyApp.Order{} = order) do
%{id: order.id, status: order.status, total: Decimal.to_string(order.total)}
end
end
# @fallback_to_any true — provide a default implementation for all unimplemented types:
defimpl MyApp.Serializable, for: Any do
def to_map(value), do: %{value: inspect(value)}
end
# Behaviours — compile-time interface enforcement (different from Protocols):
defmodule MyApp.PaymentGateway do
@callback charge(amount :: Decimal.t(), token :: String.t()) ::
{:ok, String.t()} | {:error, String.t()}
@callback refund(charge_id :: String.t(), amount :: Decimal.t()) ::
{:ok, String.t()} | {:error, String.t()}
@optional_callbacks [refund: 2]
end
defmodule MyApp.StripeGateway do
@behaviour MyApp.PaymentGateway
@impl true
def charge(amount, token) do
# Stripe API call...
{:ok, "ch_stripe_#{:rand.uniform(999_999)}"}
end
@impl true # @impl true causes compile warning if callback signature is wrong
def refund(charge_id, amount) do
# Stripe refund API call...
{:ok, "re_stripe_#{charge_id}"}
end
end
Macros and metaprogramming
# quote/2 produces AST; unquote/1 injects values into the quoted AST:
defmodule MyApp.Validators do
defmacro validate_presence(field) do
quote do
def unquote(:"validate_#{field}")(changeset) do
Ecto.Changeset.validate_required(changeset, [unquote(field)])
end
end
end
end
# __using__/1 — executed when a module calls `use MyApp.Resource`:
defmodule MyApp.Resource do
defmacro __using__(opts) do
schema = Keyword.fetch!(opts, :schema)
quote do
import Ecto.Query
alias MyApp.Repo
alias unquote(schema)
def list, do: Repo.all(unquote(schema))
def get!(id), do: Repo.get!(unquote(schema), id)
defoverridable list: 0, get!: 1
end
end
end
defmodule MyApp.OrderContext do
use MyApp.Resource, schema: MyApp.Order
# Inherits list/0 and get!/1; can override with defoverridable.
end
# __before_compile__/1 — hook that fires at the END of the using module's compilation:
defmodule MyApp.Pluggable do
defmacro __using__(_opts) do
quote do
@plugs []
import MyApp.Pluggable, only: [plug: 1]
@before_compile MyApp.Pluggable
end
end
defmacro plug(name) do
quote do
@plugs [unquote(name) | @plugs]
end
end
defmacro __before_compile__(_env) do
quote do
def __plugs__, do: Enum.reverse(@plugs)
end
end
end
Phoenix framework and LiveView
Phoenix LiveView is the highest-ROI Phoenix feature for real-time UIs: server-rendered HTML that updates over a persistent WebSocket connection, with stateful client-server synchronization managed by the BEAM. A Phoenix consultant on retainer maintains the LiveView architecture: designing the mount/3 initialization, optimizing list rendering with streams, integrating PubSub for real-time updates, and using Phoenix.Presence for user awareness.
LiveView lifecycle
defmodule MyAppWeb.DocumentLive do
use MyAppWeb, :live_view
# mount/3 — called once on static render (connected? false) and once on WebSocket connect.
# connected?/1 check prevents PubSub subscriptions on the static render pass:
def mount(%{"room_id" => room_id}, _session, socket) do
if connected?(socket) do
Phoenix.PubSub.subscribe(MyApp.PubSub, "room:#{room_id}")
Phoenix.Presence.track(socket, socket.assigns.current_user.id, %{
name: socket.assigns.current_user.name,
joined_at: System.system_time(:second)
})
end
documents = MyApp.Documents.list_for_room(room_id)
socket =
socket
|> assign(:room_id, room_id)
|> assign(:loading, false)
|> stream(:documents, documents) # initialize stream (see below)
{:ok, socket}
end
# handle_event/3 — client-side events dispatched via phx-click, phx-submit, etc.:
def handle_event("create_document", %{"title" => title}, socket) do
case MyApp.Documents.create(%{title: title, room_id: socket.assigns.room_id}) do
{:ok, doc} ->
# stream_insert/4 prepends or appends a single item — does NOT re-render the list:
{:noreply, stream_insert(socket, :documents, doc, at: 0)}
{:error, changeset} ->
{:noreply, assign(socket, :changeset, changeset)}
end
end
def handle_event("delete_document", %{"id" => id}, socket) do
doc = MyApp.Documents.get!(id)
{:ok, _} = MyApp.Documents.delete(doc)
# stream_delete/3 removes a single item from the stream — does NOT re-render the list:
{:noreply, stream_delete(socket, :documents, doc)}
end
# handle_info/2 — PubSub messages and other process messages:
def handle_info({:document_created, doc}, socket) do
{:noreply, stream_insert(socket, :documents, doc, at: 0)}
end
def handle_info({:document_deleted, doc}, socket) do
{:noreply, stream_delete(socket, :documents, doc)}
end
# assign_async/3 — LiveView 0.20+: non-blocking async data load with :loading/:ok/:error states:
def mount(%{"report_id" => id}, _session, socket) do
socket =
socket
|> assign(:report_id, id)
|> assign_async(:report_data, fn ->
# Runs in a separate Task; socket updates when complete:
{:ok, %{report_data: MyApp.Reports.generate(id)}}
end)
{:ok, socket}
end
# handle_async/3 — receives the async task result (LiveView 0.20+):
def handle_async(:report_data, {:ok, data}, socket) do
{:noreply, assign(socket, :report_data, {:ok, data})}
end
def handle_async(:report_data, {:exit, reason}, socket) do
{:noreply, assign(socket, :report_data, {:error, reason})}
end
# push_event/3 — send a custom event to the JS client hook:
def handle_info({:scroll_to, doc_id}, socket) do
{:noreply, push_event(socket, "scroll_to_doc", %{id: doc_id})}
end
end
LiveView streams
LiveView streams solve the large-list re-rendering problem. Without streams, assigning a list to the socket (assign(socket, :documents, list)) causes the LiveView diff engine to produce a full-list diff on every update — sending thousands of DOM patches over the WebSocket even when only one item changed. With streams, the diff engine tracks individual items by DOM ID and sends only the item that was inserted, updated, or deleted.
# In mount/3 — initialize stream with a list:
socket = stream(socket, :documents, documents)
# stream/4 options:
# at: 0 — prepend new items (default: append)
# limit: 100 — cap to N items (oldest are removed as new ones arrive)
# reset: true — clear and reinitialize the stream (use for filter/sort changes)
# stream_insert/4 — insert or update a single item:
socket = stream_insert(socket, :documents, new_doc) # append
socket = stream_insert(socket, :documents, updated_doc) # update in-place (by id)
socket = stream_insert(socket, :documents, new_doc, at: 0) # prepend
# stream_delete/3 — remove a single item (matched by doc.id):
socket = stream_delete(socket, :documents, doc_to_delete)
# HEEx template — phx-update="stream" activates client-side stream diffing:
# The stream produces a list of {dom_id, item} tuples.
# Each item must have an :id field (or use stream_insert with explicit dom_id).
# In the .heex template:
# <div id="documents" phx-update="stream">
# <div :for={{dom_id, doc} <- @streams.documents} id={dom_id}>
# <span><%= doc.title %></span>
# <button phx-click="delete_document" phx-value-id={doc.id}>Delete</button>
# </div>
# </div>
#
# Each item renders as a <div id="documents-{doc.id}"> DOM node.
# Only the changed node is patched — not the entire list.
#
# Replacing Enum.map in templates (the common pre-stream anti-pattern):
# BEFORE (re-renders all 2000 items on every event):
# <div :for={doc <- @documents}>...</div>
#
# AFTER (sends only the changed item):
# <div id="documents" phx-update="stream">
# <div :for={{dom_id, doc} <- @streams.documents} id={dom_id}>...</div>
# </div>
Phoenix.Presence and PubSub
# Phoenix.Presence — distributed presence tracking over PubSub:
defmodule MyApp.Presence do
use Phoenix.Presence,
otp_app: :my_app,
pubsub_server: MyApp.PubSub
end
# In the supervision tree:
{MyApp.Presence, []}
# In a LiveView — track a user and subscribe to presence diffs:
def mount(%{"room_id" => room_id}, _session, socket) do
if connected?(socket) do
MyAppWeb.Endpoint.subscribe("room:#{room_id}")
MyApp.Presence.track(socket, socket.assigns.current_user.id, %{
name: socket.assigns.current_user.name,
cursor: nil
})
end
presences = MyApp.Presence.list("room:#{room_id}")
{:ok, assign(socket, :presences, presences)}
end
# handle_info/2 receives presence diffs from Phoenix.Presence:
def handle_info(%Phoenix.Socket.Broadcast{
event: "presence_diff",
payload: %{joins: joins, leaves: leaves}
}, socket) do
presences =
socket.assigns.presences
|> Map.merge(joins)
|> Map.drop(Map.keys(leaves))
{:noreply, assign(socket, :presences, presences)}
end
# Phoenix.PubSub — broadcast and subscribe:
Phoenix.PubSub.subscribe(MyApp.PubSub, "room:#{room_id}")
Phoenix.PubSub.broadcast(MyApp.PubSub, "room:#{room_id}", {:document_created, doc})
# broadcast!/3 raises on error; broadcast/3 returns :ok | {:error, term}
# Channel — raw WebSocket with topic routing and message interception:
defmodule MyAppWeb.RoomChannel do
use Phoenix.Channel
def join("room:" <> room_id, _params, socket) do
send(self(), :after_join)
{:ok, assign(socket, :room_id, room_id)}
end
def handle_info(:after_join, socket) do
MyApp.Presence.track(socket, socket.assigns.user_id, %{online_at: System.system_time()})
push(socket, "presence_state", MyApp.Presence.list(socket))
{:noreply, socket}
end
def handle_in("new_message", %{"body" => body}, socket) do
broadcast!(socket, "new_message", %{body: body, user_id: socket.assigns.user_id})
{:noreply, socket}
end
# intercept/1 — intercept outgoing events before they are pushed to clients:
intercept ["new_message"]
def handle_out("new_message", payload, socket) do
# Filter messages for muted users:
if socket.assigns.user_id in socket.assigns.muted_users do
{:noreply, socket}
else
push(socket, "new_message", payload)
{:noreply, socket}
end
end
end
Ecto and database patterns
Ecto is Elixir’s database library and data mapping layer. Unlike an ORM, Ecto separates schema definition, changeset validation, and query composition into explicit, composable data structures. An Elixir architect on retainer designs schema associations, changeset pipelines, and query composition patterns — and diagnoses N+1 patterns using Dataloader and Ecto query logging.
Schema design and changesets
defmodule MyApp.Order do
use Ecto.Schema
import Ecto.Changeset
schema "orders" do
field :status, Ecto.Enum, values: [:pending, :processing, :shipped, :delivered]
field :total, :decimal
field :notes, :string
belongs_to :customer, MyApp.Customer
has_many :items, MyApp.OrderItem, on_delete: :delete_all
has_one :shipment, MyApp.Shipment
many_to_many :tags, MyApp.Tag, join_through: "orders_tags"
timestamps(type: :utc_datetime_usec) # inserted_at and updated_at
end
# embedded_schema — non-persisted schema for form validation (no DB table):
defmodule AddressForm do
use Ecto.Schema
import Ecto.Changeset
embedded_schema do
field :line1, :string
field :city, :string
field :postal_code, :string
field :country, :string
end
def changeset(form, params) do
form
|> cast(params, [:line1, :city, :postal_code, :country])
|> validate_required([:line1, :city, :country])
|> validate_length(:postal_code, min: 4, max: 10)
|> validate_format(:postal_code, ~r/^[A-Z0-9\s-]+$/i)
end
end
# Changeset pipeline — cast, validate, constrain:
def changeset(order, attrs) do
order
|> cast(attrs, [:status, :total, :notes, :customer_id])
|> validate_required([:status, :total, :customer_id])
|> validate_number(:total, greater_than: Decimal.new(0))
|> unique_constraint(:id, name: :orders_pkey)
|> foreign_key_constraint(:customer_id)
|> cast_assoc(:items, with: &MyApp.OrderItem.changeset/2, required: true)
# cast_assoc handles insert/update/delete of associated records in one changeset.
# put_assoc/3 replaces the entire association (use for many_to_many tag replacement):
# |> put_assoc(:tags, tags)
|> prepare_changes(fn changeset ->
# prepare_changes/2 runs inside the Repo.transaction before INSERT/UPDATE:
if get_change(changeset, :status) == :shipped do
now = DateTime.utc_now()
put_change(changeset, :shipped_at, now)
else
changeset
end
end)
end
end
Ecto queries and Dataloader
import Ecto.Query
# Composable query building:
def base_query, do: from(o in MyApp.Order, where: not is_nil(o.id))
def filter_by_status(query, status), do: where(query, status: ^status)
def with_customer(query), do: preload(query, :customer)
def order_by_recent(query), do: order_by(query, desc: :inserted_at)
def list_pending do
base_query()
|> filter_by_status(:pending)
|> with_customer()
|> order_by_recent()
|> MyApp.Repo.all()
end
# fragment/1 — raw SQL for expressions Ecto does not model:
from o in MyApp.Order,
where: fragment("? << ?", o.ip_address, ^"10.0.0.0/8"),
select: {o.id, fragment("date_trunc('day', ?)", o.inserted_at)}
# subquery/1 — use a query as a subquery:
top_customers =
from c in MyApp.Customer,
join: o in assoc(c, :orders),
group_by: c.id,
having: count(o.id) > 5,
select: %{id: c.id}
from c in subquery(top_customers), select: c.id
# Repo.stream/2 — stream large result sets in chunks (must be inside Repo.transaction):
MyApp.Repo.transaction(fn ->
MyApp.Order
|> where([o], o.status == :pending)
|> MyApp.Repo.stream(max_rows: 500) # fetches 500 rows at a time from the DB cursor
|> Stream.each(&process_order/1)
|> Stream.run()
end, timeout: :infinity)
# Dataloader — batch loading to eliminate N+1 in GraphQL resolvers and context functions:
# Add to Application children: {Dataloader, []}
# Define a source backed by Ecto:
defmodule MyApp.Loaders do
def new do
Dataloader.new()
|> Dataloader.add_source(
MyApp.Repo,
Dataloader.Ecto.new(MyApp.Repo,
query: fn queryable, _params -> queryable end
)
)
end
end
# In an Absinthe resolver — load association via Dataloader instead of Repo.get:
# BEFORE (N+1 — one Repo.get per order):
def resolve_customer(%{customer_id: id}, _args, _resolution) do
{:ok, MyApp.Repo.get(MyApp.Customer, id)}
end
# AFTER (batched — all customer IDs are collected, one IN query runs):
def resolve_customer(order, _args, %{context: %{loader: loader}}) do
loader
|> Dataloader.load(MyApp.Repo, MyApp.Customer, order.customer_id)
|> Absinthe.Resolution.Helpers.on_load(fn loader ->
customer = Dataloader.get(loader, MyApp.Repo, MyApp.Customer, order.customer_id)
{:ok, customer}
end)
end
Ecto.Multi for atomic transactions
defmodule MyApp.OrderService do
alias MyApp.{Repo, Order, Inventory, Invoice}
alias Ecto.Multi
def create_order_with_invoice(order_params, invoice_params) do
Multi.new()
# Multi.insert/4 — name + changeset; name is used to reference result in later steps:
|> Multi.insert(:order, Order.changeset(%Order{}, order_params))
# Multi.run/3 — arbitrary function with access to all previous results:
|> Multi.run(:inventory_check, fn _repo, %{order: order} ->
case Inventory.reserve(order.items) do
{:ok, reservation} -> {:ok, reservation}
{:error, item} -> {:error, "#{item.name} is out of stock"}
end
end)
|> Multi.insert(:invoice, fn %{order: order} ->
# Dynamic changeset based on previous result:
Invoice.changeset(%Invoice{}, Map.put(invoice_params, :order_id, order.id))
end)
|> Multi.update(:finalize_order, fn %{order: order, invoice: invoice} ->
Order.changeset(order, %{invoice_id: invoice.id, status: :processing})
end)
# Multi.delete_all/3, Multi.update_all/3 for bulk operations
|> Repo.transaction()
# Returns {:ok, %{order: order, inventory_check: reservation, invoice: invoice, finalize_order: order}}
# or {:error, failed_step_name, failed_value, changes_so_far}
end
# Upsert with on_conflict:
def upsert_order(params) do
%Order{}
|> Order.changeset(params)
|> Repo.insert(
on_conflict: {:replace, [:status, :total, :updated_at]},
conflict_target: [:external_id]
)
# on_conflict: :replace_all replaces all fields (use with caution — overwrites timestamps)
end
end
Distributed Elixir
One of Elixir’s most significant advantages over stateless service architectures is the ability to run a cluster of BEAM nodes that share process groups, communicate over Erlang distribution, and coordinate without a Redis intermediary. An Elixir architect on retainer designs the cluster topology, configures libcluster autodiscovery, and implements distributed process groups with the pg module.
libcluster topology configuration
# mix.exs:
{:libcluster, "~> 3.3"}
# config/runtime.exs — runtime topology (reads env vars at startup, not compile time):
config :libcluster,
topologies: [
# Kubernetes — autodiscovers pods by headless service DNS:
k8s: [
strategy: Cluster.Strategy.Kubernetes,
config: [
mode: :dns,
kubernetes_node_basename: System.get_env("RELEASE_NAME", "my_app"),
kubernetes_selector: "app=my-app",
kubernetes_namespace: System.get_env("POD_NAMESPACE", "default"),
polling_interval: 10_000
]
],
# Gossip — multicast UDP discovery for local dev (no config required):
gossip: [
strategy: Cluster.Strategy.Gossip,
config: [
port: 45892,
if_addr: "0.0.0.0",
multicast_addr: "230.1.1.251",
multicast_ttl: 1,
secret: System.get_env("CLUSTER_SECRET", "dev_secret")
]
]
]
# In Application.start/2 — add Cluster.Supervisor to the supervision tree:
topologies = Application.get_env(:libcluster, :topologies, [])
children = [
{Cluster.Supervisor, [topologies, [name: MyApp.ClusterSupervisor]]},
# ... rest of children
]
# Node inspection at runtime:
Node.list() # [:app@node2, :app@node3] — all connected nodes
Node.connect(:"app@192.168.1.5") # manually connect a node
Node.self() # :"app@node1"
pg process groups and distributed fan-out
# pg (Erlang process groups) — distributed process group membership across all nodes.
# Unlike Phoenix.PubSub (which broadcasts messages), pg tracks PIDs across the cluster.
# In Application.start/2 — start a named pg scope:
:pg.start_link(MyApp.PGScope)
# Join a process into a group (call from the process you want to track):
def init({room_id, user_id}) do
:pg.join(MyApp.PGScope, "room:#{room_id}", self())
{:ok, %{room_id: room_id, user_id: user_id}}
end
# Get all members of a group across all nodes in the cluster:
def broadcast_to_room(room_id, message) do
members = :pg.get_members(MyApp.PGScope, "room:#{room_id}")
# members is a list of PIDs — includes PIDs on remote nodes
Enum.each(members, fn pid ->
send(pid, {:broadcast, message})
# send/2 works across nodes — Erlang distribution handles remote delivery
end)
end
# get_local_members/2 — only PIDs on the current node (avoids cross-node overhead):
local_members = :pg.get_local_members(MyApp.PGScope, "room:#{room_id}")
# Combining pg with DynamicSupervisor for distributed per-room processes:
defmodule MyApp.RoomSupervisor do
# Start a room process on this node only if it isn't already running anywhere in the cluster:
def ensure_room_started(room_id) do
case :pg.get_members(MyApp.PGScope, "room:#{room_id}") do
[_pid | _] ->
# Already running somewhere in the cluster — don't start another
:already_running
[] ->
DynamicSupervisor.start_child(MyApp.RoomSupervisor, {MyApp.RoomServer, room_id})
end
end
end
mix release and runtime configuration
# config/runtime.exs — evaluated at startup; reads environment variables:
import Config
config :my_app, MyApp.Repo,
url: System.get_env("DATABASE_URL") || raise("DATABASE_URL not set"),
pool_size: String.to_integer(System.get_env("POOL_SIZE", "10")),
ssl: System.get_env("DATABASE_SSL", "false") == "true"
config :my_app, MyAppWeb.Endpoint,
secret_key_base: System.get_env("SECRET_KEY_BASE") || raise("SECRET_KEY_BASE not set"),
url: [host: System.get_env("PHX_HOST", "localhost")]
# rel/env.sh.eex — shell script evaluated before the release starts:
# Sets RELEASE_COOKIE (must be identical across all nodes) and RELEASE_NODE (unique per pod):
# export RELEASE_COOKIE="${RELEASE_COOKIE:-$(cat /etc/erlang-cookie)}"
# export RELEASE_NODE="app@$(hostname -f)"
# Production introspection without restarting:
# bin/my_app eval — runs Elixir code in the running node's context:
# $ bin/my_app eval "IO.inspect Node.list()"
# $ bin/my_app eval "IO.inspect :recon.proc_count(:memory, 5)"
# bin/my_app rpc — remote procedure call into a running node:
# $ bin/my_app rpc "MyApp.Repo.aggregate(MyApp.Order, :count, :id) |> IO.inspect"
# Graceful shutdown — preferred over hot code upgrades for most teams:
# config :my_app, MyAppWeb.Endpoint, server: true
# SIGTERM triggers Application.stop/1 which calls Supervisor.stop/3 with :shutdown reason,
# which calls terminate/2 on each GenServer with shutdown timeout from child_spec.
# Set shutdown: 30_000 in child_spec to allow 30 seconds for terminate/2 to flush state.
Production observability
The BEAM VM provides observability tools that no other runtime matches: live process inspection, production tracing without restart, ETS table inspection, and binary memory leak detection. An Elixir architect on retainer uses these tools to diagnose production issues that would require a restart and log analysis on any other platform.
:recon and :observer_cli
# :recon — production-safe BEAM introspection (add {:recon, "~> 2.5"} to mix.exs):
# proc_count/2 — top N processes by a given attribute:
:recon.proc_count(:memory, 10)
# Returns [{pid, value, [{registered_name, ...}, {initial_call, ...}, ...]}]
# Attributes: :memory, :message_queue_len, :heap_size, :reductions, :binary
:recon.proc_count(:message_queue_len, 5)
# Shows processes with the largest mailboxes — often the source of GenServer bottlenecks
# bin_leak/1 — find processes holding large shared binary references:
:recon.bin_leak(10)
# Returns top N processes by binary memory; helps diagnose binary memory leaks
# where large binaries are referenced in process state and not garbage collected
# recon_trace:calls/3 — production tracing without restart:
# Traces up to 100 calls to SessionServer.handle_info/2 for 10 seconds:
:recon_trace.calls({MyApp.SessionServer, :handle_info, 2}, 100,
[{:scope, :local}, {:time, 10_000}])
# Trace with match spec — only trace calls where first argument matches pattern:
:recon_trace.calls(
{MyApp.OrderContext, :create, 1},
50,
[{:scope, :global}]
)
# Cancel all traces when done:
:recon_trace.clear()
# :observer_cli — terminal-based observer (add {:observer_cli, "~> 1.7"} to mix.exs):
:observer_cli.start()
# Shows: process list sorted by memory/reductions; ETS tables; ports; node memory breakdown
# Navigate: press ? for help, p for process list, e for ETS, o for overall node stats
# Process mailbox inspection:
pid = Process.whereis(MyApp.SomeWorker)
Process.info(pid, :message_queue_len) # {:message_queue_len, 47_302}
Process.info(pid, :memory) # {:memory, 1_048_576}
:sys.get_status(pid) # GenServer internal state (calls format_status/2)
Telemetry and Prometheus export
# Telemetry — lightweight event emission and attachment:
# Phoenix and Ecto emit built-in events automatically:
# [:phoenix, :endpoint, :stop] — HTTP request duration
# [:phoenix, :live_view, :mount, :stop] — LiveView mount duration
# [:ecto, :repo, :query] — database query duration and result size
# Custom event emission:
:telemetry.execute(
[:my_app, :order, :created], # event name (list of atoms)
%{duration: duration, total: order.total}, # measurements (numeric values)
%{customer_id: customer_id, channel: channel} # metadata (any values)
)
# TelemetryMetrics — define metrics from Telemetry events:
defmodule MyApp.Telemetry do
import Telemetry.Metrics
def metrics do
[
summary("phoenix.endpoint.stop.duration",
unit: {:native, :millisecond},
tags: [:method, :route]
),
counter("my_app.order.created.total",
tags: [:channel]
),
last_value("my_app.session.count",
measurement: :count
),
distribution("ecto.repo.query.total_time",
unit: {:native, :millisecond},
reporter_options: [buckets: [10, 50, 100, 500, 1000]]
)
]
end
end
# TelemetryMetricsPrometheus — export metrics to Prometheus scrape endpoint:
# In Application.start/2:
{TelemetryMetricsPrometheus, [metrics: MyApp.Telemetry.metrics()]}
# Exposes GET /metrics — Prometheus scrapes this endpoint every 15 seconds.
# Manual :telemetry.attach/4 for custom handling:
:telemetry.attach(
"my_app-order-handler", # unique handler ID (string)
[:my_app, :order, :created], # event name
fn event, measurements, metadata, _config ->
Logger.info("Order created",
total: measurements.total,
customer: metadata.customer_id
)
end,
nil # config passed to handler (nil is fine)
)
:telemetry.detach("my_app-order-handler")
Benchee for performance measurement
# Benchee — benchmarking with statistical analysis:
# {:benchee, "~> 1.3", only: :dev} in mix.exs
# Run with: mix run benchmarks/query_comparison.exs
Benchee.run(
%{
"N+1 queries (before)" => fn ->
orders = MyApp.Repo.all(MyApp.Order)
Enum.map(orders, fn order ->
MyApp.Repo.get(MyApp.Customer, order.customer_id)
end)
end,
"Dataloader (after)" => fn ->
loader = MyApp.Loaders.new()
orders = MyApp.Repo.all(MyApp.Order)
loader = Enum.reduce(orders, loader, fn order, acc ->
Dataloader.load(acc, MyApp.Repo, MyApp.Customer, order.customer_id)
end)
loader = Dataloader.run(loader)
Enum.map(orders, fn order ->
Dataloader.get(loader, MyApp.Repo, MyApp.Customer, order.customer_id)
end)
end
},
time: 10, # seconds of measurement per scenario
warmup: 2, # seconds of warmup before measurement
memory_time: 2, # seconds measuring memory allocation
parallel: 1, # concurrent benchmark processes
formatters: [
{Benchee.Formatters.Console, comparison: true, extended_statistics: true}
]
)
# Output: ips (iterations per second), average, std dev, p99, memory allocated per iteration
# Comparison column shows speedup factor between scenarios
Logging Elixir retainer hours so clients understand the work
Elixir retainer work is invisible in the same way that all platform engineering is invisible: a GenServer leak diagnosis that drops the process count from 847,000 to 12,400 produces no new endpoint, no new screen, and no change visible to a product manager. A LiveView stream migration that replaces Enum.map in a HEEx template with stream/4 and phx-update="stream" produces a faster UI but looks identical from the client’s perspective until they look at WebSocket payload sizes. An Ecto N+1 elimination that integrates Dataloader and collapses 2,000 individual queries into one batched IN clause produces a faster page but leaves no trace in the feature changelog.
The work log entry is what connects the invisible Elixir platform investment to its concrete business outcome. A well-written entry captures the advisory category (OTP supervision tree design, GenServer lifecycle audit, LiveView stream migration, Ecto Dataloader integration, Phoenix PubSub design, libcluster topology configuration, :recon production tracing, Telemetry instrumentation, Benchee performance measurement), the specific module or context being worked on, the task performed, the :recon or Telemetry finding that revealed the problem, the specific API decisions made in the fix, and the before/after metric.
HourTab turns this structured work log into a public retainer URL that the client can bookmark — a live view of hours logged, progress against the monthly allocation, and the work summaries behind each entry. When the client asks “what has our Elixir architect been doing this month?”, the HourTab URL answers with the :recon.proc_count analysis that identified the session leak, the DynamicSupervisor child spec change that fixed it, and the process count and memory numbers before and after — without requiring a status call or a separately maintained report.
The log entry format that works best for Elixir retainers: [Advisory category] — [Module or context]. Task: [what was investigated]. Work: (1) [tool used and finding]; (2) [root cause identified]; (3) [fix implemented with specific API calls]. Total: [hours]. [Before metric] to [after metric]. User-visible new features: zero. That last line — “user-visible new features: zero” — is what makes the log entry honest and useful: it tells the client exactly why the hours produced no feature, while the before/after metric tells them exactly what those hours produced instead.
Retainer structure for Elixir developer engagements
An Elixir developer retainer typically covers four functional areas: feature development (new LiveView modules, new Ecto schemas and context functions, new Phoenix channels, new GenServer workers), OTP architecture advisory (supervision tree design, DynamicSupervisor and Registry patterns, GenServer lifecycle audit, process leak diagnosis with :recon), performance optimization (LiveView stream migration, Ecto N+1 elimination with Dataloader, Benchee-measured query comparison, Telemetry instrumentation), and distributed systems design (libcluster topology, pg process group architecture, mix release operational configuration). Each area should have its own hour allocation in the retainer agreement so that platform work is not competing with feature development for the same pool of hours.
Monthly retainer amounts for Elixir developer advisory and OTP architecture consulting typically range from $5,000 to $10,000 per month for OTP architecture advisory retainers (15 to 30 hours per month at mid-to-senior rates of $155 to $280 per hour), increasing to $15,000 to $28,000 per month for full-platform Elixir consulting engagements (30 to 60 hours per month) covering GenServer and Supervisor design, Phoenix LiveView stream migration, Ecto N+1 elimination, distributed libcluster topology, and production observability with :recon and Telemetry. Senior OTP architects billing at $215 to $410 per hour typically structure retainers at 20 to 45 hours per month, covering one deep architecture engagement per week plus ongoing advisory and code review.
The retainer pays for itself when it prevents a single process leak from reaching production scale: a GenServer session store that accumulates processes without termination will run acceptably at 10,000 concurrent users and catastrophically at 100,000. The architectural mistake that causes the leak — a missing handle_info clause, a supervision restart strategy set to :permanent instead of :temporary, a Process.flag(:trap_exit, true) call missing from init/1 — takes 2 to 4 hours to diagnose and fix. Left unaddressed, it takes the platform offline and consumes 40 to 80 hours of engineering time across multiple teams in an emergency. Monthly retainer advisory prevents that accumulation before it becomes an incident.
HourTab is a public retainer dashboard for freelance Elixir developers and OTP consulting firms. Upload your time-tracker CSV and get a shareable URL your client can bookmark — a live view of hours logged, remaining allocation, and work log summaries. No client login, no portal. Try it free with one active retainer.