Blog › ICP guides

Crystal developer on retainer: static typing, macros, Fiber concurrency, and systems programming on monthly retainer

September 28, 2026 · ~19 min read

At 2:14 AM on a Wednesday, a data pipeline written in Crystal stopped producing output. No error was logged. No process crashed. The Crystal binary was still running — CPU at 0%, memory stable — but the pipeline had produced zero records for eleven hours by the time the on-call engineer noticed during the morning data quality check. The culprit was a single unrescued exception inside a consumer Fiber block. Crystal’s Fiber scheduler silently terminates any Fiber that raises an uncaught exception; the Fiber simply ceases to exist, and the Channel(Message) buffer it was consuming continued to receive sends from the producer until the buffer capacity of 100 was reached, at which point the producer Fiber blocked permanently on channel.send. The application appeared healthy — the process was alive, memory was stable — because the producer was blocked waiting for a receive that would never come. The pipeline had been deadlocked for eleven hours without a single log line.

The Crystal developer on retainer diagnosed it in one session. They set CRYSTAL_WORKERS=1 to run the scheduler on a single OS thread (eliminating scheduler concurrency as a variable), added a bare rescue Exception => ex block inside the consumer Fiber body that sent the exception to a separate error_channel = Channel(Exception).new, and ran the pipeline with the known-bad input record that triggered the original failure. The exception surfaced immediately: a JSON::ParseException raised when a downstream record had a malformed timestamp field, triggering the nil dereference that killed the Fiber. The fix was two changes: rescue blocks inside every Fiber body routing exceptions to an error channel, and a monitoring Fiber that drains the error channel and logs structured error records. The diff was twenty-two lines. The pipeline never silently deadlocked again.

Crystal language fundamentals: type system, union types, and nil safety

Crystal is a statically typed, compiled language with Ruby-inspired syntax that compiles to native code via LLVM. Its type system performs full type inference — x = 42 infers x : Int32, name = "hello" infers name : String — so explicit type annotations are optional but available: def process(records : Array(Record)) : Hash(String, Int32). The most distinctive feature of Crystal’s type system relative to Ruby is compile-time nil safety through union types. A method that might return nil has return type T | Nil, and any attempt to call a method on the result without a nil check is a compile error: user = find_user(id); user.name fails if find_user returns User | Nil. The compiler requires the programmer to handle the nil case explicitly, either via a nil guard (if user = find_user(id); user.name; end), via nil? check (user = find_user(id); user.name unless user.nil?), via not_nil! when the caller has out-of-band knowledge that the value cannot be nil at this point (find_user(id).not_nil!.name — raises NilAssertionError at runtime if wrong), or via the nilable operator &. for method chaining that short-circuits on nil (find_user(id)&.name returns String | Nil).

Union types generalize beyond T | Nil to arbitrary unions: result : String | Int32 | Bool is a valid type, and exhaustive pattern matching via case result; when String; when Int32; when Bool; end is a compile-time checked dispatch that the compiler uses to prove that every branch of the union is handled. The is_a?(Type) predicate serves as a type narrowing guard — inside the if result.is_a?(String) branch, the compiler narrows result to String and permits all String methods without further annotation. This compile-time type narrowing is the Crystal equivalent of TypeScript’s type guards or Rust’s if let pattern matching, and it is the mechanism that makes Crystal’s zero-overhead abstractions safe: the compiler eliminates the union dispatch at the narrowed branch because the type is statically known. A retainer engagement covering type system architecture audits codebases for not_nil! calls that assert non-nil without justification (each is a potential runtime NilAssertionError), identifies methods with overly broad return types that force union handling throughout the call graph when a redesigned interface could return a concrete type, and advises on the tradeoff between T | Nil return types (which force callers to handle nil at compile time) and exception-raising methods (which push nil handling to runtime but simplify call-site code for callers that want early-exit semantics).

Generics in Crystal use type parameter syntax similar to Java or C#: class Stack(T); def push(item : T); def pop : T | Nil; end. Generic constraints are expressed via module inclusion: def process(item : T) forall T accepts any type, while def process(item : T) forall T; {% raise "T must include Comparable" unless T.includes?(Comparable) %}; end uses a compile-time macro check to enforce interface requirements (Crystal uses duck typing at the method resolution level, so generic constraints are typically checked at instantiation time by the compiler reporting “method not found” rather than via explicit constraint declarations). Abstract classes and modules define interface contracts: abstract class Parser; abstract def parse(input : String) : Result; end forces all concrete subclasses to implement parse with the specified signature or receive a compile error. A retainer engagement covering type architecture designs module inclusion hierarchies that express domain concepts (a Serializable module with required to_json : String and from_json(raw : String) : self class methods) rather than relying on duck typing that defers type errors to call sites.

Fiber concurrency, Channel CSP, and macro authorship

Crystal’s concurrency model is based on green threads called Fibers, scheduled cooperatively on a pool of OS threads whose size is controlled by the CRYSTAL_WORKERS environment variable (default: the number of logical CPUs). Fibers are cheap — each costs approximately 4 KB of stack space at creation — and are created with spawn: spawn do; process_record(record); end. Communication between Fibers uses CSP-style channels: ch = Channel(String).new creates an unbounded channel, ch = Channel(String).new(100) creates a buffered channel with capacity 100. A sender calls ch.send(value) — which blocks if the channel is full (buffered) or if no receiver is ready (unbounded). A receiver calls ch.receive — which blocks if the channel is empty. The select statement coordinates multi-channel operations: Crystal::Select.new.when(ch1) { |v| handle_v1(v) }.when(ch2) { |v| handle_v2(v) }.execute wakes when any of the listed channels has a value, enabling fan-in patterns. Channel#close signals channel closure; receivers get Channel::ClosedError on subsequent receive calls, or a nil return from receive? (the non-raising variant), enabling graceful pipeline shutdown.

The most important operational property of Crystal Fibers — and the most common source of production incidents in Crystal systems — is that an unrescued exception inside a Fiber block terminates that Fiber silently. Unlike Go goroutines (which panic the entire program on unrecovered panics) or Elixir processes (which log a supervisable crash message and restart via OTP supervisors), a Crystal Fiber that raises an uncaught exception simply ceases to exist, with no log output, no signal to any other Fiber, and no change in the process exit code. This means a consumer Fiber in a Channel(T)-based pipeline can be killed by a malformed input record, and the producer will continue sending to the buffered channel until the buffer fills, at which point the producer blocks permanently — a silent deadlock indistinguishable from an idle but healthy process. The retainer-level solution is defensive Fiber design: every spawn block that performs non-trivial work wraps its body in rescue Exception => ex, routes the exception to an error channel (error_ch.send(ex)), and either exits the loop cleanly or re-raises after logging, depending on whether the error is record-level (skip the bad record, continue processing) or pipeline-level (halt, drain, and signal upstream). A monitoring Fiber drains the error channel and writes structured error log entries. The architecture adds approximately 15 lines of scaffolding per Fiber body and eliminates the class of silent pipeline hangs entirely.

Crystal’s macro system is a compile-time code generation facility that operates on the AST before type checking. Macros are defined with macro name(args); {{ body }}; end, where {{ expr }} evaluates an expression at compile time and {% stmt %} executes a compile-time statement (if, for, raise). The most powerful macro introspection tool is @type, which at compile time refers to the type in whose scope the macro is expanded, providing access to @type.instance_vars (a list of all instance variable names and types), @type.methods, @type.ancestors, and @type.abstract?. This enables macros that generate boilerplate from the type structure: a generate_accessors macro that iterates @type.instance_vars and emits def var.name : var.type getter and def var.name=(v : var.type) setter method definitions, a to_h macro that generates a Hash(String, JSON::Any) from all instance variables, or a validation macro that generates an errors : Array(String) method that checks each instance variable against its type annotation. The @[Annotation] system allows custom annotations that macros can read: @[MyApp::Validate(min: 1, max: 100)] attached to an instance variable is readable in a macro via var.annotation(MyApp::Validate), providing a structured configuration mechanism for generated code. A retainer engagement covering macro authorship designs these systems so that they are debuggable — pp! variable_name inside a macro body prints the compile-time value, and the -Dmacro_debug compiler flag prints the full macro expansion output before type checking begins.

How HourTab tracks Crystal developer retainer hours

Crystal retainers produce the same invisibility problem as all systems-programming retainers, amplified by the fact that Crystal’s compile-time guarantees — the nil safety eliminations, the type narrowing, the macro-generated boilerplate — are invisible to anyone who was not in the codebase before the changes were made. A client who hires a Crystal developer on retainer sees their pipeline running reliably, their HTTP service producing correct JSON responses, their C library wrapper not leaking memory — and has no way to connect that reliability to the 14-hour Fiber deadlock diagnosis session that moved the system from “silently hangs every few days” to “hasn’t deadlocked in 60 days.” The work log entry “fixed Fiber deadlock, 14h” describes duration and leaves the client unable to explain the mechanism to their engineering leadership or to evaluate whether 14 hours was proportionate to the benefit. The gap between what was done (twenty-two lines of rescue scaffolding and error channel routing) and what was prevented (silent production pipeline hangs that required manual process restarts discovered by next-morning data quality checks) requires a structured explanation of how Crystal Fibers die, why the scheduler does not log or signal on Fiber exception, and how the error channel pattern surfaces failures that would otherwise be invisible — context that takes five minutes to write once, per log entry, and prevents twenty minutes of client confusion per billing cycle.

HourTab gives Crystal developers a public retainer-hours URL they paste into the first message of every client engagement. The client opens the URL and sees the current burn-down: hours purchased, hours used, hours remaining, and a work log of every session. For Crystal retainers specifically, the work log entries carry more information than the burn-down chart alone can convey. Each entry should name the Crystal mechanism involved (Fiber exception handling, Channel buffer capacity, union type nil guard, macro @type.instance_vars introspection, lib/fun C binding layout, crystal-db pool checkout_timeout), the diagnostic approach used (CRYSTAL_WORKERS=1 single-thread isolation, pp! compile-time output, --release profile run, crystal tool hierarchy for type dependency graph), the specific file and method, the change made and the reason it was necessary (rescue inside Fiber body because Crystal Fibers silently terminate on uncaught exceptions without scheduler notification), and the before-and-after observable metric (pipeline silent deadlock frequency: weekly → zero over 60 days; compile errors from nil audit: 23 nil not handled errors eliminated; memory growth from C binding: 2.1 MB per 10,000 operations → flat after GC finalizer). Entries at that level of specificity turn an invoice line item into a documented systems improvement that the client can reference internally and that provides the evidence base for retainer renewal conversations.

FAQ: Crystal developer retainers