Blog › ICP guides
Rust developer on retainer: ownership and borrow checker, async Rust with Tokio, error handling with anyhow and thiserror, and Cargo workspace governance on monthly retainer
August 8, 2026 · ~18 min read
A 40-person infrastructure startup chose Rust for its message-broker service eighteen months ago. The decision made sense: the broker needed to handle hundreds of thousands of messages per second with microsecond-level tail latencies, and the team could not afford the garbage-collection pauses that a JVM or Go runtime would introduce. Rust’s zero-cost abstractions, deterministic memory management, and fearless concurrency guarantees looked exactly right on paper. Eighteen months later, the startup had a working service — and a set of problems that only a fractional Rust engineer with production async Rust experience could diagnose.
The first problem was a deadlock that appeared intermittently under high publish load. The symptom: the broker process would occasionally freeze with CPU at 0%, no panics, no timeout errors, no log output. The engineering team suspected the Tokio runtime but could not reproduce the issue in staging. A Rust engineer reviewing the code identified the root cause in twenty minutes: the BrokerService::publish method acquired an Arc<Mutex<HashMap<String, Vec<Sender<Message>>>>> guard to look up subscribers, then called subscriber.send(msg).await while holding the guard. A MutexGuard held across an .await point means the future is holding the lock while suspended — if every Tokio executor thread is waiting to acquire the same lock, the executor deadlocks. The fix required replacing std::sync::Mutex with tokio::sync::Mutex and restructuring the critical section to drop the guard before any .await call.
The second problem was a six-week argument with the borrow checker in the routing-table module. The team had designed a RoutingTable struct holding a HashMap<String, RouteEntry>. A method needed to look up an entry, inspect it, and conditionally insert a new related entry — but holding the result of get() as a &RouteEntry reference kept a shared borrow of the entire map live, preventing the subsequent insert() call from taking a mutable borrow. The team had been cloning the entry to work around the error, but the entry contained a large nested structure and the clone was showing up in Criterion benchmarks. A Rust engineer restructured the method to a two-phase approach that dropped the immutable reference before the mutable operation, eliminating both the borrow checker conflict and the unnecessary clone.
The third problem was a C library integration. The broker needed to call a proprietary codec library distributed as a static archive with a C header. The team had written FFI bindings by hand and had a working integration, but the unsafe blocks had grown to 400 lines without a documented safety invariant in sight. The Rust engineer audited the FFI boundary, identified two places where a raw pointer could dangle if the caller dropped the Rust-owned backing buffer before the C function returned, added the lifetime constraints that made the compiler enforce the correct usage, and generated a proper header file with cbindgen for the reverse direction.
Rust developers, Rust engineers, and systems engineers on monthly retainer — fractional Rust engineers, async Rust consultants, and Rust platform advisors — do their highest-value work in the borrow checker architecture, Tokio async design, error handling, unsafe audit, and Cargo workspace governance that produces the safe, high-performance, observable systems backend the CTO reports to the board. This guide covers all six technical areas in depth with real code examples, and shows how to structure a Rust developer retainer that makes the hours behind each systems function visible.
Rust ownership, borrowing, and lifetimes
Rust’s ownership system is the language’s defining feature and its steepest learning curve. The compiler enforces three rules simultaneously: every value has exactly one owner; when the owner goes out of scope, the value is dropped (memory freed, destructors run); and references must never outlive the value they point to. These rules eliminate entire classes of bugs — use-after-free, double-free, data races — at compile time, without a garbage collector. The cost is that the rules must be satisfied for the program to compile.
The borrow checker: shared vs. exclusive references
At any point in the program, for a given value, the borrow checker enforces one of two states: either any number of immutable shared references (&T) exist, or exactly one mutable exclusive reference (&mut T) exists — never both simultaneously. This is the exclusivity invariant. It prevents data races at compile time: a data race requires at least two concurrent accesses to the same memory location where at least one is a write; the borrow checker’s exclusivity invariant makes it impossible to have a concurrent write with any other access within safe Rust.
The routing table problem from the opening narrative is the canonical borrow checker conflict pattern:
use std::collections::HashMap;
struct RoutingTable {
routes: HashMap<String, RouteEntry>,
}
impl RoutingTable {
// This does NOT compile:
fn ensure_sibling(&mut self, key: &str) {
if let Some(entry) = self.routes.get(key) {
// `entry` is &RouteEntry — shared borrow of `self.routes` is live
let sibling_key = entry.sibling_key.clone();
// ERROR: cannot borrow `self.routes` as mutable because it is
// also borrowed as immutable (via `entry`)
self.routes.insert(sibling_key, RouteEntry::default());
}
}
}
The shared borrow from self.routes.get(key) returns a &RouteEntry that borrows the entire HashMap. The insert call needs a mutable borrow of the same HashMap. Both borrows overlap in the same scope. The compiler rejects this because an insert could reallocate the HashMap’s internal buffer, which would invalidate the &RouteEntry pointer — a use-after-free that the borrow checker prevents. The fix: end the immutable borrow before the mutable operation by cloning only what you need from the entry, then dropping the reference:
impl RoutingTable {
// This compiles: borrow ends before the mutable operation
fn ensure_sibling(&mut self, key: &str) {
// Phase 1: borrow immutably, extract only the String we need
let sibling_key = self.routes.get(key)
.map(|entry| entry.sibling_key.clone()); // clone just the key, not the full entry
// Immutable borrow of self.routes ends here — entry reference dropped
// Phase 2: mutable operation — no live shared borrows
if let Some(sibling_key) = sibling_key {
self.routes.entry(sibling_key)
.or_insert_with(RouteEntry::default);
}
}
}
The clone here is cheap: sibling_key is a String, not the full RouteEntry. The original team’s workaround was cloning the entire RouteEntry struct (which contained a Vec<SocketAddr> and nested metadata), which showed up in Criterion benchmarks as unnecessary heap allocation.
Move semantics, Clone, and Copy
When a value is assigned to a new variable or passed to a function, Rust moves ownership by default: the old binding becomes invalid, and any attempt to use it after the move is a compile error. This prevents double-free bugs — only one binding is responsible for dropping the value.
let msg = Message::new("hello");
let msg2 = msg; // msg is moved into msg2; msg is no longer valid
// println!("{}", msg.body); // ERROR: use of moved value: `msg`
println!("{}", msg2.body); // OK
// Explicit clone for heap-allocated types:
let msg3 = msg2.clone(); // msg2 is still valid; msg3 is an independent copy
println!("{} {}", msg2.body, msg3.body); // both valid
Types that implement Copy (integers, floats, booleans, and tuples of Copy types) are duplicated on assignment rather than moved: the old binding remains valid after assignment because the value is represented entirely by its stack bits and a bit-copy is safe and cheap. Types that allocate on the heap (String, Vec<T>, Box<T>) do not implement Copy because a bit-copy would produce two pointers to the same heap allocation, causing a double-free on drop.
Lifetimes and function signature annotations
Lifetimes are the compiler’s mechanism for tracking how long references are valid. In most cases, the compiler infers lifetimes automatically (lifetime elision rules cover the majority of function signatures). Explicit 'a annotations are required when the compiler cannot infer the relationship between the lifetime of an input reference and an output reference:
// Without lifetime annotation, the compiler cannot determine
// whether the returned &str comes from `haystack` or `needle`:
fn first_word<'a>(haystack: &'a str, _needle: &str) -> &'a str {
haystack.split_whitespace().next().unwrap_or(haystack)
}
// The `'a` annotation tells the compiler: the returned reference
// lives at least as long as `haystack`, not necessarily as long as `needle`.
// Struct holding a reference requires a lifetime parameter:
struct MessageRef<'a> {
body: &'a str, // body borrows from some external string
}
impl<'a> MessageRef<'a> {
fn first_word(&self) -> &'a str {
self.body.split_whitespace().next().unwrap_or(self.body)
}
// The output lifetime is 'a (tied to body's source), not 'self.
// This is correct: the returned &str lives as long as the borrowed string,
// not just as long as this MessageRef instance.
}
'static is the lifetime that extends for the entire duration of the program. String literals (&str from a string literal like "hello") are 'static because they are embedded in the binary. tokio::spawn requires T: Send + 'static — the spawned future must not hold any references with a non-'static lifetime, because the Tokio runtime may move the future between threads and the future may outlive the scope that created it.
Rc<T> vs Arc<T> for shared ownership
When multiple owners need to share access to a value, Rust provides reference-counted smart pointers. Rc<T> (single-threaded reference counting) and Arc<T> (atomic reference counting, multi-threaded) enable shared ownership without a garbage collector. Neither allows mutation without an additional wrapper:
use std::sync::{Arc, Mutex, RwLock};
use std::cell::RefCell;
use std::rc::Rc;
// Single-threaded shared mutable state (e.g., a recursive tree structure):
let shared_node: Rc<RefCell<Node>> = Rc::new(RefCell::new(Node::new()));
let clone1 = Rc::clone(&shared_node);
clone1.borrow_mut().value = 42; // RefCell enforces borrow rules at runtime
// Multi-threaded shared state — use Arc + Mutex or Arc + RwLock:
let counter: Arc<Mutex<u64>> = Arc::new(Mutex::new(0));
let counter_clone = Arc::clone(&counter);
std::thread::spawn(move || {
let mut guard = counter_clone.lock().unwrap();
*guard += 1;
// guard dropped here — Mutex released
});
// Read-dominated shared state — RwLock allows concurrent readers:
let config: Arc<RwLock<Config>> = Arc::new(RwLock::new(Config::default()));
let cfg_read = config.read().unwrap(); // multiple concurrent readers allowed
let _ = cfg_read.max_connections;
drop(cfg_read); // must drop before write lock can be acquired
The choice between Mutex<T> and RwLock<T>: use Mutex when reads and writes are roughly balanced, or when the critical section is so short that the reader/writer distinction does not matter. Use RwLock when reads vastly outnumber writes and the read critical section is long enough that multiple concurrent readers produce a meaningful throughput benefit over serialized mutex-protected reads.
Async Rust with Tokio
Tokio is the most widely deployed async runtime for Rust. An async fn is transformed by the compiler into a state machine struct that implements Future<Output = T>; each .await point is a state transition where the future may suspend (returning Poll::Pending to the executor) and will be resumed when the awaited future is ready. The Tokio executor polls futures on a thread pool and never blocks executor threads on blocking I/O — which is why holding a blocking primitive (like std::sync::Mutex) across an .await point is a correctness problem, not just a performance issue.
tokio::spawn, Send + 'static, and task communication
tokio::spawn launches a new Tokio task on the runtime’s thread pool. The spawned future must satisfy Send + 'static: Send because the runtime may move the task between OS threads between .await points; 'static because the task may outlive the scope that created it. A common mistake: capturing a reference from the outer scope inside a spawned task:
use tokio::sync::mpsc;
use std::sync::Arc;
// Pattern: spawn tasks with Arc-cloned shared state
async fn run_broker(state: Arc<BrokerState>) {
let (tx, mut rx) = mpsc::channel::<Message>(1024);
// Spawn a consumer task — note the Arc::clone before moving into the closure
let state_clone = Arc::clone(&state);
tokio::spawn(async move {
while let Some(msg) = rx.recv().await {
state_clone.process(msg).await;
}
});
// Producer: send messages to the spawned consumer
for msg in generate_messages() {
tx.send(msg).await.expect("consumer dropped");
}
// tx dropped here — channel closed — rx.recv() returns None — consumer exits
}
The Arc::clone before the move closure is the standard pattern: clone the Arc (cheap atomic increment) before moving the clone into the task, so the original state binding remains available in the outer scope. Without the clone, moving state directly would make it unavailable after the spawn.
tokio::sync::mpsc (multi-producer single-consumer) is the primary channel for task communication. mpsc::channel(capacity) creates a bounded channel; the sender’s send().await yields if the buffer is full, providing backpressure. tokio::sync::broadcast is for one-to-many fan-out (each receiver gets every message, but slow receivers may miss messages if the internal buffer overflows). tokio::sync::oneshot is for a single response from a spawned task back to the caller:
use tokio::sync::oneshot;
async fn request_with_reply(
tx: mpsc::Sender<Request>,
payload: Payload,
) -> Result<Response, BrokerError> {
let (reply_tx, reply_rx) = oneshot::channel();
tx.send(Request { payload, reply: reply_tx }).await?;
// Await the single response from the handler task:
reply_rx.await.map_err(|_| BrokerError::HandlerDropped)
}tokio::select! for racing futures and cancellation
tokio::select! polls multiple futures concurrently and executes the branch of whichever future completes first, cancelling the others. This is the idiomatic pattern for timeout logic and graceful shutdown:
use tokio::time::{timeout, Duration};
use tokio::signal;
async fn serve_with_shutdown(listener: TcpListener) {
let shutdown = signal::ctrl_c();
tokio::pin!(shutdown);
loop {
tokio::select! {
// Branch 1: accept a new connection
result = listener.accept() => {
match result {
Ok((stream, addr)) => {
tokio::spawn(handle_connection(stream, addr));
}
Err(e) => eprintln!("accept error: {}", e),
}
}
// Branch 2: shutdown signal received — exit the loop
_ = &mut shutdown => {
println!("shutdown signal received; stopping accept loop");
break;
}
}
}
}
// Timeout pattern with select!:
async fn fetch_with_timeout(url: &str) -> Result<Bytes, FetchError> {
tokio::select! {
result = http_get(url) => result,
_ = tokio::time::sleep(Duration::from_secs(5)) => {
Err(FetchError::Timeout)
}
}
}
A critical nuance: when select! cancels a branch, the future for that branch is dropped at its current .await suspension point. If the cancelled future holds a tokio::sync::Mutex guard or has partially completed an operation (written half a message to a socket, inserted one row of a multi-row transaction), the cancellation leaves state in an inconsistent intermediate state. Async-cancel-safe functions are designed to be dropped at any .await point without leaving inconsistent state. The Tokio documentation marks functions as cancel-safe or not; mpsc::Receiver::recv() is cancel-safe (no messages are lost if the recv is dropped); AsyncWriteExt::write_all() is not cancel-safe (may have written partial data).
tokio::sync::Mutex vs std::sync::Mutex in async code
The deadlock from the opening narrative — a std::sync::Mutex guard held across an .await point — is the most common async Rust production bug:
use std::sync::Mutex;
use tokio::sync::Mutex as TokioMutex;
struct BrokerService {
// WRONG for async code: std::sync::Mutex blocks the executor thread
subscribers_bad: Mutex<HashMap<String, Vec<Sender<Message>>>>,
// CORRECT for async code: tokio::sync::Mutex yields the executor thread
subscribers: TokioMutex<HashMap<String, Vec<Sender<Message>>>>,
}
impl BrokerService {
// WRONG: guard held across .await — blocks executor thread while waiting
// for send to complete; other tasks on the same thread cannot run
async fn publish_wrong(&self, topic: &str, msg: Message) {
let guard = self.subscribers_bad.lock().unwrap();
if let Some(senders) = guard.get(topic) {
for sender in senders {
sender.send(msg.clone()).await; // executor thread blocked!
}
}
// guard dropped here — but damage is done during the .await above
}
// CORRECT: collect senders under the lock, drop lock, then await sends
async fn publish(&self, topic: &str, msg: Message) {
// Scope the lock acquisition to extract only what we need
let senders: Vec<Sender<Message>> = {
let guard = self.subscribers.lock().await;
guard.get(topic).cloned().unwrap_or_default()
}; // guard dropped here — lock released before any .await
for sender in senders {
// No lock held — executor thread free to schedule other tasks
let _ = sender.send(msg.clone()).await;
}
}
}
The rule of thumb: never hold a std::sync::Mutex guard across an .await point. For very brief critical sections (a counter increment, a flag set) that contain no .await, std::sync::Mutex is appropriate and more efficient than tokio::sync::Mutex. For any critical section that needs to .await inside, use tokio::sync::Mutex. The compiler will catch std::sync::MutexGuard held across .await when the future is used with tokio::spawn (because MutexGuard<T> is !Send, making the future !Send), but it will not catch the deadlock scenario when the lock is re-acquired in the same task.
Error handling with anyhow and thiserror
Rust’s Result<T, E> type makes errors explicit in the type system: a function that can fail returns Result<T, E>, and callers must handle both the Ok(T) and Err(E) variants. The ? operator provides ergonomic error propagation: it desugars to a match that returns Err(From::from(e)) on the error path, converting the source error type to the function’s return error type via the From trait. The two crates that cover most production error handling needs are thiserror for library error types and anyhow for application-level error propagation.
thiserror for library error types
Library crates expose error types that callers need to match on and handle differently. thiserror’s #[derive(Error)] macro generates the boilerplate for Display and std::error::Error implementations:
use thiserror::Error;
#[derive(Debug, Error)]
pub enum BrokerError {
#[error("topic not found: {topic}")]
TopicNotFound { topic: String },
#[error("subscriber capacity exceeded for topic {topic}: max {max}")]
CapacityExceeded { topic: String, max: usize },
#[error("message serialization failed")]
Serialization(#[from] serde_json::Error),
// #[from] generates: impl From<serde_json::Error> for BrokerError
// so `?` on a serde_json::Error returns BrokerError::Serialization(e)
#[error("channel send failed: subscriber dropped")]
SendFailed(#[source] tokio::sync::mpsc::error::SendError<Message>),
// #[source] marks the inner error for the error source() chain
// without generating a From impl (callers construct this variant explicitly)
#[error("I/O error")]
Io(#[from] std::io::Error),
}
// Usage in library code:
pub async fn publish(
&self,
topic: &str,
msg: Message,
) -> Result<(), BrokerError> {
let payload = serde_json::to_vec(&msg)?; // ? converts serde_json::Error via #[from]
let guard = self.topics.lock().await;
let senders = guard.get(topic).ok_or_else(|| BrokerError::TopicNotFound {
topic: topic.to_string(),
})?;
// ... send payload to senders
Ok(())
}
The #[from] attribute on a variant field generates a From<SourceType> for BrokerError implementation automatically, so any function that returns serde_json::Error can be called with ? inside a function returning Result<_, BrokerError>. The #[source] attribute marks the inner error as the cause in the error source chain (accessible via std::error::Error::source()) without generating a From implementation — useful when the outer error variant should be constructed explicitly by the library, not automatically via ?.
anyhow for application error handling
Application binaries and top-level service code care about propagating errors ergonomically and adding context at each call site — they do not expose errors to callers who need to match on specific variants. anyhow::Error wraps any std::error::Error + Send + Sync + 'static and propagates with ?. The anyhow::Context trait adds .context("message") to annotate errors with call-site context:
use anyhow::{Context, Result};
// Application binary: `anyhow::Result<T>` is an alias for `Result<T, anyhow::Error>`
async fn start_broker(config_path: &str) -> Result<()> {
let config_str = tokio::fs::read_to_string(config_path)
.await
.with_context(|| format!("failed to read config file: {}", config_path))?;
// If read_to_string fails, the error message becomes:
// "failed to read config file: /etc/broker/config.toml: No such file or directory"
let config: BrokerConfig = toml::from_str(&config_str)
.context("failed to parse TOML config")?;
// Any toml::de::Error is wrapped with the context string
let broker = BrokerService::new(config)
.await
.context("failed to initialize broker service")?;
// Any BrokerError from the library is wrapped — anyhow::Error wraps any std::error::Error
broker.run().await.context("broker exited with error")
}
// Accessing the original typed error when needed:
match broker_result {
Err(e) => {
if let Some(broker_err) = e.downcast_ref::<BrokerError>() {
// handle specific BrokerError variant
} else {
// generic error handling
}
}
Ok(_) => {}
}
The design boundary: library crates (crates/broker-core, crates/broker-protocol) define and export typed thiserror error enums that callers match on. The application binary (crates/broker-server) uses anyhow throughout, converting library errors automatically via ? (since anyhow::Error accepts any std::error::Error). This separation means library users get typed errors they can handle precisely, while application code is not burdened with exhaustive matching at every call site.
Result vs panicking: when unwrap() is appropriate
unwrap() and expect("message") panic on None or Err. They are appropriate in exactly two contexts: tests (where a panic is an acceptable test failure signal), and truly-unreachable branches where a panic indicates a programmer error that should terminate the process immediately. In library and application code, propagate errors with ?. The expect("message") form is always preferable to unwrap() when a panic is intentional: the message appears in the panic output and documents why the developer believed this branch was unreachable, which is invaluable when debugging a panic in production.
Unsafe Rust and FFI
Rust’s safety guarantees are enforced within safe code. The unsafe keyword demarcates blocks and functions where the programmer takes responsibility for invariants the compiler cannot verify. The five operations that require unsafe: dereferencing raw pointers (*const T, *mut T); calling unsafe functions; accessing or modifying mutable static variables; implementing unsafe traits (Send, Sync, GlobalAlloc); and accessing fields of union types. The rule: every unsafe block must have a comment documenting which safety invariant the programmer is upholding and why it holds.
FFI with C: extern blocks and #[repr(C)]
Calling a C library from Rust requires declaring the C functions with an extern "C" block and matching the C types with Rust types. Structs passed across the FFI boundary must use #[repr(C)] to ensure the compiler uses C’s layout rules rather than Rust’s (which may reorder fields for alignment):
use std::os::raw::{c_int, c_char, c_void};
// C-compatible struct layout:
#[repr(C)]
pub struct CodecConfig {
pub sample_rate: c_int,
pub channels: c_int,
pub bitrate_kbps: c_int,
}
// Declare C functions — all calls are unsafe because the compiler
// cannot verify the C function's preconditions:
extern "C" {
fn codec_init(config: *const CodecConfig) -> *mut c_void;
fn codec_encode(
ctx: *mut c_void,
input: *const u8,
input_len: c_int,
output: *mut u8,
output_capacity: c_int,
) -> c_int; // returns encoded bytes written, or -1 on error
fn codec_destroy(ctx: *mut c_void);
}
// Safe wrapper around the unsafe FFI:
pub struct Codec {
ctx: *mut c_void, // raw pointer — not Send by default
}
// SAFETY: The codec_* C functions are documented as thread-safe
// when each Codec instance is used from only one thread at a time.
// We ensure this by requiring &mut self for encode operations.
unsafe impl Send for Codec {}
impl Codec {
pub fn new(config: &CodecConfig) -> Option<Self> {
// SAFETY: config is a valid, initialized CodecConfig struct with C layout.
// codec_init returns null on failure, non-null on success.
let ctx = unsafe { codec_init(config as *const CodecConfig) };
if ctx.is_null() {
return None;
}
Some(Codec { ctx })
}
pub fn encode(&mut self, input: &[u8], output: &mut Vec<u8>) -> Result<usize, CodecError> {
let initial_capacity = input.len() * 2; // conservative estimate
output.resize(initial_capacity, 0);
// SAFETY: ctx is non-null (checked in new()); input and output are valid
// byte slices for the duration of this call; codec_encode does not
// retain pointers to these buffers after returning.
let written = unsafe {
codec_encode(
self.ctx,
input.as_ptr(),
input.len() as c_int,
output.as_mut_ptr(),
output.capacity() as c_int,
)
};
if written < 0 {
return Err(CodecError::EncodeFailed);
}
output.truncate(written as usize);
Ok(written as usize)
}
}
impl Drop for Codec {
fn drop(&mut self) {
// SAFETY: ctx is non-null (checked in new()); codec_destroy is safe
// to call exactly once on a valid context, which Drop guarantees.
unsafe { codec_destroy(self.ctx) }
}
}
The pattern: wrap the raw FFI in a safe struct with private fields, expose only safe methods with correct preconditions enforced by Rust’s type system, and document the safety invariants in SAFETY comments above each unsafe block. The Drop implementation ensures codec_destroy is called exactly once — Rust’s ownership system provides the guarantee the C library’s lifecycle documentation requires.
cbindgen and bindgen
For the reverse direction — exporting a Rust library for C callers — cbindgen generates a C header file from Rust code annotated with #[repr(C)] and #[no_mangle] pub extern "C" functions. For generating Rust FFI bindings from an existing C header, bindgen parses the header and generates the extern "C" declarations and #[repr(C)] structs automatically, eliminating the manual binding work and the class of bugs that arise from mismatched type declarations.
Performance: Criterion benchmarking and cargo flamegraph
Rust’s zero-cost abstraction guarantee means the abstraction layers in the source code — iterators, closures, generics — compile down to the same machine code a C programmer would write by hand. But the guarantee is about the language’s design, not about any given program: business logic, algorithmic choices, and memory access patterns still determine performance. Criterion benchmarking and cargo flamegraph provide the quantitative foundation for performance work on a Rust retainer.
Criterion for statistical benchmarking
criterion runs each benchmark function for multiple iterations, applies statistical analysis (mean, median, standard deviation, confidence intervals), and compares against a saved baseline to detect regressions automatically:
// benches/broker_bench.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion, BenchmarkId};
fn bench_publish(c: &mut Criterion) {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let mut group = c.benchmark_group("publish");
for message_size in [64usize, 512, 4096, 65536] {
group.bench_with_input(
BenchmarkId::from_parameter(message_size),
&message_size,
|b, &size| {
let broker = rt.block_on(BrokerService::new_for_bench());
let msg = Message::with_payload(vec![0u8; size]);
// iter() runs the closure repeatedly, measuring each iteration
b.to_async(&rt).iter(|| async {
broker.publish(black_box("test-topic"), black_box(msg.clone())).await
});
},
);
}
group.finish();
}
criterion_group!(benches, bench_publish);
criterion_main!(benches);
// Cargo.toml:
// [[bench]]
// name = "broker_bench"
// harness = false // disable the default test harness — Criterion provides its own
black_box(x) prevents the compiler from optimizing away the benchmark’s work based on constant-folding or dead code elimination: it tells the compiler “treat this value as if it could be observed externally.” Without it, the compiler may prove that the benchmark computation has no observable side effects and eliminate the entire hot path, producing meaningless nanosecond measurements. Run benchmarks: cargo bench compiles in release mode automatically and stores results in target/criterion/; subsequent runs compare against the baseline and report improvements or regressions as percentages.
cargo flamegraph for production profiling
cargo flamegraph produces an SVG flame graph of a Rust binary’s CPU usage using OS sampling profilers: perf on Linux, DTrace on macOS. The flame graph shows function call stacks proportionally by time spent:
# Profile the broker binary under load:
cargo flamegraph --bin broker-server -- --config /etc/broker/config.toml
# Profile a specific benchmark:
cargo flamegraph --bench broker_bench -- --bench publish/4096
# Cargo.toml release profile for readable symbols in flamegraphs:
[profile.release]
debug = 1 # include line-level debug info — symbols visible in flamegraph
lto = true # link-time optimization — cross-function inlining
codegen-units = 1 # single codegen unit — maximum optimization across all crate code
opt-level = 3 # O3 equivalent (default for release, shown for clarity)
Without debug = 1 in the release profile, Rust binaries strip debug symbols by default, producing a flame graph with mangled symbol names that are difficult to map back to source code. With debug = 1, the binary is larger on disk but the flamegraph shows readable function names and, on Linux with recent perf versions, file and line annotations.
Const generics for zero-cost abstractions
Rust’s const generics allow array sizes and other constant values to be compile-time parameters, enabling the compiler to generate a specialized implementation for each constant value used. This produces zero-cost abstractions that a C programmer would achieve with macros or manually unrolled loops:
// Compile-time fixed-size ring buffer — no heap allocation
struct RingBuffer<T, const N: usize> {
data: [Option<T>; N],
head: usize,
tail: usize,
len: usize,
}
impl<T: Copy, const N: usize> RingBuffer<T, N> {
const fn new() -> Self {
RingBuffer {
data: [None; N],
head: 0,
tail: 0,
len: 0,
}
}
fn push(&mut self, val: T) -> bool {
if self.len == N { return false; } // buffer full
self.data[self.tail] = Some(val);
self.tail = (self.tail + 1) % N; // compiler unrolls modulo for power-of-2 N
self.len += 1;
true
}
}
// Usage: N is a compile-time constant — compiler generates specialized code for each N
let mut small_buf: RingBuffer<Message, 64> = RingBuffer::new(); // 64-slot buffer
let mut large_buf: RingBuffer<Message, 4096> = RingBuffer::new(); // 4096-slot buffer
// No heap allocation — both live on the stack or in a staticCargo workspace governance
Production Rust projects are almost always Cargo workspaces: a root Cargo.toml with [workspace] configuration that coordinates multiple member crates. The Rust engineer on retainer governs the workspace: keeping dependency versions consistent across members, auditing for security advisories, finding and removing unused dependencies, and managing feature flags for optional capabilities.
Workspace structure and shared dependencies
# Root Cargo.toml
[workspace]
members = [
"crates/broker-core", # core business logic — no I/O dependencies
"crates/broker-proto", # protobuf-generated types (prost)
"crates/broker-server", # binary: Tokio, HTTP, gRPC
"crates/broker-cli", # binary: clap CLI for administration
]
resolver = "2" # use the v2 feature resolver (required for correct async feature unification)
# [workspace.dependencies] — define versions once, reference in member crates (Rust 1.64+)
[workspace.dependencies]
tokio = { version = "1.38", features = ["full"] }
serde = { version = "1.0", features = ["derive"] }
anyhow = "1.0"
thiserror = "1.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
prost = "0.12"
criterion = { version = "0.5", features = ["async_tokio"] }
# Member crate: crates/broker-core/Cargo.toml
[package]
name = "broker-core"
version = "0.1.0"
edition = "2021"
[dependencies]
# Reference workspace version — no version specified here
tokio = { workspace = true, features = ["sync"] }
serde = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
[dev-dependencies]
criterion = { workspace = true }
[workspace.dependencies] eliminates version drift: without it, each member crate specifies its own version of tokio, serde, and tracing; version skew accumulates as members are updated independently and Cargo compiles multiple versions of the same crate. With workspace dependencies, updating tokio = "1.38" to "1.39" in the root Cargo.toml propagates to every member crate that uses tokio = { workspace = true }.
cargo audit and cargo udeps
# cargo audit: check Cargo.lock against the RustSec advisory database
cargo install cargo-audit
cargo audit
# Example advisory output:
# Crate: openssl
# Version: 0.10.55
# Title: Use After Free in X509 in OpenSSL
# Date: 2023-11-06
# ID: RUSTSEC-2023-0044
# Solution: Upgrade to >=0.10.57
# cargo udeps: find unused dependencies (requires nightly)
cargo install cargo-udeps --locked
cargo +nightly udeps --all-targets
# Output: unused dependencies in broker-core: `uuid` (normal)
# Remove `uuid` from broker-core/Cargo.toml
cargo audit belongs in CI: add it as a step that runs on every push to main and fails the pipeline if any advisory matches a dependency in Cargo.lock. The RustSec advisory database covers both direct and transitive dependencies, making it the Rust equivalent of Snyk or Dependabot for security monitoring. cargo udeps requires the nightly toolchain to access the unstable -Z timings feature that exposes which dependencies were actually linked; run it monthly as a retainer governance function rather than in CI, and remove identified unused dependencies to reduce compile times and the attack surface from transitively pulled-in code.
Feature flags for optional capabilities
# crates/broker-core/Cargo.toml
[features]
default = []
postgres = ["dep:sqlx", "dep:tokio-postgres"]
prometheus = ["dep:prometheus", "dep:metrics"]
wasm = ["dep:wasm-bindgen", "dep:js-sys"]
[dependencies]
sqlx = { version = "0.7", optional = true, features = ["postgres", "runtime-tokio-rustls"] }
tokio-postgres = { version = "0.7", optional = true }
prometheus = { version = "0.13", optional = true }
metrics = { version = "0.22", optional = true }
# In Rust code:
# #[cfg(feature = "postgres")]
# pub mod storage {
# use sqlx::PgPool;
# // postgres-specific implementation
# }
#
# #[cfg(not(feature = "postgres"))]
# pub mod storage {
# // in-memory fallback implementation
# }
# Build with postgres feature enabled:
# cargo build --features postgres
# cargo test --features postgres,prometheus
# cargo build --no-default-features --features wasm --target wasm32-unknown-unknown
Feature flag governance is a retainer function: features that are not covered by CI tests accumulate bit rot — code behind #[cfg(feature = "postgres")] that no CI job compiles with --features postgres may have compilation errors introduced by dependency updates months before anyone notices. The Rust engineer on retainer maintains a CI matrix that compiles and tests each significant feature combination, ensuring the feature flag surface is uniformly maintained.
HourTab for Rust developer retainers
Rust developer retainer work produces a deadlock-free async broker, a borrow checker conflict resolved without unnecessary clones, an audited FFI boundary with documented safety invariants, and Criterion benchmarks that catch performance regressions before they reach production. The hours behind each outcome — the eleven-hour async deadlock investigation identifying the MutexGuard held across an .await point and restructuring the critical section to drop the guard before any send; the seven-hour borrow checker restructuring that eliminated the clone of a large RouteEntry struct; the fourteen-hour FFI audit that identified the two dangling-pointer risks and added lifetime constraints to make the compiler enforce correct usage — are not visible to the CTO or VP Engineering without a work log that connects each hour block to the specific Rust systems function performed.
HourTab gives Rust engineers and systems consultants a retainer dashboard their engineering directors can bookmark without creating an account: the month’s committed hours, the hours consumed, and the work log entries that connect each block to the async deadlock investigation, the borrow checker restructuring session, the unsafe audit, or the Criterion benchmarking engagement. When the VP Engineering can see that 11 of the month’s 30 retainer hours went to identifying and resolving the MutexGuard/.await deadlock and 7 went to restructuring the routing-table borrow conflict to eliminate unnecessary cloning, the retainer renewal conversation is grounded in the actual distribution of Rust systems advisory work rather than an abstract sense of whether the engagement produced value.
The retainer model fits Rust systems consulting because Rust codebases are living systems with ongoing governance needs. Every new async component introduces potential MutexGuard/.await hazards that require borrow checker and async design review. Every new dependency added to Cargo.toml requires cargo audit validation against the RustSec advisory database. Every new unsafe block requires a safety invariant audit. Every new FFI boundary requires a bindgen or cbindgen integration and a review of the ownership and lifetime contracts at the C/Rust boundary. Every Rust edition upgrade (Rust releases a new edition every three years) and every significant toolchain update requires evaluating new language features — async closures stabilized in Rust 1.85, impl Trait in trait definitions, let-else bindings, and const fn expansions — for adoption across the codebase. A monthly hour commitment provides the Rust engineer’s sustained availability across the full systems codebase maintenance and evolution calendar.
For Rust retainer clients who ask how to see hours at a glance, a shared retainer URL replaces the status email: the engineering director bookmarks the HourTab URL and checks hours consumed against the retainer budget without a login or a Slack message to the consultant.
Frequently asked questions
What does a Rust developer on retainer typically do?
A Rust developer or Rust engineer on monthly retainer provides ongoing systems advisory and development across five principal service areas. First, ownership and borrow checker architecture: reviewing codebases for patterns that fight the borrow checker unnecessarily (holding an immutable borrow across a block that later needs a mutable borrow; cloning entire structs to work around a borrow conflict when a two-phase approach or a scoped drop would suffice); designing shared-state patterns with Arc<Mutex<T>> for multi-threaded mutable state and Arc<RwLock<T>> for read-heavy access; and auditing lifetime annotations to confirm they are the minimal annotations needed. Second, async Rust with Tokio: reviewing async fn bodies for MutexGuard held across .await points; designing tokio::sync::mpsc channel topologies; implementing tokio::select! patterns for timeout and cancellation; and choosing between tokio::sync::Mutex (yields the executor thread while waiting) and std::sync::Mutex (appropriate only for non-async critical sections). Third, error handling with anyhow and thiserror: structuring library crate error types with thiserror’s #[derive(Error)] macro and implementing anyhow::Context in application binaries for ergonomic error propagation. Fourth, unsafe Rust and FFI: auditing unsafe blocks for documented safety invariants; reviewing #[repr(C)] struct layouts; and generating C header files with cbindgen and Rust FFI bindings with bindgen. Fifth, Cargo workspace governance: running cargo audit against the RustSec advisory database; cargo udeps for unused dependency cleanup; Criterion benchmarking for statistical regression detection; and cargo flamegraph for CPU hotspot profiling.
What Rust development work is most commonly underlogged in a retainer?
The most systematically underlogged categories are borrow checker architecture (tracing a “cannot borrow as mutable because it is also borrowed as immutable” error to a structural data model problem, restructuring to a two-phase approach, and verifying the fix does not introduce unnecessary clones in hot paths — typically 4 to 10 hours of restructuring invisible in the resolved compile error); async Rust deadlock investigation (identifying a MutexGuard held across an .await point, replacing std::sync::Mutex with tokio::sync::Mutex or restructuring the critical section to drop the guard before any .await — typically 6 to 16 hours of investigation invisible in the fixed async task topology); thiserror error type design (designing enum variants, writing #[error("...")] display strings, adding #[from] conversions, and testing downcast_ref() across crate boundaries — typically 4 to 12 hours invisible in the published error enum); and Cargo workspace dependency governance (cargo audit advisory resolution, cargo udeps cleanup, and [workspace.dependencies] version pinning — typically 3 to 8 hours invisible in the Cargo.lock diff). Detailed work log entries that capture the specific borrow pattern identified and the restructuring applied make this invisible Rust systems investment visible.
What should a Rust developer retainer agreement include?
Rust developer retainer agreements should specify: scope boundary between feature development, systems architecture advisory, code review, and performance investigation (borrow checker review and async design advisory produce no deployable artifact — define these as in-scope functions with their own hour allocation); repository access level required (read access for borrow checker and unsafe audit; write access for pull request authorship; CI pipeline access for Criterion benchmark baseline management); unsafe audit scope (whether the retainer covers reviewing all existing unsafe blocks or only new unsafe code added during the retainer period); IP ownership for Rust code contributions, Cargo workspace configurations, and cbindgen/bindgen-generated bindings; Cargo workspace governance scope (cargo audit cadence, cargo udeps review, feature flag CI matrix maintenance, [workspace.dependencies] version governance); and a shared work log documenting each borrow checker restructuring session, async deadlock investigation, error type design engagement, unsafe audit, and Criterion benchmark run. Monthly retainer amounts for Rust developer advisory and architecture consulting typically range from $7,500 to $18,000 per month for code review and architecture advisory retainers, increasing to $16,000 to $35,000 per month for full-stack Rust systems consulting covering embedded targets, WASM compilation, or compiler plugin development.
What are typical retainer rates for Rust developers and Rust engineers?
Entry-level Rust developers with 1 to 3 years of experience, Rust proficiency, and standard library familiarity typically bill $95 to $160 per hour, with monthly retainers running 10 to 18 hours for code review and advisory work. Mid-level Rust engineers with 3 to 8 years of experience, expertise in async Rust patterns (Tokio executor, tokio::sync primitives, Send/Sync constraints), unsafe Rust, and FFI, typically bill $150 to $275 per hour, with monthly retainers running 15 to 30 hours. Senior Rust engineers with 8 to 14 years of experience, expertise in compiler contributions, embedded systems, WASM targets, and open source contributions to the Rust ecosystem, typically bill $210 to $400 per hour, with monthly retainers running 20 to 40 hours. Rust consulting firms and specialized systems architecture consultancies typically bill $180 to $320 per hour. Monthly retainer amounts range from $7,500 to $18,000 per month for code review and architecture advisory retainers, increasing to $16,000 to $35,000 per month for full-stack Rust systems consulting engagements covering async architecture, unsafe audit, embedded targets, and performance engineering at scale.
How should Rust developer retainer hours be logged?
Work log entries should capture the advisory category (borrow checker architecture, async Rust/Tokio, error handling design, unsafe audit, FFI, Criterion benchmarking, cargo flamegraph profiling, Cargo workspace governance), the specific crate or module, the task, and the finding or deliverable. Example: “Async Rust/Tokio — message-broker crate, BrokerService::publish. Task: investigate intermittent production deadlock (service freezes under high load; CPU drops to 0%, no panics). Work: (1) Identified std::sync::MutexGuard held across subscriber.send(msg).await in publish method — guard live while executor suspended; 2 hours. (2) Confirmed deadlock scenario: all executor threads blocked waiting for Mutex; held by task awaiting send that cannot complete without an executor thread; 3 hours. (3) Replaced std::sync::Mutex with tokio::sync::Mutex; restructured to drop guard in scoped block before .await; 4 hours. (4) Criterion benchmark confirmed publish throughput unchanged; 30-minute load test in staging: no deadlock; 2 hours. Total: 11 hours. Result: deadlock eliminated.” Entries that document the specific MutexGuard/.await pattern and the restructuring applied connect the 11 hours to the deadlock elimination it produced.