Name: HourTab
Author: HourTab

Question 1

What does a Guidance developer on retainer typically do?

Accepted Answer

A Guidance developer on monthly retainer covers constrained generation architecture (understanding how guidance enforces output constraints via token masking at each generation step — the guidance engine maintains a logical state machine over the grammar or JSON schema, and at each LLM generation step it queries the state machine for the set of next allowed tokens, then applies this set as a mask to the LLM's logit vector before sampling; understanding the difference between guidance.gen() for free-text generation, guidance.select() for constrained selection from a fixed list, guidance.regex() for regex-constrained generation, and JSON schema-based generation; understanding when constrained generation is appropriate versus when post-hoc parsing is sufficient), performance profiling and optimization (diagnosing constrained decoding latency issues including token trie path explosion for long enum value strings, vocabulary scan overhead for schemas with large allowed-token sets at each position, and interaction between llama.cpp tokenizer byte-pair encoding and guidance's character-level grammar constraints; using Python cProfile and line_profiler to identify which guidance internal components consume inference time; understanding that guidance.select() with a pre-sorted list of candidates is significantly faster than JSON schema enum constraints for large or long-value enum sets because select() uses a pre-built Trie for O(n_chars) path navigation rather than scanning the full vocabulary at each token), grammar design (writing guidance programs that balance constraint completeness — guaranteeing the LLM can only produce valid outputs — with generation speed; understanding that nested JSON schema constraints with deeply recursive structures amplify token path exploration; structuring guidance programs to use gen() for unconstrained fields and select() for constrained fields rather than wrapping the entire output in a JSON schema grammar), and production deployment (deploying guidance programs with local llama.cpp models via llama-cpp-python; managing server-side grammar compilation and caching; understanding that grammar objects are compiled once at startup and reused across requests, but that changes to enum value lists require a grammar recompile).

Question 2

What Guidance constrained decoding and performance issues are most common on retainer?

Accepted Answer

The most common Guidance retainer issue in production classification pipelines is constrained decoding latency explosion caused by long enum value strings interacting with the tokenizer's byte-pair encoding. The mechanism in guidance 0.1.15 with llama.cpp: when guidance uses a JSON schema enum constraint, it compiles the enum values into a token path trie. At each character position during generation, guidance queries the trie for the next allowed tokens and intersects this set with the LLM's vocabulary. For short enum values like BILLING (7 chars) or RETURNS (7 chars), the trie path exploration is shallow: 2–3 token positions per value, and the intersection set is small because short values map to 1–2 common BPE tokens. For long enum values like PROFESSIONAL_SERVICES (20 chars) or IMPLEMENTATION_SERVICES (23 chars), the path exploration is deeper: 4–6 token positions per value. More critically, llama.cpp's BPE tokenizer segments long underscore-separated technical strings inconsistently: PROFESSIONAL_SERVICES might be tokenized as PRO|FESSION|AL|_|SERV|ICES or as PROFESSIONAL|_SERVICE|S depending on the model's vocabulary. At each position, guidance must scan the full vocabulary to find tokens that are valid continuations of any in-progress enum path. With 15 enum values, some containing 20+ character strings, the vocabulary scan at each character position touches 32,000 tokens (the vocabulary size of Meta-Llama-3.1-8B). The combined effect: generation of a single 20-character enum value requires approximately 6 token positions × 32,000 vocabulary tokens × 15 active enum paths = 2.88 million token evaluations. Before the schema update (12 shorter enum values averaging 8 characters), the same calculation: 3 token positions × 32,000 × 12 = 1.15 million token evaluations. The 3 new long values added in the schema update more than doubled the token evaluation work. Generation time scales super-linearly with both enum count and maximum enum value length. The fix is to replace the JSON schema enum constraint with guidance.select(), which pre-builds a character-level Trie from the enum values at startup. During generation, the Trie navigates character-by-character and only calls the vocabulary scanner at characters where multiple valid continuations exist — typically 0–2 ambiguous positions per enum value. For a 15-value enum with values up to 23 characters, guidance.select() reduces per-classification token evaluations from 2.88 million to approximately 15 × 4 ambiguous positions × a few candidate tokens = a few hundred evaluations. The latency improvement is from 14s back to 0.9s per classification.

Question 3

What are typical Guidance developer retainer rates?

Accepted Answer

Entry-level Guidance developers with foundational constrained generation skills (writing guidance programs using gen(), select(), and regex(); applying JSON schema constraints to structure LLM output; running guidance with llama-cpp-python against locally-served GGUF models; understanding the basic control flow of guidance programs and how to insert conditional logic between constrained generation steps; debugging guidance programs that produce invalid output due to grammar incompatibilities) typically bill at $70 to $105 per hour. Mid-level Guidance engineers with production performance optimization experience (diagnosing constrained decoding latency issues using cProfile and line_profiler; identifying token trie path explosion for long or numerous enum values; replacing JSON schema enum constraints with guidance.select() for performance-critical pipelines; understanding the interaction between the llama.cpp tokenizer's byte-pair encoding and guidance's character-level grammar; profiling guidance program compilation cost and implementing grammar caching; benchmarking guidance against alternative structured output approaches including outlines, lm-format-enforcer, and Ollama's JSON mode for specific use cases; tuning llama.cpp inference parameters including n_threads, n_gpu_layers, and batch_size for guidance-constrained generation workloads) typically bill at $85 to $140 per hour. Senior Guidance architects with organization-wide structured generation governance experience (designing the constrained generation architecture for pipelines with multiple classification stages; evaluating guidance against VLLM-based structured output approaches for high-throughput production workloads; building automated performance regression tests that detect latency increases from grammar schema changes before they reach production; designing enum value vocabulary policies that balance business requirements with constrained decoding performance; training engineering teams on guidance program design principles and performance characteristics) typically bill at $125 to $185 per hour. Monthly retainer ranges: $5,500 to $10,000 per month for Guidance constrained generation pipeline production reliability; $10,000 to $16,000 per month for organization-wide LLM structured output governance.

Question 4

What should a Guidance developer retainer agreement include?

Accepted Answer

A Guidance developer retainer agreement should specify: guidance version, Python version, llama.cpp version, model name and quantization (e.g., Meta-Llama-3.1-8B-Instruct-Q4_K_M), hardware (CPU cores, GPU, RAM), the guidance program structure (grammar type: JSON schema, select, regex, or hybrid), the complete enum value list for any enum-constrained fields, expected classification throughput (requests per hour or per second), p50 and p95 latency SLA, and the performance benchmark results measured on the target hardware before deployment. The most critical technical clause in a Guidance retainer agreement is the grammar performance benchmark requirement. The agreement must specify that any change to enum value lists must be benchmarked for latency impact before deployment. The benchmark must measure p50 and p95 single-request latency under the production request rate using the production hardware configuration. This requirement exists because guidance enum constraint latency is highly sensitive to both the number of enum values and the length of individual values — a schema update that adds 3 new 20-character enum values can multiply per-request latency by 17x, which is not predictable from the business change (adding product categories) without a latency measurement. The performance gate requirement: if any schema change increases p95 single-request latency by more than 20%, the change must be reviewed for guidance.select() refactoring before deployment. The agreement must also specify the monitoring requirements: p50 and p95 classification latency must be tracked per-endpoint and alerted on if they increase more than 2x above the baseline established at the start of the retainer. Latency increases that correlate with schema changes are immediately suspects for constrained decoding performance issues.

Question 5

How should Guidance developer retainer hours be logged?

Accepted Answer

Log each Guidance retainer session with: guidance version, llama.cpp version, model name and quantization, hardware configuration, issue type (JSON schema enum constraint latency explosion after adding long enum values; guidance program produces output that bypasses grammar constraint for certain input patterns; grammar compilation timeout at server startup for complex nested schemas; guidance program crashes with RecursionError on deeply nested JSON schema; select() Trie not updating after enum list change requires server restart; regex constraint produces different output distribution than expected due to tokenizer boundary effects), root cause (Fairbrook Classification Systems Ltd.; guidance 0.1.15; Python 3.11; llama.cpp 0.2.83; Meta-Llama-3.1-8B-Instruct-Q4_K_M; 2× Intel Xeon Gold 6330 (56 cores total); customer support ticket classification; 12-value product category enum JSON schema constraint; schema update September 10 2026 adds 3 new values: PROFESSIONAL_SERVICES (22 chars), ENTERPRISE_LICENSING (20 chars), IMPLEMENTATION_SERVICES (23 chars); guidance 0.1.15 JSON schema enum constraint compiles to token path trie; at each character position during generation: vocabulary scan for all tokens that are valid continuation of any active enum path; 15 enum values × average 4.8 positions × 32,000 vocabulary scan = 2.88M token evaluations per classification; before update: 12 × 3.1 positions × 32,000 = 1.19M evaluations; p50 latency: 0.8s → 14.3s; throughput: 1,250/hour → 252/hour; team adds 4 additional servers; latency unchanged — CPU-bound constrained decoding not I/O-bound; 5 weeks SLA breach; 31,000 ticket backlog; fix: replace JSON schema enum constraint with guidance.select(sorted(PRODUCT_CATEGORIES, key=len)); Trie built at startup; per-character navigation O(n_ambiguous_positions) not O(n_values × vocab_size); post-fix p50: 0.9s; throughput: 1,190/hour; 2.5h), and hours by component (triage: 0.5h — confirm latency before vs after September 10 schema update; run cProfile on single classification call; identify token_trie.get_next_tokens() as 87% of CPU time; confirm enum value list change is the trigger; root cause: 0.5h — measure single-request latency for each enum value length: BILLING (7 chars) → 0.8s, PROFESSIONAL_SERVICES (22 chars) → 2.1s per value alone; confirm super-linear scaling; explain BPE tokenizer interaction; reproduce on minimal 2-value case; fix: 1.0h — replace JSON schema enum with guidance.select(); sort enum values by frequency (most common first for faster early trie termination); benchmark 100 classification requests before and after; confirm p50 0.9s; add latency alerting threshold 2× baseline; validation: 0.5h — run 500-request load test; confirm throughput 1,190/hour; confirm classification accuracy unchanged; document schema change performance gate in team runbook). Hours logged to HourTab with guidance version, token trie latency root cause, and guidance.select() fix so the client can audit the full 2.5-hour engagement at the shared URL without logging in.

Guidance developer on retainer: constrained decoding latency grows from 0.8s to 14s after adding 20-character enum values — `guidance 0.1.15` — `llama.cpp` tokenizer produces exponential token path expansion — 5 weeks of SLA breach — Guidance developer on monthly retainer

Why long enum values cause exponential latency growth

Discovery: profiling with cProfile

The fix: `guidance.select()` with a pre-built Trie

Validation

Related retainer posts

Guidance developer on retainer: constrained decoding latency grows from 0.8s to 14s after adding 20-character enum values — guidance 0.1.15 — llama.cpp tokenizer produces exponential token path expansion — 5 weeks of SLA breach — Guidance developer on monthly retainer