Name: HourTab
Author: HourTab

Question 1

What does an Instructor developer on retainer typically do?

Accepted Answer

An Instructor developer on monthly retainer covers structured output extraction pipeline design (defining Pydantic models that match the information schema required by the downstream application; choosing between instructor.Mode.TOOLS (uses LLM tool calling for structured output), Mode.JSON (instructs LLM to produce a JSON object), Mode.JSON_SCHEMA (provides the full JSON schema as a constraint), and Mode.MD_JSON (instructs LLM to produce Markdown-fenced JSON) based on the LLM provider's capabilities and the failure modes observed in production; understanding that each mode produces different prompting and different LLM behaviors for ambiguous schema constructs like Optional fields, Union types, and nested Pydantic models), provider-specific behavior tuning (diagnosing provider-specific structured output failures — OpenAI's JSON mode produces clean array serialization but Claude's tool use mode with anyOf schemas can produce string-serialized arrays for Optional[List[str]] fields; understanding that Instructor's retry mechanism sends the ValidationError message back to the LLM to request a corrected response, but if the error message does not identify the specific field serialization issue in terms the LLM understands, retries produce the same failure; choosing the appropriate Mode for each provider; adding model_validator(mode='before') pre-validation hooks to coerce common LLM serialization patterns — string-encoded lists, stringified integers, ISO date strings for date fields — before Pydantic runs strict validation), retry strategy configuration (configuring max_retries appropriately — 2 to 3 retries covers transient model errors without burning excessive API budget on structural failures that will always fail regardless of retry count; implementing custom retry logic using instructor's on_error callback for cases where the standard retry prompt is insufficient; adding logging at the retry level so production monitoring can detect extraction schemas that are producing systematic retries rather than occasional ones; understanding the API cost implications of retries: each retry is a full LLM call at the same token count as the original), validation and quality monitoring (building extraction evaluation pipelines that run a representative set of documents with known ground truth through the production schema and measure field-level extraction accuracy; identifying which specific fields have high ValidationError rates using field-level error tracking; monitoring the fraction of documents that exhaust max_retries and return None-filled models, as this is the primary signal of a structural schema-LLM mismatch requiring a schema or mode change), and production operations (deploying Instructor pipelines in FastAPI or async worker architectures with proper error handling for the case where all retries are exhausted; tracking per-document extraction metadata — which model was used, how many retries were required, which fields had ValidationErrors — so that downstream consumers of the extracted data can filter out low-confidence extractions; managing Instructor version upgrades, as minor versions can change how Optional types are serialized in tool schemas and may require schema or validator adjustments).

Question 2

What Instructor structured output and Claude provider issues are most common on retainer?

Accepted Answer

The most common Instructor retainer issue in production pipelines that use Claude (via instructor.from_anthropic()) is the systematic extraction failure caused by Claude returning a JSON-serialized string for Optional[List[str]] fields instead of a JSON array. The mechanism in instructor 1.3.7: when a Pydantic model contains an Optional[List[str]] field — defined as parties: Optional[List[str]] = None — Pydantic 2's JSON schema generation represents this as anyOf: [{type: array, items: {type: string}}, {type: null}]. Instructor in Mode.TOOLS sends this schema to Claude as the tool definition's input_schema. Claude models, particularly Claude 3 Haiku and Claude 3 Sonnet, occasionally interpret the anyOf with null alternative as permission to choose between representations. For documents where the field has multiple values — a contract with three parties, a legal citation with four governing law references — Claude intermittently returns the field as a JSON-serialized string: parties: '["Acme Corp", "Widget Inc", "Broker LLC"]' rather than the expected JSON array: parties: ["Acme Corp", "Widget Inc", "Broker LLC"]. This is a valid JSON value (a string) that satisfies the anyOf constraint technically (a string is not null), but it does not satisfy the array type alternative that Pydantic is expecting. Pydantic 2.7.1 in the default validation mode does not coerce a string to a list. ValidationError is raised: 1 validation error for ContractExtraction parties: Input should be a valid list [type=list_type, input_value='["Acme Corp",...]', input_type=str]. Instructor's retry mechanism constructs a follow-up message containing the ValidationError text and sends it back to Claude with a request to fix the response. The retry prompt looks like: I got a validation error: 1 validation error for ContractExtraction parties Input should be a valid list. Please fix your response. Claude's retry response continues to produce the same string serialization because the error message — Input should be a valid list — does not indicate that the JSON array syntax ["Acme Corp", "Widget Inc"] is required; Claude interprets the original response as already being a list (it is a string that looks like a list). All max_retries=3 attempts fail with the same ValidationError. Instructor logs these at WARNING level and returns a ContractExtraction model with parties=None. The downstream pipeline that iterates over parties to perform contract analysis raises AttributeError: 'NoneType' object is not iterable. This failure is systematically correlated with document characteristics: documents that have exactly one party in the list field rarely trigger it (single-value lists are represented as ["Acme Corp"] which Claude renders correctly), while documents with three or more parties have a 15–30% failure rate. The failure does not appear in OpenAI model pipelines because OpenAI's tool use API enforces stricter JSON schema compliance and does not produce string-serialized arrays. The fix has two forms. The first is a Pydantic model_validator(mode='before') that runs before Pydantic field validation and coerces string values to list where a list is expected: import json; value = json.loads(value) if isinstance(value, str) else value. This handles the string-serialized list pattern without changing the Instructor mode and without retries. The second form is switching from Mode.TOOLS to Mode.JSON_SCHEMA for Claude. In JSON_SCHEMA mode, Instructor constrains Claude's output format more tightly and Claude produces clean JSON arrays rather than string-serialized arrays. The trade-off is that JSON_SCHEMA mode does not support nested tool call structures and may require schema simplification for complex Pydantic models. For most production extraction use cases where the primary failure is Optional[List[str]] string serialization, the model_validator approach is preferred because it handles the coercion transparently without requiring a mode change.

Question 3

What are typical Instructor developer retainer rates?

Accepted Answer

Entry-level Instructor developers with foundational structured extraction skills (defining Pydantic models for structured extraction; patching OpenAI and Anthropic clients with instructor.from_openai() and instructor.from_anthropic(); calling client.chat.completions.create() with a response_model parameter; understanding max_retries and how Instructor constructs the retry prompt from ValidationError messages; running the extraction on a single document and inspecting the returned Pydantic model; understanding the difference between Optional and required fields and how each affects the extraction prompt) typically bill at $60 to $95 per hour. Mid-level Instructor engineers with production extraction pipeline experience (diagnosing and fixing provider-specific structured output failures — Claude anyOf string serialization for Optional[List[str]], OpenAI occasional None return for required fields when the document genuinely doesn't contain the field, Claude 3 Opus vs Haiku extraction quality differences for complex nested models; implementing model_validator(mode='before') coercers for common LLM serialization patterns; choosing between Mode.TOOLS, Mode.JSON, Mode.JSON_SCHEMA, and Mode.MD_JSON based on provider and schema complexity; building field-level extraction accuracy evaluation over a labeled document set; implementing per-document extraction metadata tracking — model used, retry count, ValidationError fields — to identify systematically failing schema elements; optimizing token usage by removing unnecessary Optional fields from schemas for providers where the optional/required distinction causes ambiguous output) typically bill at $85 to $135 per hour. Senior Instructor architects with organization-wide LLM extraction governance experience (designing multi-provider extraction pipelines that route different document types to the most cost-effective model based on benchmarked field-level accuracy; building automated schema regression testing frameworks that run the full Pydantic model against a 500-document golden set after every schema change and alert on field accuracy regressions above 1 percentage point; evaluating Instructor against OpenAI Structured Outputs (beta), Anthropic's explicit tool_use with manual Pydantic validation, and custom prompting approaches for specific organizational requirements; designing extraction schemas for complex legal, financial, and medical documents that balance field granularity with extraction reliability; building review workflows that route low-confidence extractions — documents that required all max_retries — to human review queues; managing Instructor version upgrades across a fleet of production extraction pipelines) typically bill at $120 to $180 per hour. Monthly retainer ranges: $5,500 to $10,000 per month for Instructor extraction pipeline production reliability; $10,000 to $17,000 per month for organization-wide LLM extraction governance with multi-provider routing.

Question 4

What should an Instructor developer retainer agreement include?

Accepted Answer

An Instructor developer retainer agreement should specify: instructor version, Python version, Pydantic version, LLM provider and model (OpenAI gpt-4o-mini, Anthropic claude-3-haiku-20240307, etc.), Instructor Mode (TOOLS, JSON, JSON_SCHEMA, MD_JSON), the Pydantic model definition or a summary of its field types including all Optional and Union fields, max_retries setting, expected daily document volume, field-level extraction accuracy requirements (e.g., parties field: 98% accuracy on documents with 2+ parties), and the downstream system that consumes the extracted Pydantic model. The most critical technical clause in an Instructor retainer agreement is the field-level accuracy requirement with provider-specific baseline documentation. The agreement must specify the extraction accuracy measured on a labeled document sample for each field in the Pydantic model, broken down by LLM provider. This is necessary because the same Pydantic schema can produce dramatically different failure rates on different providers: a schema with Optional[List[str]] fields may achieve 98% accuracy on OpenAI gpt-4o-mini but only 85% on Claude 3 Haiku in Mode.TOOLS due to the anyOf string-serialization pattern. The agreement must document the current baseline accuracy, the target accuracy, and which specific model-mode combination achieves the target. For retainer agreements involving Claude providers, the agreement must specify whether model_validator(mode='before') coercers are implemented for all Optional[List[...]] fields and all other fields where string-to-type coercion is required. If coercers are not implemented, the agreement must document the accepted residual failure rate and the maximum_retries budget allocated per document. The retry monitoring requirement: production Instructor pipelines must log the retry count for every document. A rolling 7-day metric of fraction_of_documents_exhausting_all_retries must be computed and alerted on if it exceeds 2% — this threshold signals a structural schema-LLM mismatch requiring investigation rather than transient model errors that retry successfully.

Question 5

How should Instructor developer retainer hours be logged?

Accepted Answer

Log each Instructor retainer session with: instructor version, Pydantic version, LLM provider and model, Instructor Mode, issue type (Optional[List[str]] anyOf string serialization from Claude; required field returns None when document doesn't contain value; nested Pydantic model flattening by model causing wrong extraction; max_retries exhausted on retry prompt that doesn't identify specific serialization issue; Union type causes provider to select wrong alternative; Date field returns string instead of datetime.date object), root cause (Brightfield Legal Analytics Ltd.; instructor 1.3.7; Python 3.11; Anthropic 0.28.0; Pydantic 2.7.1; FastAPI 0.111.0; legal contract extraction pipeline; 200 documents/day; ContractExtraction Pydantic model with parties: Optional[List[str]] = None and governing_law_citations: Optional[List[str]] = None; instructor.from_anthropic(client) in Mode.TOOLS; Pydantic 2 JSON schema for Optional[List[str]]: anyOf: [{type: array, items: {type: string}}, {type: null}]; Instructor sends ContractExtraction schema as tool input_schema to Claude 3 Haiku; Claude with anyOf{array, null} schema: interprets string serialization as valid — returns parties: '["Acme Corp", "Widget Inc", "Broker LLC"]' (string) instead of parties: ["Acme Corp", "Widget Inc", "Broker LLC"] (array); frequency: 15% of documents with 2+ parties in list field; Pydantic 2.7.1 strict validation: Input should be a valid list [type=list_type, input_value='[...]', input_type=str] — no coercion for str→list; Instructor retry prompt: ValidationError text: 1 validation error for ContractExtraction parties Input should be a valid list; Claude retry: same string serialization — error message does not indicate JSON array syntax required; all 3 retries fail; Instructor returns ContractExtraction with parties=None; downstream: for party in contract.parties — AttributeError: NoneType is not iterable; 11 weeks August–October 2026; ~3,300 failed documents; retainer developer: enable instructor logging (logging.getLogger('instructor').setLevel(logging.DEBUG)); observe raw Claude response parties field; confirm string-serialized list pattern; fix: add @model_validator(mode='before') to ContractExtraction; coerce: if isinstance(data.get('parties'), str): import json; data['parties'] = json.loads(data['parties']); apply same to governing_law_citations; post-fix: 0 ValidationError on parties/governing_law_citations; 0.3% residual from genuinely malformed responses; 2.0h); and hours by component (triage: 0.5h — enable DEBUG logging; run 20 problem documents; confirm ValidationError on parties field in all 20; inspect raw Claude response: parties field is string; confirm 15% failure rate over 7-day log sample; root cause: 0.5h — inspect instructor source for Mode.TOOLS schema serialization of Optional[List[str]]; confirm anyOf generation; reproduce minimal case on 3-party contract; confirm Claude returns string-serialized list with anyOf schema; confirm all 3 retries fail with same serialization; fix: 0.5h — add model_validator(mode='before') to ContractExtraction and 2 other models with Optional[List[str]] fields; test on 50 historical failure documents; confirm 0 ValidationError; validation: 0.5h — run 200-document evaluation set; compute per-field accuracy; confirm parties accuracy: 85% → 99.7%; verify 0.3% residual is genuinely malformed responses not the string-serialization pattern). Hours logged to HourTab with the instructor version, Anthropic provider, anyOf Optional[List[str]] string serialization root cause so the client can audit the full 2.0-hour engagement at the shared URL without logging in.

Instructor developer on retainer: Claude returns JSON string for `Optional[List[str]]` instead of JSON array — `anyOf` schema triggers string serialization — Pydantic ValidationError on all 3 retries — 15% extraction failure rate for 11 weeks — Instructor developer on monthly retainer

Why Claude returns a JSON string instead of a JSON array

Discovery: enabling Instructor DEBUG logging

The fix: model_validator for string-to-list coercion

Related retainer posts

Instructor developer on retainer: Claude returns JSON string for Optional[List[str]] instead of JSON array — anyOf schema triggers string serialization — Pydantic ValidationError on all 3 retries — 15% extraction failure rate for 11 weeks — Instructor developer on monthly retainer

Why Claude returns a JSON string instead of a JSON array

Discovery: enabling Instructor DEBUG logging

The fix: model_validator for string-to-list coercion

Related retainer posts

Instructor developer on retainer: Claude returns JSON string for `Optional[List[str]]` instead of JSON array — `anyOf` schema triggers string serialization — Pydantic ValidationError on all 3 retries — 15% extraction failure rate for 11 weeks — Instructor developer on monthly retainer