Blog › ICP guides

Icon developer on retainer: goal-directed evaluation, generator expressions, string scanning, and every...do exhaustion on monthly retainer

November 8, 2026 · ~16 min read

An Icon program processing structured text records was producing three missed matches per run. The program used a generator expression inside a procedure — a suspend statement that yielded pattern match positions for a regex-style pattern search — and called this generator procedure from a top-level expression in the main loop. Icon’s goal-directed evaluation drove the generator forward until the enclosing top-level expression succeeded: the first generated match position was consumed, the enclosing expression evaluated to success, and evaluation stopped. The remaining match positions produced by the suspend-based generator were never requested — Icon’s goal-directed evaluation had no reason to request further values once the enclosing expression had already succeeded. Three text records per run contained multiple matches, but only the first match was processed; the remaining matches were silently discarded without any error, warning, or indication that additional generator values had been abandoned. The Icon developer on retainer diagnosed the generator truncation: the procedure was a generator (it used suspend to yield values one at a time), but the call site was a top-level expression that satisfied its evaluation goal after the first generated value, not an every...do loop that would explicitly request every value the generator could produce. The fix restructured the call site to use every match := generateMatches(record) do processMatch(match), forcing evaluation of every generated value. Missed matches per run: 3 → 0.

The work log entry read “fixed match loop, 13h.” It names the symptom and the duration. It cannot explain to a client why Icon’s goal-directed evaluation makes generator truncation the default behavior rather than an error (Icon was designed by Ralph Griswold at the University of Arizona in the late 1970s as a successor to his earlier SNOBOL4 string processing language; Icon’s goal-directed evaluation model was designed to make pattern matching and search natural — an expression either succeeds and produces a value, or fails, and the | alternation operator and & conjunction operator automatically drive backtracking and search without explicit loop constructs; in this model, a generator produces a sequence of values on demand, but the consumer controls how many values are requested, and the default top-level expression evaluation requests exactly one — the first success), why every...do is structurally different from a while loop in a language like C or Python (in Icon, every expr do body is not syntactic sugar for a while loop — it is an evaluation directive that forces expr to be resumed repeatedly, generating each successive value in turn, and executes body for each one; the loop terminates when expr fails to produce another value; there is no Boolean exit condition: the generator’s exhaustion is the exit condition), or why mixing suspend inside a procedure with top-level calls without every is the most common class of silent bug in Icon programs (a procedure that contains suspend is a generator procedure — it can be resumed after each yielded value to produce the next — but if the call site is a regular expression that succeeds after the first value, the generator procedure is effectively truncated at the first match without any indication that additional values were left unconsumed). The 13 hours of goal-directed evaluation flow analysis, generator call-site consumption audit across the program, every...do restructuring, and verification that all generator procedures are consumed completely at each call site are not visible in the diff beyond the changed loop construct.

Icon goal-directed evaluation: success, failure, generators, and the & and | operators

Icon’s evaluation model is built on two outcomes for every expression: success, which means the expression produced a value, and failure, which means the expression produced no value. This binary outcome is not a Boolean flag — it is a structural property of the evaluation that propagates through compound expressions automatically. An expression that fails causes the enclosing expression to respond according to its own evaluation rules: the & conjunction operator fails immediately if either subexpression fails; the | alternation operator tries its right subexpression when its left subexpression fails. This failure-propagation mechanism is goal-directed evaluation: the program is directed toward a goal (the success of the overall expression), and failure at any subexpression triggers automatic backtracking and alternative generation without any explicit exception handling or loop constructs. The design was deliberately chosen to make pattern matching and exhaustive search expressible as natural single expressions rather than requiring explicit loop management.

The & conjunction operator: expr1 & expr2 evaluates expr1; if it succeeds, evaluates expr2; if expr2 succeeds, the conjunction succeeds with expr2’s value; if either fails, the conjunction fails. When used with generators on the left side of &, backtracking applies: if expr2 fails, the runtime resumes expr1 to request its next generated value, then retries expr2 with the new value. This makes & a full backtracking combinator — the left generator is exhausted in service of finding a value of expr2 that succeeds. The | alternation operator: expr1 | expr2 evaluates expr1; if it fails, evaluates expr2; if expr2 succeeds, the alternation succeeds with expr2’s value; only if both fail does the alternation fail. When used as a generator combinator, | generates all values of expr1 first (exhausting it), then all values of expr2: find("a", s) | find("b", s) generates all positions of “a” in s followed by all positions of “b” in s, concatenating the two generator sequences.

A generator is an expression that can produce more than one value on demand, yielding successive values when resumed by the evaluation machinery. Built-in generators: !L generates all elements of list L in order; 1 to n (or 1 to n by step) generates integers from 1 through n; find(s1, s2) generates each position in s2 where the string s1 begins; !T generates all values of table T; key(T) generates all keys of table T; !S generates all elements of set S. The every expression drives generator exhaustion: every expr forces expr to be evaluated to complete exhaustion, requesting each successive generated value in turn without executing a body (useful when the generator produces side effects via write calls or I/O operations embedded in the generator expression itself). every expr do body executes body for each generated value of expr, and the loop terminates naturally when expr fails to produce another value. This design means there is no explicit termination condition in an every...do loop — the generator’s own exhaustion is the exit.

The critical operational implication for retainer work: when a generator expression appears in a context that is not every — for instance, as the right-hand side of an assignment, as an argument to a procedure, or as the condition of an if statement — the expression requests exactly one generated value, and evaluation stops after the first success. A call to a suspend-based generator procedure like x := generateMatches(record) assigns only the first generated match position to x; the remaining match positions are never requested, and the generator procedure is effectively abandoned in mid-execution. No error is raised. No warning is produced. The additional values simply go unrequested. This behavior is not a bug in Icon — it is precisely how goal-directed evaluation is supposed to work, because in many contexts (finding the first match, testing whether any match exists) requesting exactly one value is the correct and intended behavior. The bug arises only when the programmer intends to process all generated values but writes a non-every call site, often by analogy with languages where a procedure call always runs to completion.

suspend, fail, and generator procedures in Icon

A procedure that contains a suspend statement is a generator procedure. suspend expr suspends the procedure’s execution, yields expr’s value to the caller, and leaves the procedure alive at the point of suspension. When the caller requests the next generated value (through an every...do loop, through a | alternation backtrack, or through a & conjunction backtrack), the generator procedure resumes from immediately after the suspend statement. The procedure continues executing, potentially reaching another suspend to yield the next value, until it either falls off the end (failure — no more values) or executes an explicit fail statement (explicit failure — no more values). This is structurally different from a normal return procedure: a procedure that uses return expr terminates immediately and cannot be resumed; the generator procedure stays alive between yields and accumulates local variable state across multiple resumes.

A concrete generator procedure for generating positions of a pattern in a string: procedure findAll(pat, str); local i; every i := find(pat, str) do suspend i; end. Each call to find(pat, str) within the every...do generates the next match position, and the suspend i yields that position to the caller. The generator procedure stays alive across each yield, and each time the caller requests the next value, the every...do inside the procedure resumes, advancing find to its next position. When find exhausts all positions, the inner every...do terminates, the procedure falls off the end, and the generator fails. An unbounded generator with a repeat loop: procedure integers(n); repeat { suspend n; n +:= 1 } end produces an unbounded sequence of integers starting from n. The generator never reaches a fail and never falls off the end; the caller is responsible for stopping the consumption (by not requesting further values, or by breaking out of the every...do with a break or return in the body). Guard conditions on suspend: suspend n := expr if condition suspends with n’s value only if condition succeeds, otherwise the guard causes the suspend to fail and the generator continues to the next iteration.

The create expression and co-expressions: co := create expr creates a co-expression that evaluates expr independently. A co-expression is a first-class value — it can be stored in a variable, passed to a procedure, and activated on demand. @co activates the co-expression and retrieves its next generated value; if the co-expression has exhausted its values, @co fails. The distinction between generator procedures and co-expressions is operational: a generator procedure is driven by goal-directed evaluation at the call site (its values are requested through the normal every/|/& machinery), whereas a co-expression is driven by explicit @ activations (the caller must explicitly request each value with an @ operation). The @ operator also transmits a value to the co-expression: value @co both activates the co-expression and transmits value to it, which the co-expression receives as the result of its own pending @ operation. This ping-pong activation pattern enables coroutine-style communication between two co-expressions: each one suspends by sending a value to the other with @, and resumes when the other sends back. The co-expression deadlock bug occurs when one co-expression exhausts its computation and falls off the end without sending a final activation to the other, which blocks indefinitely waiting for an @ activation that never arrives.

The practical retainer task of generator procedure auditing: for every procedure in the program that contains a suspend statement, catalog every call site of that procedure in the entire codebase, and classify each call site as either an every...do exhaustion call (correct for processing all generated values), a generator-combinator call inside a larger | or & expression (correct if the context drives exhaustion or backtracking), a top-level expression call (truncates to first value — correct only if exactly one value is wanted), or an assignment target in a non-every context (truncates to first value — correct only if exactly one value is wanted). For each call site classified as truncating, determine whether the intent is to process one value (correct as written) or all values (must be restructured to every...do). This audit cannot be automated by a linter because the correctness of a truncating call site depends on the program’s intent, not on the syntax of the call.

Icon string scanning: find, move, pos, tab, and the scanning environment

Icon’s string scanning system is built on a scanning environment consisting of two implicit variables: a subject string (the string being scanned) and a position cursor (an integer index into the subject, with 1 pointing before the first character and *subject + 1 pointing after the last). The s ? expr construct sets the scanning subject to s and pos to 1, then evaluates expr in that scanning environment. All string scanning functions operate on the current subject and pos implicitly: they read pos to know where scanning currently stands, advance pos as they consume characters, and return the consumed substring. This implicit shared state is what makes the scanning environment feel like a first-class scanner object rather than a collection of individual function calls, and it is also what makes nested scanning environments fragile without careful isolation.

The core scanning functions and their semantics: find(s) generates each position in the current subject where string s begins, starting from the current pos (note: find(s) in a scanning context operates on the implicit subject, while find(s1, s2) outside a scanning context operates on the explicit string s2); move(n) advances pos by n characters and returns the substring from old pos to new pos — it fails if pos + n falls outside the valid range; pos(i) succeeds without consuming any characters if the current pos equals i (and fails otherwise), useful as a guard to assert position; tab(i) advances pos to position i and returns the substring from old pos to i — it fails if i is less than the current pos; =s is a shorthand that matches and consumes string s at the current position, equivalent to tab(match(s)) where match(s) succeeds with the position just after s if s occurs at the current pos. A canonical field-extraction idiom: line ? { tab(find(":")) & move(1) & word := tab(find(":")) } scans line forward to the first colon (discarding the prefix via tab), skips the colon via move(1), and captures the next field up to the second colon into word.

The character-class scanning functions: any(cset) matches one character at the current pos if it belongs to the cset and advances pos by one (fails otherwise); many(cset) matches and consumes a maximal sequence of characters all belonging to the cset, returning the consumed substring (fails if the current character is not in the cset); upto(cset) generates the positions in the current subject where a character in the cset occurs, starting from the current pos; bal(cset1, cset2, cset3) generates positions where the subject is “balanced” — the open delimiters in cset2 and close delimiters in cset3 are matched — and a cset1 character occurs. These functions are the structured replacement for SNOBOL4’s unstructured pattern matching: where SNOBOL4 built the entire pattern as a first-class pattern object that was then matched against a string, Icon decomposes the same operations into individual scanning functions that operate sequentially on the implicit cursor, making the control flow explicit and composable with standard Icon control structures.

The dynamic scoping of the scanning environment is both Icon’s most powerful string scanning feature and the most common source of scanning bugs in retainer work. The subject and pos are dynamically scoped: they are visible to any procedure called from within a ? expr block, including nested procedure calls. If a called procedure executes its own ? expr block to scan a different string, the new scanning environment (new subject and new pos) shadows the outer scanning environment for the duration of the inner ? expr. When the inner ? expr finishes, the outer subject and pos are restored. The bug: if a developer assumes that calling a procedure from inside a scanning context is safe (the inner procedure will operate on the outer subject), but the inner procedure executes its own ? expr on a local string, the inner scan silently replaces the outer scanning environment, any scanning function called from within the inner procedure operates on the inner subject rather than the outer one, and if the inner procedure modifies pos via scanning functions before the inner ? expr restores the outer environment, the outer scan resumes from the correct pos because the dynamic scoping mechanism saves and restores it. The scanning isolation fix: move procedures that do their own scanning into separate functions that explicitly receive string arguments and operate in their own scanning environments, preventing accidental subject substitution.

Icon data structures: lists, tables, sets, records, and csets

Icon’s list type is the primary sequential data structure: mutable, heterogeneous, 1-indexed, dynamically sized. L := [1, "two", 3.0] creates a list of three elements of different types. L[1] accesses the first element (Icon uses 1-based indexing); L[-1] accesses the last element (negative indices count from the end). Mutation operations: put(L, v) appends v to the end; push(L, v) prepends v at the beginning; pop(L) removes and returns the first element; pull(L) removes and returns the last element. Stack semantics use push/pop; queue semantics use put/pop. The list generator: !L generates all elements of L in order, making list iteration idiomatic as every x := !L do .... List slices: L[2:4] returns a new list containing elements 2 and 3 (the slice includes position 2 up to but not including position 4, following Icon’s string-slice convention). *L returns the current length of the list.

Tables are Icon’s hash map type. T := table(default) creates an empty table where any key lookup that finds no entry returns default instead of failing (this is different from most languages where a missing key lookup fails or raises an error). T[key] := value sets an entry; T[key] retrieves the value or returns the default if key is absent. The absence test: \T[key] succeeds only if key is actually present in the table (the \ operator fails if its operand would have returned the default, distinguishing “key present with value equal to default” from “key absent” requires a different test). key(T) generates all keys in the table in implementation-defined order. !T generates all values in the table. Sets: S := set([1, 2, 3]) creates a set from a list; member(S, v) succeeds if v is in the set; insert(S, v) adds v; delete(S, v) removes v; !S generates all elements. Record types: record Point(x, y) declares a record type named Point with fields x and y; p := Point(3, 4) creates an instance; p.x and p.y access fields. Records are mutable: p.x := 5 modifies the field in place.

The cset (character set) type is Icon’s built-in character-class type, designed to support efficient string scanning operations. A cset literal is written with single quotes: 'aeiou' is a cset containing the five lowercase vowels. Cset operations: ++ (cset union), ** (cset intersection), -- (cset difference). These operators use doubled symbols to distinguish them from their set-numeric analogs. Built-in cset constants: &lcase is all lowercase letters; &ucase is all uppercase letters; &ascii is all ASCII characters (positions 0–127); &cset is the full character set (all 256 characters in the default Icon character encoding). Csets are used primarily with scanning functions: many('0123456789') scans a maximal sequence of digits; any(&ucase) matches one uppercase letter; upto(&lcase ++ &ucase) generates positions of any alphabetic character. String operations independent of scanning: s1 || s2 concatenates two strings; s[i:j] extracts a substring from position i to position j; reverse(s) reverses a string; map(s, from, to) translates characters in s by replacing each character that appears in from with the corresponding character in to (equivalent to POSIX tr); string(x) converts any value to its string representation; integer(s) converts a string to an integer (fails if the string is not a valid integer representation).

Icon’s design legacy: goal-directed evaluation and the SNOBOL/Icon family

Icon’s direct predecessor, SNOBOL4, was developed at Bell Labs in 1967 by Ralph Griswold, Ivan Polonsky, and James Poage. SNOBOL4 made pattern matching with backtracking the central operation of the language: a SNOBOL4 statement typically consists of a subject string, a pattern, and an optional replacement, and the pattern is matched against the subject with full backtracking. Patterns in SNOBOL4 are first-class values that can be constructed, stored in variables, and combined with concatenation and alternation operators. SNOBOL4’s string processing capabilities were genuinely powerful — it was the dominant language for text analysis and natural language processing experiments in the late 1960s and 1970s — but the language was otherwise impoverished: no structured data structures beyond arrays, limited control flow (SNOBOL4’s primary control structure was GOTO on success or failure), and no modular program organization beyond labels and GOTO. Griswold’s rationale for redesigning SNOBOL4 as Icon was to preserve the power of backtracking pattern search while embedding it in a language with modern structured programming constructs, data structures, and modular procedure organization.

Icon’s key contributions to programming language design: goal-directed evaluation as a general programming model, not just a pattern matching mechanism; generators as first-class computations that produce sequences of values on demand; co-expressions as independently activatable computations that enable coroutine-style programming without requiring a separate threading system; and string scanning as a structured replacement for SNOBOL4’s unstructured pattern objects. These contributions influenced several later language designs. Python’s generators and the yield statement are a direct descendant of Icon’s suspend semantics: Guido van Rossum has cited Icon as an influence on Python’s generator design. Ruby’s Enumerator and lazy evaluation lazy chains draw on similar ideas about sequences of values computed on demand. Haskell’s lazy evaluation achieves similar effects through different machinery: instead of explicit suspend/resume, lazy evaluation defers computation of any value until it is demanded, effectively making every computation a generator. Icon’s every...do anticipates Python’s for...in loop over generators: in both cases, the loop construct exhausts a sequence of generated values without requiring the programmer to manage the iteration state explicitly.

Unicon (Unified Icon, developed from 1994 onward) extends Icon with classes, inheritance, network I/O, database access, and threads while preserving goal-directed evaluation semantics. The Unicon thread model adds shared-memory concurrency on top of Icon’s co-expression model, with mutexes and condition variables for synchronization. A key retainer task for Unicon programs: co-expression-style ping-pong communication patterns that worked correctly in Icon’s single-threaded model can deadlock in Unicon’s threaded model if a thread exhausts its co-expression without sending the final activation to the partner thread. The diagnostic pattern is the same as in single-threaded Icon (one side exhausted, one side blocked on @), but the fix may require thread-level synchronization rather than co-expression termination protocol redesign. The canonical references: the Icon Programming Language (Griswold and Griswold, 1990, third edition) covers Icon 9.x comprehensively; the Programming with Unicon book (Jeffery, 2023) covers Unicon’s class and thread extensions. Icon programs are particularly natural for combinatorial enumeration: an every loop that generates all permutations, all subsets, or all solutions to a constraint problem requires no explicit stack management — the goal-directed evaluation machinery and generator composition handle the search tree implicitly.

How HourTab tracks Icon developer retainer hours

Icon retainer work shares the invisible-work problem common to all language engineering retainers, with the additional challenge that Icon’s most consequential retainer tasks — generator call-site consumption audit, goal-directed evaluation flow analysis, string scanning dynamic scope isolation, and co-expression activation protocol redesign — produce diffs whose surface area is not proportional to the analytical work. Restructuring a generator truncation bug is a diff that inserts the word every and a do keyword around an existing assignment; the value is correct processing of all generated match positions rather than only the first, elimination of silent match discard for all records containing multiple matches, and a call-site design that is portable to any context where the generator procedure is called in the future. Fixing a co-expression deadlock is a diff that adds an explicit termination sentinel value and a termination-detection check in one co-expression’s activation loop; the value is correct program termination on all inputs, elimination of indefinite hanging when the first co-expression exhausts its computation, and an activation protocol that handles the end-of-sequence case that the original ping-pong design did not address.

HourTab gives Icon developers a public retainer-hours URL they send to clients — typically text processing groups using Icon for structured data extraction (Icon was used in academic computing, text formatting pipelines, and data transformation throughout the 1980s and 1990s, and its string scanning capabilities made it particularly suited to field-structured log parsing and configuration file processing), language research groups studying goal-directed evaluation and generator semantics, and compiler engineering teams working on Unicon (Icon’s actively maintained successor with an active open-source community) — at the start of an engagement. For Icon retainers, each work log entry should name the mechanism (goal-directed evaluation & conjunction and | alternation flow; suspend generator procedure yield design; every...do exhaustion loop restructuring; create/@ co-expression activation; ? expr scanning environment; find/move/tab/pos scanning function sequencing; any/many/upto/bal character-class match; !list list generator; key(table) table key generator; || string concatenation; map(s, from, to) character translation), the specific procedure name and truncated generator call site, and the before/after observable metric. Icon retainers are often compared to SNOBOL developer retainers for string-processing language engineering (both languages are in the SNOBOL/Icon family and share a text-processing focus), to Haskell developer retainers for lazy evaluation and generator semantics (both languages model computation as sequences of values produced on demand, with the control flow driven by the consumer rather than the producer), and to Python developer retainers for yield/generator programming (Python’s generator model is a direct descendant of Icon’s suspend semantics, and many retainer tasks — diagnosing generator truncation, auditing call-site consumption, restructuring exhaustion loops — are structurally parallel across the two languages). HourTab’s work log makes the goal-directed evaluation analysis, generator call-site consumption audit, and every...do restructuring visible to clients who would otherwise see only the symptom — missed matches or wrong field extractions — and not understand why the fix required understanding Icon’s evaluation model at the level of how many values a given call site requests.

FAQ: Icon developer retainers