Hamilton developer on retainer: config.env_overrides module loaded second overrides usd_conversion_rate node — os.environ.get(‘USD_CONVERSION_RATE’, ‘1.0’) default returns 1.0 instead of 0.01 in production — CFO board pack shows $12.4M MRR instead of $124k — Hamilton developer on monthly retainer

Hamilton 0.23.1 • Python 3.11 • AWS EC2 • revenue_pipeline/driver.py • usd_conversion_rate() defined in two modules • config.env_overrides loaded second, overrides computation.units • USD_CONVERSION_RATE env var unset in production • os.environ default '1.0' • monthly_revenue_usd 100× inflated • Q3 board pack $12.4M vs $124k MRR • 4.0h retainer engagement

On October 22, 2026, the CFO at Meridian Analytics LLC sent an urgent Slack message to the data engineering team: the Q3 board pack showed $12.4M in MRR, but the sales team’s manual count from the CRM was $124k. The two-orders-of-magnitude discrepancy had gone unnoticed for seven weeks because the dashboard had always shown large numbers, and nobody had compared them to an independent source until the board prep cycle. The Hamilton retainer developer was brought in the same afternoon and identified the root cause in under an hour: two Python modules in the Hamilton revenue pipeline both defined a function named usd_conversion_rate(), and the configuration module was loaded second in the driver.Builder() call, silently overriding the correct constant with an environment-variable lookup that returned 1.0 instead of 0.01 in production.

Meridian Analytics built their revenue reporting pipeline using Hamilton, the functional dataflow framework from DAGWorks. In Hamilton, each Python function defines a node in the DAG. The framework resolves dependencies by matching function argument names to other functions’ return names and types. A function named monthly_revenue_usd that takes raw_revenue_cents: pd.Series and usd_conversion_rate: float as arguments will automatically receive the outputs of the functions raw_revenue_cents() and usd_conversion_rate() when executed. This automatic wiring makes Hamilton pipelines easy to reason about in isolation — but it creates a subtle failure mode when multiple modules define functions with the same name.

# computation/units.py — Meridian Analytics LLC
# Hamilton 0.23.1, Python 3.11
# Defines the cents-to-dollars conversion factor as a constant.

def usd_conversion_rate() -> float:
    """Convert raw_revenue_cents to USD. Cents → dollars = multiply by 0.01."""
    return 0.01


def monthly_revenue_usd(raw_revenue_cents: pd.Series, usd_conversion_rate: float) -> pd.Series:
    """Monthly revenue in USD, computed from raw_revenue_cents."""
    return raw_revenue_cents * usd_conversion_rate

This computation module was correct and had been in production since January 2026. In August 2026, a junior engineer added a configuration module to allow environment-specific overrides of pipeline parameters. The module was designed to make the currency conversion rate configurable so that a hypothetical multi-currency pipeline could be tested without code changes. The engineer named the override function usd_conversion_rate() to match the node it was intended to replace, which is the standard Hamilton override pattern. The critical mistake was setting the default value to ‘1.0’ — a string that coerced to float(1.0) — rather than ‘0.01’.

# config/env_overrides.py — added August 2026
# Reads pipeline configuration from environment variables.
# Intended to allow environment-specific overrides in CI and staging.

import os

def usd_conversion_rate() -> float:
    # BUG: default is '1.0', not '0.01'.
    # In local development, developer sets USD_CONVERSION_RATE=0.01 in .env.
    # In production Docker image, USD_CONVERSION_RATE is NOT set.
    # os.environ.get returns '1.0', coerced to float(1.0).
    # Downstream: monthly_revenue_usd = raw_revenue_cents * 1.0 (no conversion).
    return float(os.environ.get('USD_CONVERSION_RATE', '1.0'))

The driver was updated on September 1, 2026 to include the new config module:

# revenue_pipeline/driver.py — updated September 1, 2026

from hamilton import driver
from computation import units, aggregations, segmentation
from config import env_overrides  # added September 1 — BUG: loaded after units

dr = (
    driver.Builder()
    .with_modules(units, aggregations, segmentation, env_overrides)  # env_overrides last
    .build()
)

# Hamilton's last-in-wins rule:
# Both computation.units and config.env_overrides define usd_conversion_rate() -> float.
# Hamilton loads modules in the order they appear in with_modules().
# config.env_overrides is last → its usd_conversion_rate() overrides computation.units.
# In production: USD_CONVERSION_RATE env var not set → os.environ.get default '1.0' → 1.0.
# monthly_revenue_usd = raw_revenue_cents * 1.0 (raw cents, not dollars).
# September revenue: 1,240,000,000 cents reported as $12,400,000 instead of $124,000.

In local development, the engineer had set USD_CONVERSION_RATE=0.01 in a .env file loaded by python-dotenv. All local tests passed. The CI pipeline ran without setting the environment variable, but the CI tests only checked that the driver built successfully and that individual nodes returned the correct types — they did not assert that usd_conversion_rate() returned 0.01 in an environment without the variable set. The production Docker image built from a base image that did not include the .env file. From September 1 through October 22, 2026 — seven weeks — every monthly_revenue_usd computation in production used usd_conversion_rate = 1.0, reporting revenue in cents as if it were dollars.

The Hamilton retainer developer diagnosed the collision using Hamilton’s graph visualization tools. The driver.display_all_functions() call revealed that only one usd_conversion_rate node was present in the resolved DAG, and the originating_functions attribute showed it resolved from config.env_overrides, not computation.units. The developer verified that USD_CONVERSION_RATE was not set in the production container environment by checking the ECS task definition, confirming that the env var was absent and the default was active.

# Diagnosis: inspect which module owns each node in the resolved DAG
import hamilton.driver
from computation import units, aggregations, segmentation
from config import env_overrides

dr = driver.Driver({}, units, aggregations, segmentation, env_overrides)

# List all nodes and their originating functions
for node in dr.list_available_variables():
    fn = dr.graph.nodes[node.name].originating_functions
    if fn:
        module = fn[0].__module__
        print(f"{node.name:40s} ← {module}")

# Output (truncated):
# usd_conversion_rate                      ← config.env_overrides   ← BUG
# monthly_revenue_usd                      ← computation.units
# raw_revenue_cents                        ← computation.ingestion

# Confirmed: usd_conversion_rate resolved from config.env_overrides, not computation.units.
# In production, USD_CONVERSION_RATE unset → default 1.0 → 100x inflation.

The fix involved three steps. First, the config/env_overrides.py function was renamed to usd_conversion_rate_env_override() to prevent it from conflicting with the canonical node. Second, a new node was added to explicitly choose between the constant and the override based on whether the environment variable was set, with a hard assertion that the result was less than 1.0:

# config/env_overrides.py — fixed October 22, 2026

import os

def usd_conversion_rate_env_override() -> float | None:
    """Returns env-var override if set, else None. Does NOT conflict with usd_conversion_rate."""
    raw = os.environ.get('USD_CONVERSION_RATE')
    if raw is None:
        return None
    return float(raw)


# computation/units.py — fixed October 22, 2026

def usd_conversion_rate(usd_conversion_rate_env_override: float | None) -> float:
    """
    Cents-to-dollars conversion factor.
    Uses env override if provided (must be < 1.0), otherwise 0.01.
    """
    rate = usd_conversion_rate_env_override if usd_conversion_rate_env_override is not None else 0.01
    if rate >= 1.0:
        raise ValueError(
            f"usd_conversion_rate={rate} is >= 1.0. "
            "This is almost certainly wrong (cents-to-dollars conversion should be 0.01). "
            "If you intended a different currency unit, set USD_CONVERSION_RATE explicitly."
        )
    return rate

Third, the CI pipeline was updated with a Hamilton driver integration test that instantiates the full production driver without any environment variables set and asserts that usd_conversion_rate resolves to 0.01:

# tests/test_driver_integration.py — added October 22, 2026

import pytest
from hamilton import driver
from computation import units, aggregations, segmentation
from config import env_overrides

def test_usd_conversion_rate_default_is_correct(monkeypatch):
    """
    Verify that usd_conversion_rate resolves to 0.01 when USD_CONVERSION_RATE is not set.
    Guards against accidental module override introducing wrong default.
    """
    monkeypatch.delenv('USD_CONVERSION_RATE', raising=False)
    dr = driver.Driver({}, units, aggregations, segmentation, env_overrides)
    result = dr.execute(['usd_conversion_rate'])
    assert result['usd_conversion_rate'] == pytest.approx(0.01), (
        f"usd_conversion_rate should be 0.01 (cents-to-dollars), got {result['usd_conversion_rate']}. "
        "Check for accidental module override in driver.Builder() module list."
    )

def test_usd_conversion_rate_env_override(monkeypatch):
    """Verify that USD_CONVERSION_RATE env var is respected when explicitly set."""
    monkeypatch.setenv('USD_CONVERSION_RATE', '0.01')
    dr = driver.Driver({}, units, aggregations, segmentation, env_overrides)
    result = dr.execute(['usd_conversion_rate'])
    assert result['usd_conversion_rate'] == pytest.approx(0.01)

The retainer developer also added a quarterly module audit task to the pipeline maintenance runbook: before any new module is added to a Hamilton driver.Builder() call, a CI job scans the module for function names that already exist in the other modules in the driver, and raises an error if a duplicate is found without an explicit # INTENTIONAL_OVERRIDE: replaces {module}.{function} comment. This prevents future silent overrides from reaching production.

The Q3 revenue metrics were restated across all seven affected weeks. The correct MRR was $124k, not $12.4M. The board pack was corrected before the quarterly review. Total retainer engagement: 4.0 hours (0.5h initial diagnosis from CFO report; 1.0h trace override chain using Hamilton graph visualization; 0.5h verify env var absence in production ECS task; 1.0h rename config function and add validation node; 0.5h CI integration test; 0.5h metric restatement and documentation).

Retainer billing was logged to HourTab with the Hamilton driver version, all module names, the override chain, and the root cause. The client could verify the complete 4.0-hour work log at the shared HourTab URL without logging in.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• ZenML developer on retainer — pipeline orchestration retainer work
• Kubeflow developer on retainer — ML pipeline platform retainer work