Microsoft Promptflow developer on retainer: @tool evaluation flow missing is_aggregation=True aggregation node — per-row faithfulness scores stored in shared Python dict — last-row score (0.84) displayed as aggregate for 5 months — actual average faithfulness 0.59 — major October 2026 regression missed — Promptflow developer on monthly retainer

promptflow 1.8.0 • Python 3.11 • Azure OpenAI GPT-4o-mini • FAISS • FastAPI 0.111.0 • RAG customer support Q&A pipeline • Azure ML managed online endpoint • @tool eval node no is_aggregation=True • module-level _results dict not shared across batch runner processes • last-row score 0.84 displayed as aggregate • 5 months of misleading metric • HNSW migration drops real average from 0.71 to 0.56 • regression undetected • fix: @tool(is_aggregation=True) aggregation node • post-fix average_faithfulness 0.562 • retroactive delta −0.148 • 3.0h retainer engagement

Verdana Research Group builds a customer support Q&A pipeline backed by a RAG architecture: incoming support questions are answered by retrieving relevant passages from a product documentation corpus via FAISS and passing them to Azure OpenAI GPT-4o-mini to generate a grounded answer. The answer-generation flow runs as an Azure ML managed online endpoint. Separately, the team runs a Promptflow batch evaluation flow every week against a fixed 500-row test dataset to measure faithfulness — whether each generated answer is grounded in the retrieved context rather than hallucinated. This weekly eval run is their primary signal for the health of the RAG pipeline. For five months, from May through October 2026, the Azure ML Studio eval dashboard showed a single faithfulness metric: 0.84. The team reported this number in weekly team meetings as evidence that their pipeline was performing well. In October 2026, they migrated their FAISS index from a flat index to an HNSW index. User satisfaction scores began declining over the following three weeks. They brought in a Promptflow retainer developer in November 2026 to explain why user-reported answer quality had dropped when their eval metric showed no change. The explanation was in the eval flow code.

Their evaluation flow used a single @tool decorated function in eval_flow/compute_faithfulness.py to score each row. The function called GPT-4o-mini as an LLM judge and returned a float between 0.0 and 1.0. The engineer who wrote the function had intended to accumulate scores in a module-level dict for later averaging, but had not written a second aggregation node to perform that averaging:

# eval_flow/compute_faithfulness.py
# Verdana Research Group
# promptflow 1.8.0, Python 3.11

from promptflow import tool

# Shared dict — intended to accumulate results for averaging
_results = {}

@tool
def compute_faithfulness(answer: str, context: str, row_id: str) -> float:
    """Score faithfulness of answer given context using GPT-4o-mini judge."""

    score = call_llm_judge(answer, context)  # returns 0.0-1.0

    # Store in shared dict for later aggregation
    _results[row_id] = score

    # Return score for this row
    return score

There are two problems here, and they compound each other. The first is the module-level _results dict. Promptflow’s batch runner does not guarantee that all rows execute within the same process. In practice with promptflow 1.8.0, batch evaluation runs items in worker threads or separate processes depending on the execution configuration and the number of workers. The module-level _results dict is not shared across process boundaries: each worker process has its own copy of the module, and writes to _results in one process are invisible to other processes. Even if all rows happened to run in the same process, the dict would accumulate correctly — but there is no code anywhere in the flow that reads it back out and computes an average. The accumulation was dead code from the start.

The second problem, and the one responsible for the misleading metric, is the missing @tool(is_aggregation=True) aggregation node. In Promptflow 1.8.0, a batch eval run works as follows: for each row in the input JSONL, the framework runs all standard @tool nodes and collects their outputs into a per-row output record that is appended to outputs.jsonl. After all rows complete, the framework looks for any nodes decorated with @tool(is_aggregation=True). If found, the aggregation node is invoked once with a list of all per-row values for each of its input parameters, and its return dict becomes the aggregate metric record displayed in Azure ML Studio under the “Evaluation metrics” panel. If no aggregation node exists, Promptflow has no computed aggregate to display. The observed behavior in Azure ML Studio under this condition is that the metric display shows the output value from the last row to complete in the batch — the final write to the output buffer before the run finishes. For Verdana’s 500-row dataset, the last row is always row 500: a simple, short test case asking “What is the return policy?” that has a precise, complete answer in the product documentation. This row has consistently scored 0.84 across every weekly eval run for five months. The Azure ML Studio faithfulness metric has therefore shown 0.84 for every one of those runs, regardless of what was happening to the other 499 rows.

The retainer developer’s first step was to open the most recent batch eval run output JSONL and compute the actual mean manually:

import json
import statistics

with open("~/.promptflow/runs/eval_w42/outputs.jsonl") as f:
    rows = [json.loads(line) for line in f]

scores = [row["faithfulness"] for row in rows if row.get("faithfulness") is not None]
print(f"Count: {len(scores)}")           # Count: 500
print(f"Mean: {statistics.mean(scores):.3f}")     # Mean: 0.562
print(f"Stdev: {statistics.stdev(scores):.3f}")   # Stdev: 0.189
print(f"Last row score: {rows[-1]['faithfulness']:.3f}")  # Last row score: 0.840

The real average faithfulness for the week-42 run: 0.562. The value displayed in Azure ML Studio for the same run: 0.840. The difference is 0.278 points — driven entirely by the last row in the dataset being an unusually easy test case that was never representative of the full distribution. The standard deviation of 0.189 further indicated that the distribution was wide: many rows were scoring significantly below the mean, while the easy last row sat at the far right tail.

To understand the full history, the retainer developer ran the same analysis across the outputs.jsonl files for the past eight weekly eval runs — four from before the HNSW migration (weeks 38–41) and four from after (weeks 42–45):

import json, statistics, glob, os

run_dir = os.path.expanduser("~/.promptflow/runs")
runs = sorted(glob.glob(f"{run_dir}/eval_w*/outputs.jsonl"))

for run_path in runs:
    week = run_path.split("/eval_")[1].split("/")[0]
    with open(run_path) as f:
        rows = [json.loads(line) for line in f]
    scores = [r["faithfulness"] for r in rows if r.get("faithfulness") is not None]
    last = rows[-1]["faithfulness"]
    print(f"{week}: mean={statistics.mean(scores):.3f}  last_row={last:.3f}  n={len(scores)}")

# eval_w38: mean=0.708  last_row=0.840  n=500   # pre-HNSW
# eval_w39: mean=0.714  last_row=0.840  n=500   # pre-HNSW
# eval_w40: mean=0.711  last_row=0.840  n=500   # pre-HNSW
# eval_w41: mean=0.709  last_row=0.840  n=500   # pre-HNSW  (HNSW migration week 41)
# eval_w42: mean=0.562  last_row=0.840  n=500   # post-HNSW
# eval_w43: mean=0.558  last_row=0.840  n=500   # post-HNSW
# eval_w44: mean=0.561  last_row=0.840  n=500   # post-HNSW
# eval_w45: mean=0.563  last_row=0.840  n=500   # post-HNSW

The picture was unambiguous. Pre-HNSW average faithfulness: approximately 0.710. Post-HNSW average faithfulness: approximately 0.562. The delta was −0.148 — a 20% relative drop. The displayed Azure ML Studio metric for all eight runs: 0.840, unchanged. The FAISS HNSW index with M=16, efConstruction=200 had reduced retrieval recall for longer, more complex queries — the kind that require finding multiple relevant passages across a large corpus. HNSW’s approximate nearest-neighbor search with these parameters missed roughly 18% of relevant documents that the flat exhaustive index would have returned, and the missing context caused GPT-4o-mini to generate less grounded answers. Short queries like “What is the return policy?” were unaffected because their relevant documents were the nearest neighbors by a wide margin and were found regardless of index type. The team had shipped the HNSW migration to production on the strength of an unchanged 0.84 faithfulness metric, never knowing the real average had fallen by nearly 0.15 points.

The fix required two changes: adding a proper aggregation node and wiring it into the flow DAG. First, the aggregation node itself:

# eval_flow/aggregate_metrics.py
from typing import List
from promptflow import tool

@tool(is_aggregation=True)
def aggregate_faithfulness(faithfulness_scores: List[float]) -> dict:
    """Aggregate per-row faithfulness scores into summary statistics."""
    valid_scores = [s for s in faithfulness_scores if s is not None]
    if not valid_scores:
        return {"average_faithfulness": None, "count": 0}
    return {
        "average_faithfulness": sum(valid_scores) / len(valid_scores),
        "count": len(valid_scores),
        "p25_faithfulness": sorted(valid_scores)[len(valid_scores) // 4],
        "p75_faithfulness": sorted(valid_scores)[3 * len(valid_scores) // 4],
    }

The @tool(is_aggregation=True) decorator is the key. It tells Promptflow that this node should not be executed once per row like a standard @tool node. Instead, after all rows in the batch complete, Promptflow collects the values of the faithfulness_scores input across all rows — meaning the faithfulness output from compute_faithfulness for every row in the JSONL — and passes them as a single Python list to this function. The function runs once per batch run, returns a dict, and that dict becomes the aggregate metric record in Azure ML Studio. This is the correct architectural pattern: per-row computation in a standard @tool node, aggregate computation in a dedicated @tool(is_aggregation=True) node that receives the full collection of per-row outputs.

The flow DAG YAML required a corresponding update to declare the new aggregation node and wire its input to the per-row node’s output:

# eval_flow/flow.dag.yaml
# Updated to include aggregation node wired to compute_faithfulness output

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
inputs:
  answer:
    type: string
  context:
    type: string
  row_id:
    type: string
outputs:
  faithfulness:
    type: number
    reference: ${compute_faithfulness.output}
nodes:
- name: compute_faithfulness
  type: python
  source:
    type: code
    path: compute_faithfulness.py
  inputs:
    answer: ${inputs.answer}
    context: ${inputs.context}
    row_id: ${inputs.row_id}
- name: aggregate_faithfulness
  type: python
  source:
    type: code
    path: aggregate_metrics.py
  aggregation: true
  inputs:
    faithfulness_scores: ${compute_faithfulness.output}

The aggregation: true flag in the node definition is the YAML-side counterpart to the @tool(is_aggregation=True) Python decorator. Both are required: the decorator tells the Python runtime the function is an aggregation node, and the YAML flag tells the Promptflow orchestrator to route the per-row outputs to this node as a collected list after the full batch completes rather than passing a single row’s data to it. The inputs: faithfulness_scores: ${compute_faithfulness.output} mapping specifies that the faithfulness_scores parameter should receive the output value of the compute_faithfulness node — which means the list of all 500 per-row float outputs from that node.

With the fix deployed, the retainer developer re-ran the current week’s eval batch against the 500-row dataset. The Azure ML Studio metric panel now showed four values instead of one:

# Azure ML Studio — Evaluation metrics panel — eval_w46 (post-fix)
# All values from aggregate_faithfulness() return dict:

average_faithfulness: 0.562
count: 500
p25_faithfulness: 0.420
p75_faithfulness: 0.710

# Previous display (eval_w42 through eval_w45, no aggregation node):
# faithfulness: 0.840  ← last-row output, not an aggregate

The average_faithfulness of 0.562 matched the manually computed mean from the JSONL analysis precisely. The team could also now see the interquartile range: p25 at 0.420 and p75 at 0.710, indicating a wide spread of faithfulness performance across the dataset that the single last-row value had completely obscured. The count: 500 confirmed all rows were scored and none were dropped.

With correct metrics established, the team needed to decide what to do about the HNSW migration. The retroactive analysis was conclusive: pre-migration average faithfulness 0.710, post-migration 0.562, delta −0.148. The HNSW parameters M=16, efConstruction=200 had been chosen for their query latency improvement — roughly 40% faster at inference time compared to the flat index — but the recall cost was too high for complex multi-hop queries. The team decided to revert to a flat-style search with FAISS’s IndexIVFFlat configured with efSearch=128, which restores exhaustive recall at a modest latency increase compared to HNSW. After deploying the reverted index configuration, the week-47 eval run showed average_faithfulness: 0.695 — near the pre-migration baseline of 0.710, with the small remaining gap attributable to minor dataset drift rather than index recall differences. User satisfaction scores recovered to their pre-regression level over the following two weeks.

The postmortem documented three process failures. First, the eval flow had never been validated: no one had compared the Azure ML Studio displayed metric to the manually computed mean over outputs.jsonl before relying on it in weekly reports. A one-time validation check at eval flow creation would have caught the missing aggregation node immediately. Second, the HNSW migration had no pre/post eval run gate: the team shipped the new index without running a Promptflow batch eval on a held-out dataset before and after the migration and confirming that average faithfulness did not drop by more than an acceptable threshold. Third, the eval dataset was never randomized: the 500 rows were in fixed order, meaning the last row was always the same easy test case. Random ordering on each run would have made the last-row-as-aggregate bug self-evident within two or three runs, since the displayed metric would fluctuate wildly rather than staying locked at 0.84.

Total retainer engagement: 3.0 hours (0.5h initial triage: review Azure ML Studio eval dashboard, note static 0.84 metric across all historical runs, open latest eval run output JSONL at ~/.promptflow/runs/eval_w42/outputs.jsonl, compute statistics.mean() over 500 rows, confirm mean is 0.562 vs displayed 0.840; 1.0h root cause: read eval_flow/compute_faithfulness.py, identify plain @tool decorator with no is_aggregation=True, trace Promptflow’s batch aggregation mechanism in 1.8.0 documentation, confirm that without an aggregation node the displayed metric is the last-row output, extend the JSONL analysis to 8 historical weekly runs to compute pre-HNSW vs post-HNSW averages, identify −0.148 regression introduced in week 42; 1.0h fix: write eval_flow/aggregate_metrics.py with @tool(is_aggregation=True) returning mean, count, p25 and p75, update eval_flow/flow.dag.yaml to declare the aggregation node with aggregation: true and wire faithfulness_scores input to ${compute_faithfulness.output}, re-run week-46 eval batch, verify Azure ML Studio now shows average_faithfulness: 0.562, confirm count is 500; 0.5h validation and documentation: retroactive runs for 4 weeks before and after HNSW migration confirm −0.148 delta, revert FAISS index to efSearch=128 flat configuration, run week-47 eval confirming recovery to 0.695, write postmortem with three process failure findings and remediation checklist).

Related retainer posts

• Humanloop developer on retainer — startup-cached prompt not refreshed, GKE pods serve stale v7 for 12 days after v8 deployed
• Braintrust developer on retainer — ZeroDivisionError drops 20% of coherence scores when async LLM judge sub-evals time out
• Opik developer on retainer — dev and prod traces merge in Default Project, 68% vs 88% correctness