OpenVINO developer on retainer: compiled_model() matches inputs by tensor name — tokenizer output key attention_mask does not match optimized ONNX node name — attention mask silently passed as zeros — BERT attends to padding tokens — accuracy 89%→61% on variable-length inputs — 3 weeks — OpenVINO developer on monthly retainer

openvino 2024.0 • Python 3.11 • Intel Core i7-12700 • BERT-base-uncased classification • ONNX opset 13 export • compiled_model(tokenizer_dict) • attention_mask key does not match optimized ONNX node name • zeros passed silently • BERT attends to [PAD] tokens • [CLS] representation diluted • accuracy 89% fixed-length • accuracy 61% variable-length • developer checks model, ONNX opset, tokenizer — 3 weeks • fix: explicit index-based input mapping • accuracy 61%→89% on variable-length inputs • 1.5h retainer engagement

Eastfield Edge Inference Ltd. deployed a BERT-base-uncased text classification model on Intel CPU hardware using OpenVINO Runtime 2024.0. They exported the Hugging Face model to ONNX with Optimum (opset 13), converted to OpenVINO IR with ov.convert_model(), compiled for CPU, and called inference by passing the Hugging Face tokenizer output dict directly: compiled_model(tokenizer(text, return_tensors='np')). On their evaluation set of fixed-length 512-token sequences, classification accuracy was 89%. When they tested on their production workload of variable-length texts (ranging from 20 to 200 tokens), accuracy dropped to 61%. The developer spent three weeks checking the model architecture, ONNX opset version, tokenizer max length settings, and normalization parameters before engaging an OpenVINO retainer developer.

The inference pipeline at the time of the engagement:

# inference.py — Eastfield Edge Inference Ltd.
# openvino 2024.0, Python 3.11, Intel Core i7-12700
# BERT-base-uncased classification, ONNX opset 13

import numpy as np
import openvino as ov
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
core = ov.Core()
ov_model = core.read_model("bert_classifier.xml")
compiled_model = core.compile_model(ov_model, "CPU")


def classify(text: str) -> int:
    # Tokenize: returns dict with keys 'input_ids', 'attention_mask', 'token_type_ids'
    inputs = tokenizer(text, return_tensors="np", padding="max_length",
                       max_length=512, truncation=True)

    # BUG: passing dict directly — matched by tensor name
    # OpenVINO 2024.0 iterates dict keys and calls compiled_model.inputs[i].get_any_name()
    # After ov.convert_model() const-folding and shape inference passes,
    # the internal ONNX node name for attention_mask may differ from 'attention_mask'
    # No error is raised — unmatched inputs receive a zero numpy array
    results = compiled_model(dict(inputs))   # ← silent name mismatch for attention_mask

    logits = results[compiled_model.output(0)]
    return int(np.argmax(logits, axis=-1)[0])


# Test results:
# Fixed-length inputs (512 tokens, no padding):
#   attention_mask is all-ones (no [PAD] tokens) → zero mask = full attention = same result
#   Classification accuracy: 89%
#
# Variable-length inputs (20–200 tokens, 312–492 [PAD] tokens):
#   attention_mask should be 1 for real tokens, 0 for [PAD] tokens
#   Actual attention_mask: all zeros (silently replaced by OpenVINO due to name mismatch)
#   BERT all-zero attention mask: every token attends to every other token
#   [PAD] tokens receive attention weight; [CLS] diluted by 312–492 padding tokens
#   Classification accuracy: 61%

Why OpenVINO’s name-based input matching silently passes zeros for unmatched inputs

When compiled_model(inputs) receives a Python dict, OpenVINO 2024.0 matches each dict key to a model input by calling get_any_name() on each ConstOutput in compiled_model.inputs. If a dict key matches a model input name exactly, that tensor is used. If no match is found, OpenVINO silently uses a zero-initialized numpy array of the correct shape for that input. No exception is raised. The inference request succeeds and returns output tensors.

The reason the input names diverge is that OpenVINO’s model conversion pipeline runs optimization passes including constant folding, dead node elimination, and shape inference. These passes can rename ONNX nodes, replace them with fused operations, or re-index them. The original ONNX graph exported by Optimum has an attention_mask node named attention_mask, but after OpenVINO’s optimization, the node that corresponds to the attention mask may be renamed to an internal name like onnx::Unsqueeze_1 or /bert/embeddings/Gather_1. The tokenizer output key attention_mask no longer matches any get_any_name() call, and the input receives zeros.

The reason the bug only manifests on variable-length inputs is a coincidence of how attention masking works in BERT. When all 512 tokens are real tokens (no padding), the attention mask is all ones. An all-zeros mask allows every token to attend to every other token—which is the same as the all-ones mask for sequences without padding. The outputs are identical. When the input has 100 real tokens and 412 padding tokens, the correct attention mask has 100 ones and 412 zeros. The all-zeros mask (from the name mismatch) allows the 100 real tokens to attend to the 412 padding tokens. The [CLS] representation becomes diluted by attention from 412 uninformative [PAD] tokens. Classification accuracy degrades in proportion to the fraction of the sequence that is padding.

# Diagnosing the input name mismatch
# Print all compiled model input names and compare to tokenizer output keys

import openvino as ov
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
core = ov.Core()
ov_model = core.read_model("bert_classifier.xml")
compiled_model = core.compile_model(ov_model, "CPU")

# Step 1: Print all model input names
print("Compiled model inputs:")
for i, inp in enumerate(compiled_model.inputs):
    print(f"  [{i}] name='{inp.get_any_name()}' shape={inp.shape} dtype={inp.element_type}")

# Output (example after optimization):
# Compiled model inputs:
#   [0] name='input_ids'          shape=[1,512] dtype=i64
#   [1] name='onnx::Unsqueeze_1'  shape=[1,512] dtype=i64   ← was 'attention_mask'
#   [2] name='token_type_ids'     shape=[1,512] dtype=i64

# Step 2: Print tokenizer output keys
inputs = tokenizer("test sentence", return_tensors="np", padding="max_length",
                   max_length=512, truncation=True)
print("\nTokenizer output keys:", list(inputs.keys()))
# Output: Tokenizer output keys: ['input_ids', 'attention_mask', 'token_type_ids']

# Step 3: Check what input[1] receives when passing tokenizer dict
import numpy as np
test_inputs = dict(inputs)
# OpenVINO cannot find 'onnx::Unsqueeze_1' in test_inputs.keys()
# Assigns zero array of shape [1, 512] to input[1]
print("\nInput[1] name 'onnx::Unsqueeze_1' not in tokenizer keys:",
      'onnx::Unsqueeze_1' not in test_inputs)
# Output: Input[1] name 'onnx::Unsqueeze_1' not in tokenizer keys: True

# Step 4: Verify by checking attention_mask sum in inference
results = compiled_model(test_inputs)
# No error — zeros passed silently to input[1] (the attention_mask)

# Add a zero-mask check to catch this in production:
attention_mask = inputs['attention_mask']
assert attention_mask.sum() > 0, (
    f"attention_mask is all zeros — check input name mapping in compiled_model"
)

The fix: explicit index-based input mapping

The fix replaces dict-based name matching with explicit index-based mapping. Each tokenizer output is assigned to the corresponding compiled_model.inputs[i] by position, not by name. This is robust to node renaming during optimization. A pre-inference assertion checks that the attention mask is not all zeros, catching any future mapping errors at inference time.

# inference.py — FIXED (Eastfield Edge Inference Ltd.)
# openvino 2024.0, Python 3.11, Intel Core i7-12700

import numpy as np
import openvino as ov
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
core = ov.Core()
ov_model = core.read_model("bert_classifier.xml")
compiled_model = core.compile_model(ov_model, "CPU")

# FIXED: build explicit index mapping at load time
# Print input names for documentation; use indices for inference
INPUT_NAMES = [inp.get_any_name() for inp in compiled_model.inputs]
print(f"Compiled model input names (for reference): {INPUT_NAMES}")
# Verify the order matches tokenizer output order (input_ids, attention_mask, token_type_ids)
# before deploying. Index assignment below is explicit and name-independent.


def classify(text: str) -> int:
    inputs = tokenizer(text, return_tensors="np", padding="max_length",
                       max_length=512, truncation=True)

    # Pre-inference assertion: attention_mask must not be all zeros
    # An all-zeros mask means either the input is empty or the name mapping failed
    assert inputs['attention_mask'].sum() > 0, (
        f"attention_mask is all zeros — check tokenizer output for text: {text[:50]!r}"
    )

    # FIXED: explicit index-based mapping — robust to ONNX node renaming after optimization
    # compiled_model.inputs[0] = input_ids (positional, not name-dependent)
    # compiled_model.inputs[1] = attention_mask (even if renamed to 'onnx::Unsqueeze_1')
    # compiled_model.inputs[2] = token_type_ids
    explicit_inputs = {
        compiled_model.inputs[0]: inputs['input_ids'],
        compiled_model.inputs[1]: inputs['attention_mask'],
        compiled_model.inputs[2]: inputs['token_type_ids'],
    }

    results = compiled_model(explicit_inputs)
    logits = results[compiled_model.output(0)]
    return int(np.argmax(logits, axis=-1)[0])


# After fix:
# Variable-length inputs (20–200 tokens, padded to 512):
#   attention_mask correctly passed to input[1] regardless of ONNX node name
#   BERT attends only to real tokens; [PAD] tokens excluded
#   Classification accuracy: 61% → 89%
#
# Fixed-length inputs (512 tokens, no padding):
#   Accuracy unchanged: 89% (was already correct — zero mask = full mask for unpadded input)

Validation

# Validation log — November 2026
# openvino 2024.0, BERT-base-uncased, Intel Core i7-12700
# ONNX opset 13 export via Optimum, FP32 inference

# ---- Before fix (dict input — name mismatch — zeros for attention_mask) ----
# Fixed-length inputs (512 tokens, 0 padding tokens):
#   attention_mask: [1,1,1,...,1] (512 ones)
#   all-zeros mask = full attention = same result as correct mask (no padding to mask)
#   Classification accuracy: 89%
#
# Variable-length inputs (50 tokens, 462 padding tokens):
#   Correct attention_mask: [1,1,...,1, 0,0,...,0] (50 ones, 462 zeros)
#   Actual attention_mask passed: [0,0,...,0] (512 zeros — name mismatch)
#   BERT: [CLS] token attends equally to 50 real tokens + 462 [PAD] tokens
#   [CLS] representation = (50/512) real content + (462/512) padding noise
#   Classification accuracy: 61%

# ---- After fix (explicit index mapping — correct attention_mask) ----
# Fixed-length inputs (512 tokens, 0 padding tokens):
#   attention_mask: [1,1,...,1] (512 ones) — correctly passed via index
#   Classification accuracy: 89% ✓ (unchanged)
#
# Variable-length inputs (50 tokens, 462 padding tokens):
#   Correct attention_mask: [1,1,...,1, 0,0,...,0] — correctly passed via index
#   BERT: [CLS] attends only to 50 real tokens; [PAD] tokens excluded
#   Classification accuracy: 89% ✓

# Test set: 200 variable-length texts (20–200 tokens), 10 classes
# Variable-length accuracy: 89% (was 61%)                                   ✓
# Fixed-length accuracy: 89% (unchanged)                                    ✓
# attention_mask all-zeros assertion: 0 failures in 200 test inputs          ✓
# Inference latency (Intel Core i7-12700): 8.4ms per request (unchanged)    ✓

# ---- Hour breakdown ----
# Triage:     0.5h — accuracy 89% fixed-length vs 61% variable-length;
#             print compiled_model.inputs names; discover 'onnx::Unsqueeze_1';
#             confirm tokenizer 'attention_mask' key has no matching input name;
#             add zero-mask check; confirm zeros in production inference
# Root cause: 0.25h — OpenVINO dict matching by name; optimization renames node;
#             silent zero assignment for unmatched input confirmed
# Fix:        0.5h — explicit index mapping; zero-mask pre-inference assertion;
#             document INPUT_NAMES at load time for traceability
# Validation: 0.25h — 200 variable-length inputs; accuracy 61%→89%;
#             fixed-length accuracy unchanged; zero-mask assertion 0 failures
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $143–233 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →