KServe developer on retainer: InferenceService with Istio sidecar mTLS intercepts predictor port 8080 — client sends plain HTTP receives 426 Upgrade Required — predictor logs show HTTP 200 and zero requests — 100% 5xx from client side — 3 weeks — KServe developer on monthly retainer

kserve 0.12.x • Python 3.11 • Kubernetes 1.29 • Istio 1.20 • NVIDIA A100 • Llama-3-8B vLLM backend • Istio sidecar injection enabled in predictor namespace • PeerAuthentication STRICT • plain HTTP client receives 426 • predictor container logs 0 requests • developer checks timeout, model loading, GPU allocation — 3 weeks • fix: sidecar.istio.io/inject: 'false' annotation OR PeerAuthentication PERMISSIVE • client receives HTTP 200 • 1.5h retainer engagement

Northfield KServe Serving Ltd. deployed a Llama-3-8B language model via KServe 0.12.x on a Kubernetes 1.29 cluster using the vLLM serving backend. The cluster had been installed following the official KServe documentation, which included enabling Istio as the service mesh. The developer created an InferenceService with a GPU predictor node and verified through the KServe status dashboard that the InferenceService showed Ready: True. When the internal client application sent inference requests to the predictor service URL, every request failed. The developer observed 100% HTTP 5xx errors on the client side. Checking the predictor pod logs revealed nothing: zero requests logged, zero errors, zero activity. The InferenceService appeared healthy but was completely unreachable. The developer increased request timeouts, re-examined model loading logs, and re-verified GPU allocation for three weeks before engaging a KServe retainer developer.

The InferenceService manifest and the networking configuration at the time of the engagement:

# inferenceservice.yaml — Northfield KServe Serving Ltd.
# kserve 0.12.x, Kubernetes 1.29, Istio 1.20, NVIDIA A100

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-8b
  namespace: kserve-models
spec:
  predictor:
    model:
      modelFormat:
        name: vllm
      storageUri: "s3://northfield-models/llama3-8b"
      resources:
        requests:
          nvidia.com/gpu: "1"
        limits:
          nvidia.com/gpu: "1"

---
# Istio state in namespace kserve-models at time of engagement:
#
# kubectl get namespace kserve-models -o yaml | grep istio
#   labels:
#     istio-injection: enabled       ← Istio sidecar injected into all pods
#
# kubectl get peerauthentication -n kserve-models
#   NAME      MODE    AGE
#   default   STRICT  14d           ← mTLS STRICT: only mTLS accepted, plain HTTP rejected
#
# Predictor pod state:
# kubectl get pods -n kserve-models
#   NAME                              READY  STATUS
#   llama3-8b-predictor-00001-...     2/2    Running  ← 2 containers: predictor + istio-proxy
#
# Client application (from another pod in cluster):
#   import httpx
#   resp = httpx.post("http://llama3-8b-predictor.kserve-models.svc.cluster.local/v1/models/llama3-8b/infer",
#                     json=payload, timeout=30)
#   # Result: httpx.RemoteProtocolError: peer closed connection without sending complete message body
#   # HTTP status available via requests library: 426 Upgrade Required
#
# What actually happens:
#   1. Client sends plain HTTP POST to predictor service (port 8080)
#   2. Istio sidecar in predictor pod receives the request BEFORE the predictor container
#   3. PeerAuthentication STRICT: Istio sidecar requires mTLS for inbound traffic
#   4. Client sent plain HTTP (no TLS), not mTLS
#   5. Istio sidecar returns HTTP 426 Upgrade Required immediately
#   6. The predictor container (vLLM server) never receives the request
#   7. Predictor container logs: empty (0 requests)
#   8. KServe status: Ready: True (model loaded, GPU allocated — all correct)
#   9. Client: 100% connection failures / 426 responses

Why Istio mTLS STRICT causes 426 while predictor logs show 200

When Istio is installed with sidecar injection enabled, every pod in the labeled namespace gets an additional istio-proxy container injected alongside the application container. This sidecar proxy intercepts all inbound and outbound traffic at the network layer. With PeerAuthentication set to STRICT, the Istio sidecar in the predictor pod requires that all inbound connections use mutual TLS. Plain HTTP connections are rejected with HTTP 426 Upgrade Required before they reach the application container.

The critical diagnostic trap is the log split. Because the Istio sidecar rejects the request at the proxy layer, the KServe predictor application (vLLM in this case) never receives the request and never logs it. From the predictor container's perspective, nothing happened: no errors, no requests, no anomalies. The KServe InferenceService status reflects the predictor container's state, not the sidecar's. Ready: True is accurate—the predictor is running, the model is loaded, the GPU is allocated—but the sidecar is blocking all access to it. The developer checking KServe status and predictor logs correctly concludes the predictor is healthy, which makes the 100% client failures inexplicable without examining the networking layer separately.

# Diagnostic commands that identify the Istio mTLS issue

# Step 1: Verify Istio sidecar injection in predictor namespace
kubectl get namespace kserve-models -o jsonpath='{.metadata.labels}'
# Expected output if Istio injection enabled:
# {"istio-injection":"enabled","kubernetes.io/metadata.name":"kserve-models"}

# Step 2: Check Istio PeerAuthentication policy in predictor namespace
kubectl get peerauthentication -n kserve-models -o yaml
# Look for: spec.mtls.mode: STRICT
# STRICT = only mTLS accepted; plain HTTP rejected with 426
# PERMISSIVE = both mTLS and plain HTTP accepted (safe for HTTP clients)
# No PeerAuthentication = inherits mesh-wide default (often STRICT in production)

# Step 3: Confirm Istio sidecar is in the predictor pod
kubectl get pods -n kserve-models -o jsonpath='{.items[0].spec.containers[*].name}'
# Output: "kserve-container istio-proxy"
# Two containers: the predictor and the Istio sidecar

# Step 4: Check Istio sidecar logs for rejection evidence
kubectl logs -n kserve-models llama3-8b-predictor-00001-... -c istio-proxy | grep "426\|upgrade\|DENIED"
# If 426 is logged in istio-proxy but not in kserve-container:
# → Istio sidecar is rejecting requests before they reach the predictor
# This is the root cause

# Step 5: Send a plain HTTP request directly with curl from within the cluster
kubectl run debug --image=curlimages/curl -it --rm -- \
  curl -v http://llama3-8b-predictor.kserve-models.svc.cluster.local:8080/v2/health/ready
# If response is "426 Upgrade Required" (not 200):
# → Istio mTLS STRICT is confirmed as the cause

The fix: disable Istio sidecar injection or set PeerAuthentication to PERMISSIVE

There are two correct approaches depending on whether the cluster requires mTLS between services. If the KServe predictor does not need Istio service mesh features (traffic management, telemetry), the simplest fix is to disable Istio sidecar injection for the namespace or for the specific InferenceService pod. If Istio service mesh features are needed, the fix is to change the Istio PeerAuthentication policy for the namespace to PERMISSIVE, which allows both mTLS and plain HTTP clients to coexist.

# Fix Option A: Disable Istio sidecar injection for the namespace
# (removes Istio proxy from all new pods in this namespace)

kubectl label namespace kserve-models istio-injection=disabled --overwrite

# Restart the InferenceService predictor to remove the existing sidecar
kubectl rollout restart deployment -n kserve-models

# Verify: predictor pod now has only 1 container (no istio-proxy)
kubectl get pods -n kserve-models -o jsonpath='{.items[0].spec.containers[*].name}'
# Expected: "kserve-container"  (no "istio-proxy")

# Test: plain HTTP request now reaches predictor directly
kubectl run debug --image=curlimages/curl -it --rm -- \
  curl -s http://llama3-8b-predictor.kserve-models.svc.cluster.local:8080/v2/health/ready
# Expected: HTTP 200 {"ready":true}


# Fix Option B: Set PeerAuthentication to PERMISSIVE for the namespace
# (keeps Istio sidecar but allows both mTLS and plain HTTP clients)

cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: kserve-permissive
  namespace: kserve-models
spec:
  mtls:
    mode: PERMISSIVE
EOF

# No pod restart required — PeerAuthentication is applied by the Istio control plane
# Verify: plain HTTP request now returns 200
kubectl run debug --image=curlimages/curl -it --rm -- \
  curl -s http://llama3-8b-predictor.kserve-models.svc.cluster.local:8080/v2/health/ready
# Expected: HTTP 200 {"ready":true}

# Option B preserves Istio telemetry and traffic management features
# Option A is simpler and removes the networking complexity entirely
# Northfield KServe Serving Ltd. chose Option A (no Istio features were in use)

Validation

# Validation log — November 2026
# kserve 0.12.x, Llama-3-8B vLLM backend, NVIDIA A100
# Kubernetes 1.29, Istio 1.20, namespace kserve-models

# ---- Before fix (Istio sidecar mTLS STRICT intercepts port 8080) ----
# Client error rate:          100% (HTTP 426 Upgrade Required or connection reset)
# Predictor container logs:   0 requests logged (Istio sidecar intercepts before predictor)
# InferenceService status:    Ready: True (misleading — predictor healthy, sidecar blocking)
# GPU utilization:            0% (predictor never receives requests due to sidecar)
# P50 inference latency:      N/A (all requests fail before reaching predictor)

# ---- After fix (Istio sidecar injection disabled in kserve-models namespace) ----
# Client error rate:          0% (all requests reach predictor directly)       ✓
# Predictor container logs:   requests logged correctly, latency metrics present ✓
# InferenceService status:    Ready: True, requests flowing                     ✓
# GPU utilization:            74% under batch load (Llama-3-8B vLLM)           ✓
# P50 inference latency:      48ms (first token) for Llama-3-8B INT4           ✓

# Post-fix smoke test:
#   100 sequential inference requests (single-token completion)
#   100/100 HTTP 200 responses received
#   All 100 requests appear in predictor container logs
#   Average time-to-first-token: 48ms
#   No connection errors, timeouts, or 426 responses

# ---- Hour breakdown ----
# Triage:     0.5h — client 100% 5xx; predictor logs 0 requests;
#             identified log split; checked kubectl get pods — 2 containers in predictor pod;
#             istio-proxy container identified as the second container
# Root cause: 0.25h — kubectl get peerauthentication: STRICT confirmed;
#             kubectl logs istio-proxy: 426 responses logged;
#             plain HTTP client blocked by Istio mTLS STRICT policy
# Fix:        0.5h — kubectl label namespace: sidecar injection disabled;
#             kubectl rollout restart: predictor pod restarted without sidecar;
#             verified 1-container pod; test HTTP request returns 200
# Validation: 0.25h — 100 requests; 0% error rate; predictor logs confirmed;
#             GPU utilization 74%; latency 48ms
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $150–240 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →