vLLM 0.28.0 — CPU backend

Model facebook/opt-125m, official ARM64 CPU image. The CLI parsed the real /metrics endpoint: 51 KB payload, 122 metric families, 0 unparseable lines, 13 of 13 required metrics found. Three verbatim captures are committed to the repository as regression fixtures.

vLLM reports KV utilisation as a 0–1 fraction — its own HELP text says "1 means 100 percent usage" — and the adapter converts it. That was checked against the live payload, because a fraction read as a percentage means the KV rules can never fire, silently, forever.

vLLM 0.28.0 — NVIDIA L4 (JarvisLabs)

The first L4 session: one capture, with the server driven into KV-cache exhaustion. This is the failure vLLM is best known for, and the one the KV rules exist to catch.

The highest KV occupancy reachable on CPU was 2.86%. The KV rules fire around 90%, so until this capture they could only be evaluated against synthetic data. The 16 waiting requests are all labelled capacity, and IFA-KV-001 was observed firing during the live run.

The preemption counter is active, and that is all this capture claims. It is a single snapshot, and a rate needs two time-separated scrapes, so no preemption rate was measured in this session. The E2E Networks run below supplies that measurement.

NVIDIA L4 — E2E Networks validation

A second, independent L4 on a different provider, with a different driver. Timestamped verbatim captures were taken at idle, under sustained load, once settled, and after the load stopped.

The preemption rates were calculated independently from consecutive raw captures taken 14 seconds apart, not read from IFA's own output: 285.9, 297.2, 294.3 and 278.5 preemptions per second across four intervals. The KV cache was deliberately constrained to make exhaustion reachable on a 24 GB card, and the model is a small one, so these figures describe this configuration, not typical production behaviour.

These captures validate the vLLM and DCGM adapters against live GPU metrics, outside Kubernetes. The Kubernetes-hosted IFA API path against a GPU workload was validated in a later run, described below.

Official NVIDIA DCGM exporter

NVIDIA's official exporter image, nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04, was run successfully on the same L4 alongside the vLLM load.

All six fields the IFA DCGM adapter reads were present and correctly formatted. GPU utilization reached 100% within the first 14-second sampling window, and no 18–20 second refresh delay was observed in the E2E sampling windows. That interval is coarse, so this does not show the exporter has no refresh latency, only that the earlier delay did not appear. The earlier delay was seen with the JarvisLabs bundled exporter and should not be generalized to NVIDIA's official one.

Triton 25.12 — CPU backend

Image nvcr.io/nvidia/tritonserver:25.12-pyt-python-py3, Python backend with a minimal echo model, KIND_CPU instance group. Three verbatim captures are committed to the repository: idle, loaded, and one with summary latencies enabled.

Triton exposes a failure counter, nv_inference_request_failure, with a reason label — vLLM does not, which is why ifa deliberately reports no error rate for the vLLM adapter. That asymmetry is a real design decision driven by what each runtime actually publishes, not a template applied to both.

A CPU-only Triton server publishes no nv_gpu_* metrics at all, which makes these captures a real test of the site's central claim: an absent signal is reported as unmeasured, not as zero. The vLLM captures could not test that — vLLM on CPU still reports a KV cache.

GPU-backed Triton — NVIDIA L4

The same adapter, run against a Triton server that is actually using a GPU. The model is pinned to the GPU with no CPU fallback, and the Triton startup log confirms it was loaded on GPU device 0.

Server

Triton 25.12, official NVIDIA container

GPU

NVIDIA L4, driver 570.133.20, CUDA 12.8

Model

TorchScript GPU matrix multiplication

Backend

LibTorch (PyTorch)

Load

40 concurrent workers, about 90 seconds

Completed requests

44,432

Request errors

0

ifa check

37 families, all 10 required found, 0 unparseable lines

Triton reports nv_gpu_utilization as a 0–1 fraction. It reported 0.02, IFA converts that to 2%, and DCGM independently reported 2% at the same moment. This corroborates the conversion; it is not a high-utilization result. The model is lightweight and the L4 was nowhere near saturated, so these captures say nothing about Triton behaviour at 90% GPU load. The server-side counter ended at 44,434 against the load generator's 44,432.

Not validated here: TensorRT or ONNX Runtime backends, dynamic batching behaviour, multiple GPUs, multiple models on one GPU server, and IFA's diagnostic rules against a Triton workload.

{
  "code": "IFA-LAT-002",
  "severity": "warning",
  "window_seconds": 10,
  "evidence": [
    { "metric": "ttft_p95_ms", "source": "vllm:time_to_first_token_seconds",
      "observed": 31999.999999999993, "threshold": 1000, "comparison": ">", "unit": "ms" },
    { "metric": "queue_time_share_of_ttft",
      "observed": 44.531250000000014, "threshold": 50, "comparison": "<", "unit": "percent" }
  ]
}

Real GPU. Real Kubernetes. Real Diagnoses.

IFA was deployed into a GPU-enabled Kubernetes cluster running vLLM on an NVIDIA L4. The collector discovered the workload, read genuine inference and GPU telemetry, and generated actionable diagnostic findings through the REST API.

Single-node K3s with the NVIDIA device plugin, vLLM 0.28.0 serving facebook/opt-125m, NVIDIA's official DCGM exporter as a DaemonSet, and IFA installed with its Helm chart. The IFA image was built from main, not from the published release.

IFA-KV-001 is deliberately not marked validated in this run. With a 125M-parameter model, KV-cache usage peaked near 0.23% of the L4's cache, so preemption never occurred. That is a separate result from the earlier E2E Networks session above, where preemption counters were measured climbing at roughly 280–300 per second on an L4 with a deliberately constrained cache. Preemption has been measured on real hardware; it has not yet been reproduced through IFA inside Kubernetes.

# invalid: label values cannot contain '/'
metadata:
  labels:
    inference.io/model: meta-llama/Llama-3.1-8B-Instruct

metadata:
  annotations:
    inference.io/model: meta-llama/Llama-3.1-8B-Instruct
  labels:
    inference.io/runtime: vllm

Not written as a humble-brag, and not as "we take quality seriously" — this is a short account of what broke, why the existing tests allowed it, and what changed. A project that publishes its own defects is more trustworthy than one that claims to have none.