ONNX is an open file format for trained neural networks, and OpenVINO is Intel's toolkit for running networks fast on CPUs and Intel hardware. Together they move a model from a training framework such as PyTorch to an edge computer that has no GPU and no training code.

ONNX

  • stores a model as a graph of standard operators plus the weights
  • exported from PyTorch with torch.onnx.export, which traces the model with an example input
  • runs in many runtimes (ONNX Runtime, TensorRT, OpenVINO), so one export serves many devices
  • dynamic axes can be declared, for example a variable batch size

OpenVINO

  • converts a model to its own IR format: an .xml file for the graph and a .bin file for the weights
  • compile_model optimises the graph for a target device, such as CPU, integrated GPU or NPU, by fusing layers and choosing fast kernels
  • performance hints: LATENCY makes one request as fast as possible, THROUGHPUT runs many requests in parallel for the most total work
  • can quantise to INT8 with a small calibration set, which is often 2 to 4 times faster on CPU with little accuracy loss

Minimal Use

import openvino as ov
core = ov.Core()
compiled = core.compile_model(core.read_model("model.xml"), "CPU")
output = compiled(input_array)[compiled.output(0)]

Deployment Advice

  • the preprocessing at inference must match training exactly, so copy the same constants and functions, and test both on the same input
  • export the exact model you will run, for example the single-input backbone rather than a training wrapper
  • measure latency on the real target CPU, not on the training machine