ONNX is an open file format for trained neural networks, and OpenVINO is Intel's toolkit for running networks fast on CPUs and Intel hardware. Together they move a model from a training framework such as PyTorch to an edge computer that has no GPU and no training code.
ONNX
- stores a model as a graph of standard operators plus the weights
- exported from PyTorch with
torch.onnx.export, which traces the model with an example input - runs in many runtimes (ONNX Runtime, TensorRT, OpenVINO), so one export serves many devices
- dynamic axes can be declared, for example a variable batch size
OpenVINO
- converts a model to its own IR format: an
.xmlfile for the graph and a.binfile for the weights compile_modeloptimises the graph for a target device, such as CPU, integrated GPU or NPU, by fusing layers and choosing fast kernels- performance hints:
LATENCYmakes one request as fast as possible,THROUGHPUTruns many requests in parallel for the most total work - can quantise to INT8 with a small calibration set, which is often 2 to 4 times faster on CPU with little accuracy loss
Minimal Use
import openvino as ov
core = ov.Core()
compiled = core.compile_model(core.read_model("model.xml"), "CPU")
output = compiled(input_array)[compiled.output(0)]Deployment Advice
- the preprocessing at inference must match training exactly, so copy the same constants and functions, and test both on the same input
- export the exact model you will run, for example the single-input backbone rather than a training wrapper
- measure latency on the real target CPU, not on the training machine