The pinhole camera model describes how a point in 3D space maps to a pixel. With a depth camera, the same model run backwards turns a pixel plus its depth into a 3D point, which is how robots locate objects they see. Real lenses add distortion, covered in Properties of the Lens.
Projection (3D To Pixel)
In the camera frame (OpenCV convention: right, down, forward):
- : focal lengths in pixels
- : the principal point, near the image centre
- these four numbers are the intrinsics, often written as the matrix
Back-Projection (Pixel Plus Depth To 3D)
- depth cameras such as the RealSense report -depth (distance along the optical axis), not distance along the ray
- colour and depth must be aligned to the same image, otherwise every point lands centimetres off
Robust Depth
- single depth pixels have holes and speckle, so take the median of a small patch (for example 5 by 5) and ignore zero or invalid values
Object Size From A Box
- by similar triangles, an object pixels wide at depth is metres wide
- the depth lands on the face turned to the camera, so to reach an object's centre, move the point further along its own ray by about half the object's size
Convention Trap
- OpenCV uses down and forward, while OpenGL (and many simulators) uses up and backward
- code that mixes them can pass a round-trip test (project then back-project) and still be wrong on a real camera, so test against a measured physical point too
Other Uses Of The Model
- field of view: for image height
- camera calibration (for example with a checkerboard) estimates and the lens distortion