Large numeric datasets, such as long sensor recordings or image stacks, are stored as chunked arrays: the array is split into fixed-size blocks that can be read one at a time. HDF5 and Zarr are the two most common formats. Basic file reading is in File Input and Output in Python.
Why Chunk
- a recording can be far larger than RAM, so it must be read in pieces
- a training loop needs random access to small windows, not the whole file
- each chunk can be compressed on its own
HDF5
- one file holding a tree of groups and datasets, like folders and arrays
- chunking and compression per dataset
- mature, used widely in science and in robotics demonstration data
- weaknesses: one big file, so parallel writing is hard, and reading from many processes needs care (open the file separately in each process, and the library has a global lock)
Zarr
- stores each chunk as its own file (or object in cloud storage), plus small metadata files
- many processes can read different chunks in parallel with no shared handle, which suits PyTorch DataLoader workers
- easy to append to and to place on network or cloud storage
- weakness: many small files can be slow on some file systems if chunks are too small
Choosing A Chunk Shape
- match the chunk to how the data is read, for example one training sample per chunk
- too small: many files and high overhead
- too large: each read pulls in data that is not needed
Storage Tips
- float16 halves size and read time when the values do not need float32 precision (for example samples from a 12-bit ADC)
- store labels and metadata next to the data, plus the code version that built it, so every dataset can be rebuilt and traced
- build large datasets with parallel reader processes and a background writer thread, so the slow disk does not stall the processing