Background information

There are a lot of questions about video files on the OpenCV mailing list. Most of them were asked because of the still incomplete documentation of OpenCV, but also because many users don't know some background details of video coding. So this is a short primer to get you started.

Video codecs and file containers

It is a common misunderstanding to mix up codecs and container file formats. Typically video files consist of a file container, distinguishable by file extension, that contains video, audio and other 'tracks'. These tracks are formatted and compressed according to some codec. If you want to express some video format, you should always give information on both, for example 'divx video in AVI file container'. In rare cases, the compressed data stream in its raw form is written into files without a surrounding container structure, for example .mpg or .mpeg files that might contain a raw MPEG2 data stream. You should avoid creating such files.

To be able to read a certain video snippet therefore requires that your library is able to understand the file format and has the given codec installed.

Color depth, color spaces, chroma subsampling

In addition to the compression algorithm, what you can get out of your material depends on the color space and / or color depth that your data is coded in. Usually, computer screens and imaging devices like cameras express colors as values of red, green and blue parts of the light spectrum. These three color channels are digitized with a certain precision, the bit depth. In OpenCV, it is common to use BGR channel order and a depth of 8 bits per channel.

For many algorithms however, a different color model makes more sense. YCbCr or YUV (see Charles Pointon's Color FAQ) represent color information by splitting into a 'grayscale' or luminance part (Y) and two 'color kind' channels CbCr or UV. Because the human eye is more sensitive to luminance (brightness) than to chrominance (kind of color), it is a common idea to save bandwith / file size by reducing the spatial resolution of the UV or CbCr channels. In combination with compression, oftentimes the compression parameters for chroma components are different to the luminance component as well.

Uncompressed video

HighGui is meant as a simple tool for experimentation. If you try to tune or evaluate algorithms, it is often useful to do that with uncompressed video files, because you gain independence of errors resulting from lossy compression / decompression. If your original video files or camera input has RGB coded material, use uncompressed RGB. Oftentimes, 'original' material might already be coded with a lossy video codec, most often some kind of MPEG, DIVX, WMV derivate that internally works with YUV color space and chroma subsampling. In such a case, you should convert to uncompressed YUV once and then use the uncompressed material for further experiments. A similar idea holds true for recordings from FireWire cameras that typically provide YUV data.

Tools for conversion are for example VirtualDub on Windows, ffmpeg (Linux, Mac OS X) or QuickTime Pro (commercial, Windows, Mac OS X).

Compatibility list

To be able to read your video files on all platforms, you should choose a file container and a codec (that is a color depth and channel ordering for uncompressed video) that is supported by all back ends. In detail this is Video for Windows, ffmpeg and QuickTime.

Currently these are the recommended codecs:

Container

FourCC

Name

Description

AVI

'DIB '

RGB(A)

Uncompressed RGB, 24 or 32 bit

AVI

'I420'

RAW I420

Uncompressed YUV, 4:2:0 chroma subsampled

AVI

'IYUV'

RAW I420

identical to I420

Because of chroma subsampling, uncompressed YUV files have 50% size of uncompressed RGB or 37.5% compared to RGBA.

If size is a problem, there are several lossless video compressors available. No single one however is supported on all platforms at the same time. Look for HuffYUV, CorePNG, Motion PNG or Motion JPEG2000.

To get information on what codec type / fourcc was used to generate a given video, use a tool like GSpot (Windows).

Conversion to OpenCV supported video format howtos

Using mencoder

The mencoder video encoder is an open source video encoder which is available on Mac OS X, Windows, and Linux. To convert existing videos to the I420 format supported by OpenCV on all platforms, install the mencoder binary and use:

Using VirtualDub

Change color encoding

File>Open>SomeRandomVideo.avi

Video>Filters>Click "Add">Select "Convert format">Select desired format

(e.g. 4:2:0 Planar YCbCr (YV12), or 32-Bit RGB)

To alter output video compression/codec

Video>Compression>Select appropriate codec

(Note that not all codecs listed may be operable. The "Video codec information" box will tell you FOURCC code, library file name, format restrictions, etc.)

To view original file type

File>File Information

Tools

Codecs that will help you store large videos losslessly

Tools for editing video snippets, changing color, resampling, etc.

Tools to get more information out of video container files

OpenCVWiki: VideoCodecs (last edited 2009-04-14 21:27:34 by JamesCarroll)