Visual Analytics for Understanding Images and Video

Images and video recordings are commonly categorised as unstructured data, which means that they are not primarily suited for computer analysis. The contents of unstructured data cannot be adequately represented by numbers or symbols and require the power of human vision for extracting meaningful information. While images and video are well suited for human visual perception, coping with large amounts of unstructured data may be daunting for humans. Hence, humans need computer support but cannot be substituted by computers. Computer processing of unstructured data begins with deriving some kind of structured data. One possible kind of structured data is summary statistics characterising each image or video frame as a whole. Such data can be used for arranging the images or frames by similarity to provide an overview of an image collection or a video recording and enable search for data items similar to a given sample. Another possibility is to use the existing image processing techniques for detecting particular objects represented in the visual contents and computing their characteristics, such as sizes, shapes, and positions. These characteristics can then be analysed in various ways suitable for structured data, but these analyses must be complemented with human perception and understanding of the original visual contents. This chapter includes several examples of approaches in which computers support the unique capabilities of humans. At the end, we summarise these approaches in a general scheme showing the possible operations and the types of data that are derived and analysed.