Video summarization using event‐related potential responses to shot boundaries in real‐time video watching

Our aim was to develop an event‐related potential (ERP)‐based method to construct a video skim consisting of key shots to bridge the semantic gap between the topic inferred from a whole video and that from its summary. Mayer's cognitive model was examined, wherein the topic integration process of a user evoked by a visual stimulus can be associated with long‐latency ERP components. We determined that long‐latency ERP components are suitable for measuring a user's neuronal response through a literature review. We hypothesized that N300 is specific to the categorization of all shots regardless of topic relevance, N400 is specific for the semantic mismatching process for topic‐irrelevant shots, and P600 is specific for the context updating process for topic‐relevant shots. In our experiment, the N400 component led to more negative ERP signals in response to topic‐irrelevant shots than to topic‐relevant shots and showed a fronto‐central scalp pattern. P600 elicited more positive ERP signals for topic‐relevant shots than for topic‐irrelevant shots and showed a fronto‐central scalp pattern. We used discriminant and artificial neural network (ANN) analyses to decode video shot relevance and observed that the ANN produced particularly high success rates: 91.3% from the training set and 100% from the test set.