Self-supervised learning of video representations from a child's perspective

TL;DR

Train self-supervised video models on longitudinal, egocentric headcam recordings collected from a child over a two year period in their early development, suggesting that important temporal aspects of a child's internal model of the world may be learnable from their visual experience using highly generic learning algorithms and without strong inductive biases.

Abstract

Children learn powerful internal models of the world around them from a few years of egocentric visual experience. Can such internal models be learned from a child's visual experience with highly generic learning algorithms or do they require strong inductive biases? Recent advances in collecting large-scale, longitudinal, developmentally realistic video datasets and generic self-supervised learning (SSL) algorithms are allowing us to begin to tackle this nature vs. nurture question. However, existing work typically focuses on image-based SSL algorithms and visual capabilities that can be learned from static images (e.g. object recognition), thus ignoring temporal aspects of the world. To close this gap, here we train self-supervised video models on longitudinal, egocentric headcam recordings collected from a child over a two year period in their early development (6-31 months). The resulting models are highly effective at facilitating the learning of action concepts from a small number of labeled examples; they have favorable data size scaling properties; and they display emergent video interpolation capabilities. Video models also learn more accurate and more robust object representations than image-based models trained with the exact same data. These results suggest that important temporal aspects of a child's internal model of the world may be learnable from their visual experience using highly generic learning algorithms and without strong inductive biases.

Figures (6)

• Figure 1: Illustration of the spatiotemporal MAE objective. Top row shows the original sequence of frames (from child S in SAYCam). Middle row shows the masked sequence, where 90% of the spatiotemporal "patches" are randomly masked out. Bottom row shows the predictions from a model trained on child S. The model is trained to predict the masked patches from the visible patches at the pixel level.
• Figure 2: Top-5 validation accuracy in the SSV2 (a) and Kinetics-700 (b) action recognition tasks. Results are shown for both 10-shot (left) and 50-shot conditions (right). Dashed horizontal lines show the chance baseline. Orange represents the models pretrained with the child headcam data. Cyan represents the models pretrained with Kinetics-700 data. Magenta represents a purely image-based reference model. Green represents a reference model trained from scratch on the downstream task only (no pretraining).
• Figure 3: t-SNE embeddings of video clips from SSV2. Each point corresponds to a clip from the validation set. Clips belonging to 10 "developmentally realistic" action categories (shown in the legend) are highlighted with different colors. (Left) Results from a 0-shot model pretrained on child S, but not finetuned with any data from SSV2. (Middle) Results from a model pretrained on child S and finetuned on 10-shot SSV2. (Right) Results from a model pretrained on child S and finetuned on 50-shot SSV2. Numbers in parentheses represent the top-5 validation accuracy for the corresponding categories in the 50-shot condition.
• Figure 4: A data size scaling experiment for child S. Spatiotemporal MAEs are trained on all data from child S and on subsets of it over a four orders of magnitude range in data size. The performance of the resulting models are evaluated in the SSV2 (a) and Kinetics-700 (b) action recognition tasks (individual dots). A log-linear model is fit to the data to extrapolate performance beyond the 194 hours of data we currently have from S. Shaded areas represent 95% confidence intervals around the linear fits. A developmentally relevant time scale of 2.5 years is indicated by the vertical dashed line.
• Figure 5: Two examples illustrating the emergent video interpolation capabilities in a spatiotemporal MAE model trained on child S. In both cases, the top row shows the original sequence of frames; the middle row shows the masked sequence where the four central frames are masked out; and the bottom row shows the model completions of the masked frames. The original sequences are from the other two children in SAYCam (A and Y, respectively). Note that the model completions in both cases are not simple copies of the nearby visible frames; they are rather novel frames that indicate some understanding of the consequences of the camera movements on the image. Further examples can be found at https://github.com/eminorhan/video-models/tree/master/comps.