ViViT

Embedding Video Clips

Transformer model for Video

Model 1: Spatio-temporal attention

This model simply forwards all spatio-temporal tokens extracted from the video, $z_0$, through the transformer encoder

Model 2: Factorised encoder

Model 3: Factorized self-attention

Factorised self-attention (Model 3). Within each transformer block, the multi-headed self-attention operation is factorised into two operations (indicated by striped boxes) that first only compute self-attention spatially, and then temporally\

Model 4: Factorised dot-product attention

Spatial Attention: Across H and W dimension
Temporal Attention: Across T dimension

Ablation

Model Varients

The unfactorised model (Model 1) performs the best on Kinetics 400. However, it can also overfit on smaller datasets such as Epic Kitchens, where we find our “Factorised Encoder” (Model 2) to perform the best