SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision

Xubo Liu1,2, Egor Lakomkin2, Konstantinos Vougioukas2, Pingchuan Ma2, Honglie Chen2, Ruiming Xie2, Morrie Doulaty2, Niko Moritz2, Jáchym Kolář2, Stavros Petridis2, Maja Pantic2, Christian Fuegen2,
1University of Surrey, 2Meta AI
Accepted at CVPR 2023

Recently reported state-of-the-art results in visual speech recognition (VSR) often rely on increasingly large amounts of video data, while the publicly available transcribed video datasets are limited in size. In this paper, for the first time, we study the potential of leveraging synthetic visual data for VSR. Our method, termed SynthVSR, substantially improves the performance of VSR systems with synthetic lip movements. The key idea behind SynthVSR is to leverage a speech-driven lip animation model that generates lip movements conditioned on the input speech. The speech-driven lip animation model is trained on an unlabeled audio-visual dataset and could be further optimized towards a pre-trained VSR model when labeled videos are available. As plenty of transcribed acoustic data and face images are available, we are able to generate large-scale synthetic data using the proposed lip animation model for semi-supervised VSR training. We evaluate the performance of our approach on the largest public VSR benchmark - Lip Reading Sentences 3 (LRS3). SynthVSR achieves a WER of 43.3% with only 30 hours of real labeled data, outperforming off-the-shelf approaches using thousands of hours of video. The WER is further reduced to 27.9% when using all 438 hours of labeled data from LRS3, which is on par with the state-of-the-art self-supervised AV-HuBERT [1] method. Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16.9% using publicly available data only, surpassing the recent state-of-the-art approaches [2, 3] trained with 29 times more non-public machine-transcribed video data (90,000 hours). Finally, we perform extensive ablation studies to understand the effect of each component in our proposed method.

Presentation at CVPR

8-min presentation at CVPR 2023.

Supplementary Video

Synthetic lip movement videos generated from Librispeech and CelebA.


[1] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. In ICLR 2022
[2] Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Transformer-based video front-ends for audio-visual speech recognition. In INTERSPEECH 2022
[3] Dmitriy Serdyuk, Olivier Siohan, and Otavio de Pinho ForinBraga. Audio-visual speech recognition is worth 32x32x8 voxels. In ASRU 2021