KAIST uses a diffusion language model to cut silent lip-reading word error to 19.4%
Prof. Ro Yong-man's team decodes the clearest words first and fills ambiguous ones from context instead of reading left to right; on the 433-hour LRS3 dataset it scored 19.4% WER, which it says is the lowest among methods trained only on LRS3 labels.
Key facts
- Team: Prof. Ro Yong-man, KAIST School of Electrical Engineering
- Method: diffusion large language model for visual speech recognition
- Benchmark: LRS3, 433 hours of video
- Result: 19.4% word error rate
- Next goal: smaller on-device models for wearables and robots