Alibaba Tongyi Lab Open-Sources Fun-CineForge: A Breakthrough in Film-Level AI Dubbing

On March 16, 2026, Alibaba Tongyi Lab announced the release and open-sourcing of Fun-CineForge, the first multi-modal large model designed to support film-level dubbing across multiple scenes. Alongside the model, the lab released a high-quality dataset construction methodology, aiming to close the loop between data and modeling for professional AI dubbing.

Addressing the Challenges of Cinematic Dubbing

Creating high-quality dubbing for film requires navigating four strict requirements: precise lip synchronization, emotional expressiveness aligned with character attributes, consistent timbre across multiple characters, and accurate time alignment—even when speakers are obstructed or off-screen.

Existing AI dubbing solutions have historically struggled with these demands due to the scarcity of high-quality multi-modal datasets and the limitations of traditional models, which rely heavily on clearly visible lip movements. This dependency fails in complex scenarios involving rapid shot changes, facial occlusions, or multiple speakers.

Technical Innovation: The 'Time Modality'

Fun-CineForge seeks to overcome these bottlenecks through a unified design of data and model. Built upon the CosyVoice3 speech synthesis architecture, the model inputs silent video clips, dubbing text, character attributes, emotional cues, and time information to generate synchronized speech.

A standout innovation is the introduction of the "Time Modality." While traditional Text-to-Speech (TTS) models focus on text, audio, and visuals, Fun-CineForge utilizes time information as a distinct modality. This allows the model to determine when speech starts and ends and who is speaking during specific time intervals. Crucially, this serves as a strong supervisory signal when the visual modality is missing (e.g., the speaker's face is hidden), ensuring voices appear in the correct time window.

The CineDub Dataset

To fuel this model, the team developed an automated production pipeline called CineDub, capable of converting raw film footage into structured multi-modal data. This process includes vocal separation, text transcription, and joint audio-video speaker separation.

Utilizing a general large model chain-of-thought for bidirectional correction, the pipeline significantly reduced error rates:

• Chinese Character Error Rate (CER) reduced from 4.53% to 0.94%.
• English Word Error Rate (WER) reduced from 9.35% to 2.12%.
• Speaker Separation Error Rate reduced from 8.38% to 1.20%.

The dataset covers diverse scenarios including monologues, narration, dialogues, and multi-speaker scenes, derived from over 350 Chinese and English films and TV series.

Performance and Availability

Experiments demonstrate that Fun-CineForge outperforms existing open-source models (such as DeepDubber-V1 and InstructDubber) in naturalness, emotional expression, timbre similarity, and lip-sync accuracy. It is the first model to effectively support dual and multi-person dialogue scenes with accurate time alignment.

However, the lab noted that performance may fluctuate with longer videos, currently supporting inference for video clips under 30 seconds.

Resources:

• Project Page: https://funcineforge.github.io/
• GitHub: https://github.com/FunAudioLLM/FunCineForge
• HuggingFace: https://huggingface.co/FunAudioLLM/Fun-CineForge
• ModelScope: https://www.modelscope.cn/models/FunAudioLLM/Fun-CineForge/