跳到主要内容Skip to main content

音视频联合生成Joint Audio-Video Generation

Joint Audio-Video Generation

  • 图像语音视频Image, speech & video
  • 第四段 · 看懂Stage 4 · Under the Hood
  • 引用核验 2026-11-19refs checked 2026-11-19
解决什么问题What problem it solves

让画面和对白、音效在同一次生成里对齐,而不是先出视频再配音。Align picture with dialogue and sound effects in one generation pass instead of generating video and dubbing it afterwards.

这是什么What it is

模型在同一个生成过程里同时建模视觉帧与音频 token,口型、动作声、环境声按时间轴对齐。

与「视频生成 + TTS 后配」的区别:联合生成能让人物说话口型与环境声同步,后配通常对不上。

代价是训练数据与算力更贵,评测也要同时看画质、音质与同步,不能只挑一帧好看。

它不等于「自动配乐」:音乐版权、音色授权与内容标识仍是单独的问题。

The model jointly models visual frames and audio tokens in one generation process, aligning lip movement, action sounds and ambience on a timeline.

Unlike video generation plus TTS, joint generation can synchronise a speaker’s lips with ambience; post-dubbing usually does not match.

The price is more expensive training data and compute, and evaluation must check picture, audio and sync together rather than one pretty frame.

It is not automatic scoring: music copyright, voice licensing and content labelling remain separate issues.

为什么需要知道Why it matters

2026 年的视频模型已经把声音当成一等公民;只看画面不看音画同步,会高估模型。Video models in 2026 treat sound as a first-class citizen; judging only the picture overestimates the model.

出处Sources
  • 字节 Seed · Seedance 2.5ByteDance Seed · Seedance 2.5 ↗
  • Google DeepMind · VeoGoogle DeepMind · Veo ↗
  • 学 AI · 视频生成是怎么做的Learn AI · How video generation works ↗