Starcast
Join the waitlist
Machine Learning Street Talk (MLST)
How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu
Coming soon to iPhone. Already have it? “Listen in Starcast” takes you straight there.
On this episode
About this episode
The car making a left turn at the start of this episode was never filmed. Cosmos 3 generated it. Ming-Yu Liu, who leads the Cosmos research at NVIDIA, explains how one model can describe a video, generate one, and produce robot actions. He walks Tim through the architecture. A vision language model reasons one token at a time; its weights then initialise a bidirectional diffusion generator for video, audio and action, and a shared temporal position scheme lines up signals that run at different rates. Ming-Yu treats "world model" as a set of tools, not one definition: forward dynamics, inverse dynamics and policy, trained together under a capacity limit so that each helps the others. He also…