AI-generated video: developments in digital human technology
Breakthroughs in image generation, including Stable Diffusion, DALL·E and Midjourney, have accelerated progress in video and avatar generation. Technologies that turn a still image plus speech or motion input into video are opening up applications for virtual avatars, digital presenters and virtual performers. Interest spans media, marketing, streaming, games and education.
This article reviews developments in virtual and digital human generation, including open-source implementations and research projects. We focus on two projects and examine the related technologies around them:
• MoDA（Mapping-Once Audio-driven Portrait Animation）
• LivePortrait（Efficient Portrait Animation with Stitching / Retargeting）
Technical categories and architecture
Digital human generation can be considered through the following technical dimensions.
| Dimension | Examples | Challenges and development directions |
|---|---|---|
| Input modalities | Speech to facial, lip and expression motion; still image plus reference video to animation; text to video | Combining speech, expressions and viewpoint changes |
| Motion-driving design | Lip synchronization, including phoneme-to-mouth-shape mapping; facial motion; head rotation; blinking; gaze | Coordinating components, maintaining smooth motion, and controlling overfitting and distortion |
| Resolution and image quality | Higher resolution, super-resolution correction and preservation of facial detail | Noise control and consistent upsampling quality |
| Execution efficiency | Inference speed and real-time performance | Sampling steps, lightweight model design, quantization and distillation |
| Controllability | Motion editing and transfer, pose control and looping | User interaction and fine-grained adjustment |
| Safety and ethics | Synthetic-media detection, copyright management and deepfake prevention | Watermarking, traceability and bias mitigation |
MoDA: audio-driven portrait animation
Overview
MoDA, short for Mapping-Once Audio-driven Portrait Animation with Dual Attentions, generates portrait video from audio. It aims to produce natural animation by coordinating lip movements, head motion and eye movements.
Its central idea is to use dual attention mechanisms to handle lip synchronization and other facial movements separately while integrating the results. The approach has three stages:
1. Mapping-Once network
Maps audio to an intermediate speaker or motion representation in a single pass
2. Facial Composer network
Generates detailed facial landmarks
3. Temporal-Guided Renderer
Renders the final video while maintaining temporal consistency
This separation makes it easier to optimize lip movement and other actions, such as head movement and blinking, while improving coordination between them. Reported experiments show advantages in naturalness and expressiveness over earlier methods.
Strengths, limitations and applications
Strengths
•Designed specifically for audio-to-face animation, making it directly relevant to speech-driven avatars
•A relatively simple architecture that can be adapted for additional controls and extensions
•Potential for supporting multiple speakers and a wider range of facial shapes
Limitations and challenges
•Does not generate backgrounds, full-body movement, walking or other non-facial actions
•May have limits in resolution and the representation of fine facial detail
•Requires tuning for long-sequence stability and suppression of motion jitter
Potential applications
•Virtual presenters that speak in time with narration, such as AI newsreaders or service characters
•Online education and voice-interactive support avatars
•Support for live streams and synchronized voice-and-face character performances
LivePortrait: efficient portrait animation and retargeting
Overview
LivePortrait, published by KwaiVGI on GitHub, is an open-source implementation that animates a still portrait by transferring motion from a reference video.
Stitching and retargeting control are central concepts. The architecture combines and interpolates motion sources, allowing movements from a reference video to be applied to a different face while keeping transitions smooth.
A PyTorch implementation is available in the GitHub repository, providing a basis for further improvements and extensions.
Strengths, limitations and applications
Strengths
•Supports the common workflow of applying motion to a still image
•An architecture that offers flexibility in combining motions and controlling interpolation
•Open-source implementation allows customization and integration with other modules
Limitations and challenges
•Audio-driven lip synchronization requires integration with an additional method
•Existing limits in resolution and preservation of facial detail
•Temporal consistency and jitter control may require further improvement
Potential applications
•Transferring motion from a reference video to another portrait
•Character animation and supporting tools for visual effects
•Content tools that animate a user's own photographs with facial expressions
Other related technologies to watch
Understanding the following research and technology trends alongside MoDA and LivePortrait can help teams plan for future extensions and production use.
Veo（Google / DeepMind）
In May 2024, Google announced Veo, a text-to-video model associated primarily with DeepMind.
•Veo generates video from natural-language prompts, with reports describing ambitions for longer video at 1080p resolution.
•Reports on later versions, Veo 2 and Veo 3, also describe a direction towards audio synchronized with the generated video, including music and sound effects.
•Veo is a general-purpose video model, however. At the time of writing, fine-grained controls specifically for facial avatars and their motion remain limited.
Combining general-purpose video generation with the facial motion controls of MoDA or LivePortrait is a promising architectural direction.
Sora（OpenAI）
OpenAI announced its text-to-video model Sora in February 2024.
•Sora targets videos of up to around one minute, with detailed depth of field, scene development and object motion.
•Sora alone does not fully address requirements for precise avatar control, lip synchronization or user-specified motion.
Hybrid implementations that combine general-purpose generation with specialist facial modules such as MoDA and LivePortrait are another area for research and development.
A conceptual technology map
The diagram below groups the technologies discussed here into general video generation, facial avatar modules and acceleration techniques.
Planning around these modules, then improving and optimizing each one, provides a practical starting point for a digital human generation platform.
Production applications and practical challenges
Example scenarios
•Marketing videos and public relations content featuring virtual presenters
•Virtual newsreaders or narrators providing spoken information
•VTubers, virtual hosts and support for live streaming
•Interactive avatar-based materials for education and e-learning
•Customer support avatars combining voice interaction and facial expression
Challenges and risks
1. Synthetic media and deepfakes
Generated video may be mistaken for a real person. Watermarking, detection and traceability need to be considered during implementation.
2. Copyright and image rights
Face, voice and motion data can involve complex rights. License management and clear data acquisition policies are important.
3. Unnatural motion or facial expressions
Longer sequences can suffer from jitter, abrupt transitions, facial distortion or inconsistent lighting. Maintaining quality requires careful stabilization.
4. Compute costs and real-time performance
Model size, sampling steps, inference speed and GPU costs all impose practical constraints.
5. Diversity and fairness
Design needs to account for speaker characteristics, skin tone, age and varied expressions, including possible bias in training data.
6. User experience and controls
Content creators need practical controls for motion adjustment, expression editing and synchronization.
A suggested implementation process
1. Build a prototype
Use MoDA or LivePortrait to build a basic pipeline from audio and a reference face to generated video.
2. Evaluate quality and tune the system
Assess frame consistency, jitter and the naturalness of expressions using both quantitative and qualitative measures.
3. Improve speed and efficiency
Consider distillation, quantization and hybrid integration with one-step generation models.
4. Extend the controls
Add motion editing, motion transfer, looping and synchronization with interactive applications.
5. Put safety measures in place
Plan synthetic-media detection, watermarking, usage policies and appropriate restrictions.
Closing observations
MoDA and LivePortrait are useful specialist building blocks for facial motion generation, particularly where controllability and editing matter. Meeting production requirements for image quality, real-time performance, safety and interaction will depend on how these modules are combined with general-purpose generation and acceleration techniques.