Open Source Project #196: LiveTalking — Real-Time Interactive Streaming Digital Human Engine, Wav2Lip/MuseTalk/ER-NeRF, 9k Stars

Introduction

"Real-time interactive streaming digital human engine — synchronized audio/video dialogue."

This is article #196 in the "One Open Source Project a Day" series. Today's project is LiveTalking — a real-time interactive streaming digital human engine with 9,140 Stars, created by lipku (Li Hengzhong), Apache-2.0, and already widely deployed in production.

LiveTalking solves a specific engineering problem: given an existing video clip of a person, how do you make them "say" arbitrary text or audio input in real time, output as a low-latency audio/video stream, while supporting multi-user concurrency, mid-speech interrupts, and streaming to live platforms — all at production grade.

What You'll Learn

• The four rendering models LiveTalking supports and their respective use cases
• The four-layer system architecture: API / Logic / Rendering / Streaming
• Three output modes: WebRTC (browser), RTMP (live streaming), virtual camera
• Plugin extension mechanism (TTS, Avatar, Output modules)
• Production features: multi-session concurrency, interrupt dialog, action orchestration, voice cloning

Prerequisites

• Basic Python and command-line experience
• General familiarity with deep learning inference (models, GPU acceleration)
• Some background in live streaming or WebRTC is helpful but not required

Project Background

Overview

LiveTalking is a production-grade digital human streaming framework, not an experimental demo. Its positioning is squarely at the engine layer: receive text or audio input, run TTS and lip-sync inference, output a synchronized audio/video stream, so that upstream applications (LLM dialogs, live streaming systems, customer service platforms) can plug in directly.

Author / Team

• Author: lipku (Li Hengzhong)
• Primary language: Python 3.10+
• License: Apache-2.0
• Website: livetalking.ai
• Documentation: doc.livetalking.ai

Project Stats

• ⭐ GitHub Stars: 9,140+
• 🍴 Forks: 1,449+
• 📄 License: Apache-2.0
• 📅 Created: 2023-12-19

Core Pipeline

User input (text / audio)
    ↓
LLM generates reply (optional — connect Qwen or any LLM)
    ↓
TTS synthesizes speech (EdgeTTS / GPT-SoVITS / CosyVoice / Tencent TTS, etc.)
    ↓
Acoustic feature extraction (Mel spectrogram, etc.)
    ↓
Lip-sync inference (Wav2Lip / MuseTalk / ER-NeRF)
    ↓
Post-processing (blend generated lip region back onto high-res source video)
    ↓
Audio/video streaming (WebRTC / RTMP / virtual camera)

Each connection is assigned a unique sessionid. API calls route to the correct session by ID, supporting multi-user concurrency with isolated compute per session.

Supported Rendering Models

LiveTalking supports four lip-sync rendering models with distinct trade-offs:

Real-time inference FPS benchmarks (≥25 FPS required for real-time):

Backend logs expose two metrics: inferfps (GPU inference frame rate) and finalfps (final streaming frame rate). Both must be ≥ 25 for a real-time output.

Quick Start

Installation

git clone https://github.com/lipku/LiveTalking.git
conda create -n livetalking python=3.12
conda activate livetalking
 
# Install PyTorch matching your CUDA version (CUDA 12.8 example)
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
  --index-url https://download.pytorch.org/whl/cu128
 
cd LiveTalking
pip install -r requirements.txt

Tested on Ubuntu 22.04 + Python 3.12 + PyTorch 2.9.1 + CUDA 12.8.

Download Models

Download pre-trained models from Quark Drive or Google Drive:

# Place wav2lip256.pth in models/ and rename to wav2lip.pth
# Extract wav2lip256_avatar1.tar.gz into data/avatars/

Start the Server

# Wav2Lip model with WebRTC transport
python app.py --transport webrtc --model wav2lip --avatar_id wav2lip256_avatar1
 
# MuseTalk model
python app.py --transport webrtc --model musetalk --avatar_id musetalk_avatar1
 
# RTMP output (push to live platforms)
python app.py --transport rtmp --model wav2lip --avatar_id wav2lip256_avatar1

Note: TCP:8010 and UDP:1-65536 must be open on the server (WebRTC uses a wide UDP port range).

Connect and Use

• Browser: Open http://serverip:8010/index.html, click "Start Connection," type text in the input box
• API: Call POST /human with text input, or POST /humanaudio with an audio file

System Architecture: Four Layers

API Layer

Each connection gets a unique sessionid; API calls route to the correct session by ID.

Logic Layer

• LLM engine: Connects to Qwen or any LLM to generate conversation replies (optional — you can bypass this and pass text directly to TTS)
• TTS engine: Modular design supporting EdgeTTS, GPT-SoVITS, CosyVoice, Tencent Cloud TTS, and others
• Voice cloning: Clone the target person's voice so the digital human speaks with a specific vocal identity
• Feature extraction: Synchronously extract acoustic features (Mel spectrogram) from audio, used as input to lip-sync inference

Rendering Layer

• Model inference: Use Wav2Lip, MuseTalk, or other models to generate lip-sync frames driven by audio features
• Post-processing: Smoothly blend the generated lip region back onto the original high-res video, preserving quality outside the mouth area

Streaming Layer

Three output modes for different deployment scenarios:

Plugin Extension Architecture

LiveTalking uses a decentralized registry (registry.py) for plugin-style extension of three module types:

TTS module: Register a new speech synthesis engine

from registry import register_tts
 
@register_tts("my_tts")
class MyTTS:
    def speak(self, text: str) -> bytes:
        # return audio data
        ...

Avatar module: Register a new rendering model (e.g., a custom lip-sync algorithm)

Output module: Register a new streaming output (e.g., a proprietary streaming protocol)

This design lets third-party developers extend the system without modifying core code.

Production Features

Interrupt and Resume

While the digital human is speaking, a new input can interrupt it — the current audio stops and new lip-sync rendering starts immediately. This is a production requirement for AI customer service: without it, users must wait for the current speech to finish before the digital human can respond, creating an unnatural experience.

Action Orchestration

When the digital human isn't speaking, play a custom idle video (nodding, smiling, body movement) rather than showing a static freeze frame, making the digital human feel more alive.

Full-Body Video Compositing

Composite the lip-sync region (face) onto a full-body video, producing a full-body digital human rather than just a talking head portrait.

Multi-Session Concurrency

• Silent sessions: concurrency is CPU-bound (video encoding)
• Speaking sessions: concurrency is GPU-bound (lip-sync inference)
• Higher output resolution consumes more CPU; mid-resolution (256px) gives the best performance-per-cost ratio

Admin Dashboard

/admin.html provides a real-time monitoring panel: view all active session states, manage global configuration, and force-stop any session.

Use Cases

LiveTalking also provides a dedicated virtual livestreamer solution: LiveStream, optimized specifically for 24/7 unattended live streaming.

Web Pages

Resources

Official Links

• 🌟 GitHub: lipku/LiveTalking
• 🌐 Website: livetalking.ai
• 📖 Documentation: doc.livetalking.ai
• 🖥️ Commercial demo: livetalking.top
• 🐛 FAQ: doc.livetalking.ai/docs/faq
• 💬 Discord: discord.gg/n5jSPCT3Uf
• 📺 Demos: Wav2Lip · MuseTalk

Summary

LiveTalking represents a typical path for digital human technology moving from research into production engineering. Three decisions stand out:

Multi-model support rather than a single bet: Wav2Lip (speed-first), MuseTalk (quality-first), and ER-NeRF (3D) each address different hardware and quality requirements. Users can match the model to their actual GPU budget and latency constraints. This is far more practical than projects that only support one rendering approach.

Four-layer decoupled architecture: API, logic, rendering, and streaming layers are independent, and TTS, rendering, and output modules are all replaceable via plugin registration. You can swap EdgeTTS for your company's proprietary TTS, or replace WebRTC with a private protocol, without touching the core code.

Interrupt handling is a genuine engineering challenge: Having a digital human interrupted mid-speech requires coordinating audio buffer queue flushing, GPU inference task cancellation, and immediate takeover by new input. Getting this right signals that the project was designed for real interactive scenarios, not just batch video generation.

Explicit performance baselines: The README provides actual FPS numbers for different GPU models and a hard requirement that inferfps ≥ 25 for real-time output. This lets users make informed hardware decisions before buying or renting a GPU.

If you're building an AI digital human product, LiveTalking provides a production-tested streaming engine that plugs directly into LLM + TTS stacks — one of the most fully engineered options currently available in the open-source community.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.