XTalker

Diffusion-based talking-head avatar generation in motion space

XTalker generates talking-head videos from a single source image and an audio clip. An audio-conditioned flow-matching / diffusion model predicts motion in a compact LivePortrait-style keypoint space, which is then decoded back to video frames. The design is inspired by VASA-1, with additional improvements to lip-sync accuracy, head-pose stability, and long-form generation.

Features

• Motion-space diffusion / flow matching — audio-driven motion is predicted in a low-dimensional LivePortrait keypoint space instead of pixel space, making training and inference lightweight.
• Viseme-aware supervision — optional phoneme/viseme priors (via Montreal Forced Aligner) improve lip-sync accuracy.
• Long-form, streaming-friendly inference — segmental generation with overlap-window blending supports arbitrarily long output videos.
• LivePortrait-based rendering — reuses the LivePortrait appearance/motion pipeline for high-quality, identity-preserving frame synthesis.

Installation

conda create -n xtalker python=3.10 -y
conda activate xtalker
pip install -r requirements.txt

Pretrained Weights

LivePortrait

Download the LivePortrait base model weights, used for rendering:

pip install -U "huggingface_hub[cli]"
huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights \
    --exclude "*.git*" "README.md" "docs"

Only the base models are required:

pretrained_weights/
└── liveportrait/
    └── base_models/
        ├── appearance_feature_extractor.pth
        ├── motion_extractor.pth
        ├── spade_generator.pth
        └── warping_module.pth

If you cannot access Hugging Face directly, use a mirror:

export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights \
    --exclude "*.git*" "README.md" "docs"

Audio encoder and MFA

XTalker currently uses WavLM Base Plus-style audio features. Download the pretrained audio encoder and point wav2vec_model_path in config/default_config.yaml to the local folder:

pip install -U "huggingface_hub[cli]"
huggingface-cli download microsoft/wavlm-base-plus \
    --local-dir pretrained_weights/wavlm-base-plus

For phoneme/viseme supervision, install Montreal Forced Aligner (MFA) in a separate conda environment and download the English alignment models:

conda create -n mfa -c conda-forge montreal-forced-aligner -y
conda activate mfa
mfa --help
mfa model download dictionary english_us_mfa
mfa model download acoustic english_mfa

MFA is only needed when building TextGrid/viseme labels. The XTalker training environment and the MFA environment can stay separate.

Update liveportrait_model_path, liveportrait_retargeting_model_path, wav2vec_model_path, train_video_paths, train_video_wav_dir, audio_feature_dir, liveportrait_emb_path, and viseme_label_path in config/default_config.yaml to point at your local copies and generated data.

Data Preparation

Place your training videos (.mp4) in one or more folders listed under train_video_paths in config/default_config.yaml. Each video basename should be unique, because audio features and LivePortrait embeddings are matched by basename. The default frame rate is 25 FPS (target_fps=25).

The scripts in data_process/ are lightweight project scripts, so update the path constants near the top of each script before running them. Command-line configuration overrides should use --key=value.

1. Extract audio

export PYTHONPATH=.
python data_process/01ExtraAudio.py

01ExtraAudio.py reads the training videos and writes one .wav file per video to train_video_wav_dir. These WAV files are used by both WavLM feature extraction and MFA alignment.

2. Precompute WavLM audio features

python data_process/02precompute_audio_features_wavlm.py

02precompute_audio_features_wavlm.py loads the local WavLM model, resamples audio to 16 kHz when needed, chunks long audio for memory safety, and writes one .pt feature file per WAV into audio_feature_dir. Training loads these cached features instead of running the audio encoder every iteration.

02precompute_audio_features.py is the older wav2vec2 variant. Use it only if wav2vec_model_path and audio_feature_dir are set for wav2vec2 features.

3. Extract LivePortrait motion embeddings

python data_process/03preprocess_liveportrait_emb.py \
    --config=./config/default_config.yaml \
    --num_job_splits=1 \
    --job_id_from=0 \
    --job_id_to=1 \
    --preprocessed_liveportrait_emb_path=/path/to/liveportrait_emb

03preprocess_liveportrait_emb.py runs the LivePortrait motion extractor over each video frame and saves <video>.mp4.liveportrait_emb.pt. These tensors are the motion-space training targets used by the dataloader. Set liveportrait_emb_path in config/default_config.yaml to the same output directory used by --preprocessed_liveportrait_emb_path.

4. Build motion statistics

python data_process/04Calculate_dist.py

04Calculate_dist.py scans the LivePortrait embedding folder and computes per-dimension expression statistics. The resulting JSON should be referenced by dist_path in config/default_config.yaml and is used to normalize motion targets and select stable expression dimensions.

5. Build viseme labels

Viseme labels are used when use_viseme_prior=true. MFA requires paired audio and transcript files with matching stems, for example 0001.wav and 0001.txt.

If transcripts are missing, set step=1 in data_process/05preprocess_viseme.py and run:

python data_process/05preprocess_viseme.py

This step optionally extracts WAV files from cases and uses a local Whisper model to write transcript .txt files.

Then generate MFA TextGrid alignments. You can either set step=2 in data_process/05preprocess_viseme.py, or run MFA directly from the mfa environment:

conda activate mfa
mfa align /path/to/wav_txt_dir english_us_mfa english_mfa \
    /path/to/textgrid_out --clean --jobs 8

Finally convert MFA phone intervals into frame-level labels:

conda activate xtalker
python data_process/06prepocess_viseme_label.py \
    --textgrid_dir=/path/to/textgrid_out \
    --out_json=/path/to/viseme_labels.json \
    --fps=25

06prepocess_viseme_label.py reads the phones tier in each TextGrid and writes per-frame phoneme_ids, viseme15_ids, viseme6_ids, and segment-duration tracks. Set viseme_label_path to this JSON file before training.

6. Optional prior rebuilding

The repository already includes default priors under data_process/priors/. Only rebuild them when changing the dataset, motion statistics, or viseme template assets:

• 07viseme15_blended.py rebuilds viseme15_mouth_templates.json by blending image-derived viseme controls with dataset-derived LivePortrait statistics.
• 07viseme_map_live.py is the older 6-viseme template builder.
• viseme_analysis.py plots phoneme and viseme duration/frequency statistics from MFA TextGrid files.

Training

Train the diffusion / flow-matching model:

python train_flow_match.py \
    --config=./config/default_config.yaml \
    --batch_size=1 \
    --output_dir=./exp_output/run_train_flow_match

Any key in config/default_config.yaml can be overridden from the command line, e.g. --n_motion_frames=250. Checkpoints are written to <output_dir>/checkpoints, validation videos to <output_dir>/validation, and metrics to <output_dir>/train_metrics.log.

Inference

python train_flow_match.py \
    --config=./config/default_config.yaml \
    --load_from_checkpoint=./exp_output/run_train_flow_match/checkpoints/step_XXXXX \
    --batch_size=1 \
    --output_dir=./exp_output/run_inference_flow_match \
    --demos_videos_path=config/demo_videos.yaml \
    --only_run_log_validation=True

Inference reuses train_flow_match.py in validation-only mode. Outputs are saved under <output_dir>/validation/<validation_output_name>. inference.py implements the core generation logic (XTalkerPerformer) and can also be driven directly for custom pipelines; see config/eval_paths.yaml / config/demo_videos.yaml for the expected source-image/audio manifest format.

Notes & Known Limitations

• Training videos must be at target_fps=25 (xtalker/myutils.py).
• The released training recipe is validated with n_motion_frames=1 and overlap_win_size=0. Predicting more frames per step is supported, but you should add cross-attention to keep the attention computation tractable.

License

This project is released under the Apache License 2.0.

Acknowledgements

We gratefully acknowledge LivePortrait for its valuable contribution to high-quality, identity-preserving portrait animation and rendering. We would also like to thank JoyVASA for its motion decoder and warp/stitch rendering design, which our LivePortrait renderer wrapper is based on. Thanks to both teams for open-sourcing their work.