FaceEdit3D

3D-Aware Face Editing via Warping-Guided Latent Direction Learning

Yuhao Cheng, Zhuo Chen, Xingyu Ren, Wenhan Zhu, Zhengqin Xu, Di Xu, Changpeng Yang, Yichao Yan

CVPR 2024

Paper | Project page

Requirements

• Please refer to eg3d installation guidelines: https://github.com/NVlabs/eg3d.

• Download pretrained eg3d models ffhqrebalanced512-128.pkl from https://github.com/NVlabs/eg3d/blob/main/docs/models.md, and put it in Codes.

• Install Mediapipe: https://github.com/google-ai-edge/mediapipe.

• Download pretrained warp-aware encoder models from https://drive.google.com/file/d/1L576COZyXX0Lwz8U71NbJ2Vtp7SZBZY0/view?usp=sharing, and put it in Codes.

Editing

• For one random seed, e.g., 0, you can obtain a facial image with its 3D landmarks.

python generate_3D_landmarks.py --seeds 0

• Then, you will obtain a source_3d_ldms.txt and a target_3d_ldms.txt, then you can edit the locations of ldms in target_3d_ldms.txt.

• You can obtain warped images with:

python gen_warped_images.py --outdir=out --network=ffhqrebalanced512-128.pkl

• You can obtain final edited images with:

python gen_inversion_warped_images.py --outdir=out --network=ffhqrebalanced512-128.pkl

# --edit_shape controls editing facial shape, and --edit_exp controls editing facial expressions.
# We also provide some editing directions in editing_direction, which can be used by --direction.

Definition of the landmarks

Acknowledgements

Some functions or scripts in this implementation are inspired by external sources. We appreciate the authors for their excellent work.

Here are some valuable resources we have benefited from:

• eg3d for the pretrained generative model.
• GOAE for the encoder structure and pretrained models.
• PTI for GAN inversion.
• WarpGAN for triplane Warp.
• Mediapipe for 2D facial landmarks detection.

Citation

@inproceedings{cheng20243d,
  title={3D-Aware Face Editing via Warping-Guided Latent Direction Learning},
  author={Cheng, Yuhao and Chen, Zhuo and Ren, Xingyu and Zhu, Wenhan and Xu, Zhengqin and Xu, Di and Yang, Changpeng and Yan, Yichao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={916--926},
  year={2024}
}