Madelineorla

MIMESIS

Personal R&D · Python · Personal IP · Present WIP

A real-time facial performance capture pipeline built for webcam use, for portable and easy Real-Time Face. MediaPipe extracts 478 landmarks from webcam input, feeding a custom-trained actor-specific AI model that predicts 52 ARKit-compatible blend shapes. Then streams them live into Unreal Engine 5 via LiveLink, subsequently driving a MetaHuman rig.

Range of Motion capture, used to train the actor specific AI model. 31 guided facial expressions that aid in understanding personalised blendshape data. This is an early iteration demonstrating the full pipeline functioning end-to-end.
First working Mimesis stream: webcam input processed through the AI model, blendshape data streamed via LiveLink Hub into UE5, driving a MetaHuman rig in real time.

.How Mimesis works ~

1. Capture & Landmark Extraction. OpenCV reads a continuous webcam feed. Frame by frame, it is passed to MediaPipe FaceLandmarker, which extracts 478 facial landmarks in real time. It then tracks the precise geometry of the face including lips, brows, jaw, cheekbones, eyes, so on and so forth.

2. Blend Shapes. The landmark data is processed in two different ways. One way is using geometry to compute the values of the blendshapes mathematically. I have made sure this model covers areas such as the ‘jaw open’ and ‘brow furrow’ where many other models fall short. Additionally, there is a pipeline whereby you can train theAI on individual-specific faces. You perform 31 guided FACS expressions during a ROM. The AI will now train itself with this data rather than library data which would have applied a generic average.

3.Retargeting & Calibration. There is an option to set a new neutral pose to eliminate any offsets created in the ROM process. It understands and can map the full ARKit library of expressions.

4. LiveLink. 52 ARKit compatible blendshape values and head rotation data gets shrunk into livelink data packets and streamed via UDP to LiveLink Hub. This subsequently drives the MetaHuman rig in Unreal Engine 5.

Main Software Used~