Facial animation with Motion Capture - CREW-Brussels/EXP GitHub Wiki

Live Link

Metahuman (if Metahumans will be used)

Apple ARkit

Apple ARkit face support

On iPhone —> Live Link app —> turn on Live Link ARkit —> Streaming Live Link settings —> IP address of your computer’s network 

In UE5 —> Window —> Virtual Production —> live link —> check if finds iPhone (when iPhone sees your face in front of the camera)

In Metahuman BP —> switch on Use ARkit face + ARkit Face Subj (iPhone)

!! Metahumans rely on curves, control rig, and bone-based deformations – instead of morph targets.

Character requirements:

52 blendshapes (morph targets): 

1. To create the shape keys, faceit addon for Blender can be used for the standards of ARkit.

In general, the names should be as follow: https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation 

If they need to be renamed (in UE5), follow this video.

1. Include the morph targets in the UE5 import window + check if they work in the skeletal mesh. 
2. In Skeletal mesh of the BP instance, check Update Animation in Editor
3. Create ABP
4. In Anim Graph, add Live Link Pose (iPhone) and connect to Output Pose. In Event Graph, add Evaluate Live Link Frame (iPhone + basic role).
5. Test with the Live Link App.
6. (Optional) Add Live Link Skeletal Animation component to the BP.

To be sure that your workflow is correct, repeat it with https://polywink.com/en/15-facial-animation-for-iphone-x.html – this one is especially prepared for ARkit.

https://faceit-doc.readthedocs.io/en/latest/ 

Requirements:

1. Position & Orientation The character has to be placed in the center of the scene (X,Y) and face the -Y direction. The height (Z) doesn't matter.

---

1. Face Geometry The character has to have a main face mesh (skin). This mesh should consist of a connected surface. In other words, all vertices in the face should be connected by edges. (Other facial parts, like teeth, tongue, eyeballs, hair, etc. do not need to be connected and can also be part of the same object.)

Best Practices

1. Neutral Expression The character should be modeled in a neutral expression (Eyes open, mouth shut).

---

1. Mirror Modifiers can lead to problems in the rigging process. You should consider to apply them. Note, applying a modifier fails when there are shape keys on the mesh. Check out the Apply Modifiers (with shape keys) operator to keep your shape keys safe.

---

1. Surface Deform Modifiers should be applied or disabled. (see 2)

---

1. Animations The character should not be animated when starting the rigging process. It is no problem if the objects are already attached to a body rig. If so, simply put the armature to Rest Position in Armature settings. This will suppress the animation. Alternatively, add a fake user to the action and remove it until you are done with rigging.

---

1. Clean Topology As Faceit involves regular rigging and animation workflows it is important to present clean topology with nice edge flow. You should present clean geometry, without double vertices and similar artefacts. You can find a lot of recources on good facial topology. Here is one of them.

https://faceit-doc.readthedocs.io/en/latest/setup/ 

1. Select mesh and click Register Face Objects
2. To check for geometry errors, click Check Geometry. If there is not one single connected surface (like in the 3D scan), there will be a warning message.
3. Add Existing Rig in the empty field, if available.
4. Assign vertex groups. Face Main is mandatory.

https://faceit-doc.readthedocs.io/en/latest/landmarks/ 

1. Place the landmarks and adjust to the shape

1. Project landmarks and adjust

1. Check from all sides. If not sure, consult the documentation link.
2. Manually or automatically define eyes. In the case of Wieke, manual is needed since there are no vertex groups for the eyes.

https://faceit-doc.readthedocs.io/en/latest/rigging/ 

1. Generate FaceIt Rig

1. Go back to Landmarks if some stuff is off
2. Bind with preset settings

https://faceit-doc.readthedocs.io/en/latest/create-expressions/ 

1. Load ARkit Expressions
2. Check expressions to see if any changes are needed to the mesh or landmarks or expressions themselves
3. Edit expressions 
4. When happy, Bake the expressions into Shape Keys.

Follow the steps from the first page.