Face Tracking Template Setup - Adjerry91/VRCFaceTracking-Templates GitHub Wiki

Introduction

This setup will go over how to set up the face tracking template animations to drive blend shapes and eye movements to Unity for VRChat.

Prerequisites

  • Unity 2022 (Required after v5.2.0+)
  • VRChat SDK 3.7.0+ (Required after v6.3.2+ for VRC Constraints used in the face tracking debug panel)
  • VRCFaceTracking v5 setup and working. i.e. Test public face tracking avatars first before doing customs.
  • Avatar with SRanipal, ARkit, and UnifiedExpressions Blend Shapes (Case Sensitive) If you don’t know take look at this Face Tracking Conversion for naming
  • Avoid unpacking FBX, make sure eye bones are assigned rig configuration before unpacking.
  • Face tracking animations are pointed to the Body skinned mesh render by default. If face tracking shapes are on a different skinned mesh render, you will need to change VRCFury component to rewrite animations clips prefix. For example if face tracking shapes are on Face mesh you will need to rewrite Body to Face. Rewrite feature is NOT available on Modular Avatar prefabs. See Additional Setup - Non-Standard Mesh Names section

Face Tracking Menu Controls

Face tracking template has menu controls to allow toggling of the following:

  • EyeTrackingActive (Bool) - If true enables all eye tracking animations (VRCFaceTracking) and disables VRChat eye tracking
  • LipTrackingActive (Bool) - If true enables all lip tracking animations (VRCFaceTracking)
  • VisemesEnable (Bool) - If true enables Visemes. Tip - Recommend to leave VisemesEnable on with lip tracking as face tracking has network lag and is not useful for talking.
  • EyeDilationEnable (Bool) - If true enables dilation tracking
  • FacialExpressionsDisabled (Bool) - If true disables facial expressions. This does not do anything within the face tracking template, it is intended to be used in transitions for hand gestures on the FX layer to disable them when either eye or lip tracking is activated.

Prefab Setup

  1. Add/Import VRCFury or Modular Avatar
  2. Add/Import VRCFT Jerry's Templates from Jerry's VRCFT Templates Listing repository listing URL https://Adjerry91.github.io/VRCFaceTracking-Templates/index.json
  3. Go to the Packages/VRCFT - Jerry’s Templates/Prefabs folder. You will see the different available blend shape VRCFury and Modular Avatar Face Tracking prefab templates.
    image
    Tip - you can change the view of the project window text on the bottom right slider
    Note - There are two version of Unified Expressions template. Use normal one for TongueOut blend shape and use TongueSteps for TongueOutStep1 and TongueOutStep2 blend shapes.
  4. Drag the corresponding template prefab to the base of the avatar.
    image

Additional Setup - Modify Animators

Blocking hand gestures and blinking overlaying with the face tracking.

  1. Show Animator and Animation Windows
    image10\
  2. Create the following parameters:
    • EyeTrackingActive (Float)
      Note - in parameters list EyeTrackingActive is a bool but in the animator it has to be float because of the face tracking template type casting.
    • FacialExpressionsDisabled (Bool)
      image1
  3. Add conditions for all hand gestures transitions to disable hand emotes. Click plus icon and add FacialExpressionsDisabled and set it to False. The toggle will automatically turn off gestures when starting face or eye tracking.
    image3
  4. If blink is within the FX animator, add transition logic to disable the blinking animation.
    • EyeTrackingActive Greater 0.5 is equivalent to True
    • EyeTrackingActive Less 0.5 is equivalent to False
      image7 image8 Note the multiple arrows on the exit transitions is equivalent to “OR” logic statement. Right click transition from and select make transition then select exit as the destination. Click the exit transition arrows and you will see a new transition added to the list. Expand the settings, uncheck the exit time and change the transition duration desired, this is the time delay for the transition.

Additional Setup - Non-Standard Mesh Names

This section is only needed when the face tracking blend shapes are NOT on Skinned Mesh Renderer named “Body” in the root of the avatar.

  1. Unpack the prefab
  2. Under VRCFury Full Controller go to “Advanced Options”
  3. Click the “+” to add Path Rewrite Rules
  4. Add “Body” to the “If animated path has this prefix” (This is same for all avatars)
  5. Add the name of the mesh to the “Then” section”; this can also be a path to the skinned mesh. In this example face tracking shapes are on “Zeedah” skinned mesh renderer.
    image
  6. Debug panel will also need to be remapped to the mesh with the face tracking blendshapes. Change to the new mesh name in the “Name of skinned mesh object on avatar”
    image

Testing in Unity

Testing in Unity verifies your model works without needing to go VRChat. It easier to debug problems in Unity than in game.

  1. Import Lyuma Av3Emulator (Use version 3.2.6, there is bugs with 3.2.6+ that break face tracking)
  2. Click on play mode in Unity
  3. Enable Avatar 3.0 Emulator
    image6
  4. Click on the avatar to test. You will see a Gesture Manager Av 3 Menu and Lyuma Av 3 Runtime on the avatar.
    image18
  5. Go in menu an enable eye and lip tracking
    image9
  6. Scroll down the inspect to the “User Input” section and expand floats.
    image12
  7. Test each of the sync float parameters to see to test that they control the avatar.
    image4