2D Eye Tracking - Xiexe/OldPatreonExclusiveThings GitHub Wiki

This is a shader based eye tracking solution that works as an extension/plugin on top of XSToon.

There are benefits to using this over the solution provided by VRChat, as it offers additional fine tuning customization that can also be controlled by animations/gestures. Additionally, eyes can be a much wider array of shapes besides a perfect sphere, avoiding the need to use hollow eye sockets with floating iris. The benefit of non-hollow eye sockets is more accurate shading, as light will hit an actual eye ball, instead of creating shadows inside a socket.

Please see the following YouTube video on how to set up the avatar and shader.

https://www.youtube.com/watch?v=fV6WsAgmXko

Eye Tracking Settings:

Left Right Adj.

Realigns the eye left and right.

Up Down Adj.

Realigns the eye up and down.

Attention Span

Percentage of time the eyes spend looking at its target.

Follow Power

How strongly the eyes follow the target.

Look Speed

How fast the eye shifts from following a target, to not following a target, or the transition time. Usually a value of .9 looks more natural, otherwise the eye will slowly shift to look around.

Refocus Frequency

How often the micro movements of the eye looks around the target of focus.

Iris Size

Useful for making custom gestures, such as the popular shrinking iris crazy face.

Follow Limit

How far from the center of the avatars vision other players need to be, to still be tracked by the eye. Higher values will allow players at the very peripheral of vision to still be tracked, while smaller values will cause the eyes to ignore tracking the player if they aren't closer to the center of view.

Offset Limit

While Follow Limit controls how far to track a player, Offset Limit controls how far the eye will rotate while tracking a player.

Guide to preparing a model and configuring the shader

The following written guide is provided as an alternative to the video for posterity, but assumes knowledge of how to use Blender and Unity. Please watch the video for an exact step by step guide.

There are three prerequisites to using the shader.

  1. Your eyes must be on a separate material.
  2. The texture should be transparent with only the iris/pupil in the center.
  3. You must be on the latest version of XSToon.

Setting up the model in Blender:

Note: The following explains how to set up the model, but keep in mind this is for the typical MMD style eye setup, where the iris is a floating mesh in front of a concave eye. There are additional ways to set this up, however these instructions follow the guide as presented on the original Youtube tutorial.

Delete the meshes that make up the floating iris for both eyes, but do not delete the concave eye socket. Select the first ring of edges that connects the eye socket to the face for only one eye, and duplicate it. Separate this ring by selection so it is a new mesh. Remove all shape keys so that Unity will not create a Skinned Mesh Renderer for it. Add a mirror modifier to this mesh and remove the armature modifier from it as well. Start selecting vertices on this mesh in vertical lines to fill in as quads, going from left to right, until the entire mesh is filled with quads/tris. It is recommended to pull this eye mesh back a bit and scale it up so that when the character blinks, you don't see the eye poking through the eye lid. Scale the eye mesh along the Z and X plane very slightly so it's very slightly larger than the socket. Change the scene view to Front Orthographic and then UV Unwrap the mesh using Project From View. Realign and scale the UV's to cover the transparent eye texture until the eyes on the avatar looks like they are in the right spot. The iris should be centered so that it looks like the character is looking straight ahead. Export the main armature/model to Unity, which will now have the model without eyes for the moment. On the eye mesh, apply the mirror modifier. Export as a separate mesh making sure only the eye mesh is selected, 'Experimental Apply Transforms', and 'Selected Objects' is checked.

Setting up the shader in Unity:

Drag the eye mesh into the scene hierarchy, and you'll note that the mesh is in the right spot, but without a material. Create a new material, and drop it onto the eye mesh. Drag the 2D Eye Tracking zip into the main XSToon plugins folder, and then extract it there. You should end up with a Patreon/EyeTracking folder within the plugins folder. Change the new material's shader from Standard to Xiexe->Toon2.0->Patreon->XSToon 2D Eye Tracking Cutout Assign the eye texture to the Main Texture slot. Make sure the texture import settings has the Alpha Is Transparent checked, and the Wrap Mode set to Clamp. You should immediately see the eyes start to move, if not, make sure Animated Materials is enabled for the scene view. Reparent the eyes mesh to the head bone of the armature.

Within the material shader settings, you can make adjustments as needed and then upload the avatar.