Sending data to VSeeFace - ebikatsudon/VTubeStudio GitHub Wiki

If you have a 3D model and an iPhone/iPad, you can use VTube Studio in conjunction with VSeeFace on your PC to take advantage of iOS's high-quality face tracking. This functionality is included in the iOS version of VTube Studio completely for free, so you don't have to buy the PRO version if you want to use this feature (any support is of course appreciated regardless).

For a list of supported iOS devices, check the wiki page about Hardware Requirements.

What is VSeeFace?

VSeeFace is a free, highly configurable face- and hand-tracking avatar puppeteering program for 3D Virtual YouTubers (as opposed to Live2D avatars). VSeeFace supports VRM and VSFAvatar avatar formats, with a focus on robust tracking and high image quality. The program is being continuously developed by Emiliana (@emiliana_vt) and Deat (@Virtual_Deat.)

VSeeFace mainly uses webcam tracking, but as of version 1.13.36n, it also has integrated support for high-quality iPhone tracking using VTube Studio. You do not need VTube Studio on your PC/Mac to use this feature, only on your iPhone/iPad. Your iPhone/iPad can directly connect to VSeeFace on your PC through your local network (USB connection is not supported).

If you are a 3D VTuber and have an iPhone, definitely give VSeeFace a try!!

How do I use VTube Studio to control my 3D model in VSeeFace?

Setting up VSeeFace

First, set up your 3D model in VSeeFace. You can refer to the tutorial section on the VSeeFace website if you need help getting set up. All iOS blendshapes are supported when using VSeeFace with VTube Studio on your iPhone/iPad.

Setting up VTube Studio

Next, download and start VTube Studio on your iPhone/iPad. In VTube Studio, there are three icons in the main menu relevant for using the app with VSeeFace.

  1. Camera preview:
    • IMPORTANT: MUST be clicked at least once after installing VTube Studio to initialize tracking.
    • Will show your face with the tracking mask.
    • Can be left open or closed while using VTube Studio with VSeeFace.
  2. Quick-calibrate:
    • Instantly calibrates the tracking without showing the camera preview.
    • Will reset the current face position as "looking towards the camera."
  3. Settings:
    • Displays all settings, including VSeeFace settings.

"VSeeFace Setup Part 1"|width=1024px

To access the VSeeFace settings, tap the Settings button in the main menu. In the first tab (gear icon), scroll down to the last section named "VSeeFace Config."

You will see the following settings:

"VSeeFace Setup Part 2"|width=512px

Just toggle the "Activate" button, and VTube Studio will now be ready to send blendshape data to VSeeFace. Once VSeeFace connects (and if you don't have the settings open), your mobile device's screen will automatically dim to save energy. You can disable this feature by deactivating "Streaming Mode" (also located in the first Settings tab, under "Stream to PC (WiFi)").

Important: You'll need the IP/port of your Apple device for VSeeFace. To view this, press the "Show IP list" button and make a note of the IPv4.

Connecting VSeeFace

Go back to VSeeFace on your PC. Open "Settings" and select "General Settings". Scroll down to the section labeled "iFacialMocap/FaceMocap3D/VTube Studio."

"VSeeFace Setup Part 3"|width=1100px

Select "VTube Studio" in the dropdown menu next to "Tracking app" and enter the IP/port you got from VTube Studio. The port is currently always 21412.

Next, check all the boxes for receiving the data: "Receive facial features" (blendshapes), "Receive head movement" and "Receive eye gaze". If you're already using blendshapes for eye gaze, you may not want to check the "Receive eye gaze" box.

At this point, you're all done! VSeeFace should now automatically connect to VTube Studio, and the tracking should be active. If you leave the tracking on when quitting VTube Studio on your iPhone/iPad, it will automatically resume the next time you start the app.

If you'd like, at this point you can also optimize other settings in VSeeFace, such as various tracking sensitivities, gaze strength, and so on.

In VTube Studio, it's recommended to keep the camera preview off and to turn the screen brightness all the way down to save energy and make sure your device doesn't get hot. Ideally, you should also charge it while using the app. Under normal circumstances, especially with newer devices, there shouldn't be any excessive heating while using VTube Studio. Older devices like the iPhone X may get a bit warm, but should not become hot.

Connectivity Issues

If VSeeFace and VTube Studio can't establish a connection, it's usually due to one of the following network issues:

  • Your firewall or antivirus/security software is blocking the connection. Add an exception for VSeeFace or disable the firewall/security software temporarily to see if that resolves the issue. Make sure to re-enable them afterwards!
  • Your PC and iPhone/iPad are not on the same WiFi/local network. If your router has multiple WiFi networks, try all of them.
  • Your network is not set to "Private" on your PC. If your network is set as "Public," change it to "Private."

For more detail on connectivity issues, refer to the Connection Issues & Troubleshooting page. All of the issues on that page also apply to the VSeeFace connection.

You can find more VSeeFace-specific info and troubleshooting steps on the VSeeFace website, https://www.vseeface.icu/.

Additional Resources

If you need help with VTube Studio, check the VTube Studio Discord.

If you have any issues, questions, or feedback for VSeeFace, please join the #vseeface channel of @Virtual_Deat’s Discord server: https://discord.gg/BjBgk7k