VRChat now offers native SDK-level Face Tracking support for avatars in the Open Beta client and Avatars SDK beta.
VRChat has technically supported Eye and Face Tracking for years over OSC (Open Sound Control) and the community-made VRCFaceTracking (VRCFT) app. However, avatars did not natively support full facial tracking, and instead needed to be configured with add-ons. Without them, avatars could only natively use Eye Tracking data for basic gaze and eyelid movement.
Now, VRChat’s latest Open Beta client on Steam and Avatars SDK Version 3.10.6-beta.1 and higher bring support for SDK-level authoring of Face Tracking. Initially described as “High Fidelity Face Tracking,” the feature allows creators to set up facial expressions for an avatar via a new dedicated section in the Avatar Descriptor component.
0:00
Native Face Tracking on the avatar Lumina from Extension Clothing with Hash’s integration package.
The new system maps facial expressions to blendshapes and animator parameters, supports the Unified Expressions standard from VRCFT, and integrates with VRChat’s audio-driven visemes, eye look, and eyelids. It does not currently support Blended Shapes or Simplified Parameters.
Even further, the system allows for network-synced Face Tracking without contributing to the avatar’s synced parameter budget. On paper, this should mean a reduced synced parameter load for the avatar, more visually consistent facial expressions for other users, and better performance.
VRChat’s native Face Tracking supports OSC methods like VRCFT (in its latest beta release on Steam) as well as VRChat’s Selfie Expression feature, which allows you to mirror facial expressions to an avatar on desktop PC and Android via a webcam or front-facing camera.
VRChat says that the functionality has been in closed beta testing for a while, and that the company has been gathering feedback from prominent community members and creators such as Hash, the VRCFT team, and many others.
Creator and developer Hash has updated several of their popular Booth avatar packages with beta variants that take advantage of the new SDK functionality, and you can make the switch if you own them.
0:00
Example video using the new Face Tracking SDK, provided to VRChat by Hash showing their Face Tracking work on the Lapwing avatar from Kuji.
Few headsets currently have both the hardware support and third-party APIs for Eye Tracking and camera-based Face Tracking. The most popular options are the Pico 4 Pro and Enterprise, Meta Quest Pro, and Samsung Galaxy XR, though there are also open-source and community-built solutions for other headsets such as the Meta Quest 3 and 3S. VRCFT is also able to pass data from headsets and add-ons that only support Eye Tracking, such as the Bigscreen Beyond 2e.
David Heaney
Notably, the upcoming Meta VR Glasses feature hardware support for both Eye and camera-based Face Tracking, using dedicated sensors for the eyes and capturing facial signals through the same downward-facing cameras used for Hand Tracking.
However, as spotted by dataminer and programmer Nori, Meta will not give third parties API access to the visual Face Tracking data source at launch as it did with the Meta Quest Pro, where apps including Steam Link and Virtual Desktop are already able to forward abstracted tracking data to OSC apps such as VRCFT. Instead, developers will only have access to Eye Tracking and must rely on the audio-driven data source, known as Audio to Expressions. Camera-based Face Tracking will be limited to first-party usage, such as the new realistic Hologram avatars.
This restriction has been met with some frustration from developers, and Meta Developer Advocate Dilmer Valecillos stated the feedback would be passed along to the appropriate teams.
A feedback post on the Meta VR Developer website requesting a camera-based Face Tracking API for Meta VR Glasses is currently the fourth most-voted investigation, and is marked as “Backlogged – A team has this on their backlog to address in the future.”
