Skip to content

Hardware: Hand tracking#

Hand tracking is an advanced input method that allows users to interact with virtual environments using their bare hands, without the need for physical controllers. This technology is now integrated into next-generation VR headsets such as the Meta Quest 3 and the HTC Vive Focus 3, leveraging onboard cameras and sophisticated gesture recognition algorithms.

Why hand tracking ?#

Advantages Drawbacks
Enhanced Immersion: Using natural hand movements creates a more intuitive and realistic VR experience. Lower Precision: Compared to physical controllers, hand tracking can be less accurate, particularly for complex or fast actions.
Improved Accessibility: Ideal for new users or individuals with physical limitations, as no controllers are required. Lighting Sensitivity: Performance can be affected by poor lighting conditions or overly bright environments.
Ease of Use: Quick to set up—simply put on the headset and start interacting. Latency Issues: Some delay may be noticeable, especially in high-speed or precision-based applications.
Greater Freedom: Less hardware to manage, which improves comfort and mobility. Lack of Haptic Feedback: Without physical feedback, interactions can feel less tactile or responsive.

Hand tracking represents a significant step toward more natural and user-friendly virtual experiences. While it's not yet a full replacement for controllers in all use cases—particularly where precision and haptics are crucial—it’s a powerful feature that continues to evolve and expand the possibilities of immersive technology.

XR Twin is compatible with the hand tracking of the main virtual reality headsets on the market, as long as they comply with the OpenXR standard. Similarly, XR Twin can interpret the finger tracking of Manus virtual reality gloves or the Leap Motion hand motion capture device.

Hand tracking with Focus 3#

The HTC Vive Focus 3 supports hand tracking, so users can interact with virtual environments using their hands—no controllers required. By utilizing the headset’s built-in cameras and advanced computer vision algorithms, it can track hand and finger movements in real time, offering a more natural and intuitive way to engage with VR content.

Quick setup for hand tracking with Focus 3#

To use Hand tracking with your Focus 3 headset, follow this steps:

  1. Setup your headset (see Quick setup your Focus 3)
  2. Open XR Twin and open your project.
  3. From XR Twin Edition mode:
    1. From the main menu, select Project>Add User... to open the Add User popup,
    2. From the Add User popup, verify that you are using a VR+Hands avatar.
      If not, select VR as the display device, then click on the hand of the silhouette and select Hands as the hand tracking device. Click Add user to scene button to put this avatar to your scene and close the popup.
      Device-VR HandTracking_hand AddUserToSceneButton
    3. Click Start button to launch the simulation.
  4. When in Simulation:
    1. Put on your headset but keep your controllers down and still (on the table).
    2. Watch your hands.
    3. You are now using the Focus 3 hand tracking and can interact with physicalized objects as usual.

How can I teleport ?

If you want to move around, you'll need your controllers from time to time. Take your controllers in hand and use teleport or any other navigation mode to move around the scene. When you've reached the desired position, place your controllers back down and still again.
A few seconds later, hand tracking resumes.

The same applies to the VR menu (to change navigation mode or use one of the VR-tools).

Troubleshooting#

If you encounter any problems with Focus hand tracking.

In XR Twin software#
  • Sometimes, when the Focus 3 is in standby mode, hand tracking does not reactivate after Start Simulation mode. In this case, click Stop, wake up the helmet and then click Start again.
  • If the hands are not tracked, from XR Twin Edition mode, verify that the User is VR+Hands.
  • If the headset does not "move" the view (no tracking), verify and/or try the following:
    • From SteamVR, SteamVR is the Current OpenXR Runtime. If not, set it so.
    • From the headset's Library menu, VIVE Business Streaming is launched (connection is effective). If not, launch again.
    • Reopen VIVE Business Streaming. Sometimes you'll need to restart XR Twin in this case.
From the Focus 3 headset#

From in the headset, see the Head Up Display menus (HUD).

  • Hand tracking feature is active by default, but can be enable/disable.

    • From the VIVE Lobby menu, go to Settings > Inputs
    • Enable Hand tracking.

  • Hand tracking feature can be inactive in VIVE Streaming app.

    • With your head in the headset, double-click the Menu button on the left controller to display the VIVE Streaming Settings menu. Click on the Input tab. Hand tracking option must be checked.
In VIVE Streaming Settings software#
  • As an alternative to the HUD method described above, from the VIVE Streaming app, click the gear button to open the Settings popup, then select Input. Hand tracking option must be checked.
In SteamVR software#
  • From SteamVR menu, select Settings>OpenXR>MANAGE OPENXR API and set all layers "Off".

Hand tracking with Quest#

The Meta Quest 3 features advanced hand tracking capabilities built directly into the headset, allowing users to interact with virtual environments using only their hands—no controllers needed. By combining high-resolution cameras, depth sensors, and machine learning, the system can accurately detect and interpret complex hand and finger movements in real time.

This technology delivers a highly intuitive and immersive experience, making it well-suited for both consumer and professional use cases. Whether in training, scenarios or remote collaboration, hand tracking on the Quest 3 simplifies interaction and enhances user engagement.

Quick setup for hand tracking with Quest#

To use the Hand Tracking feature with your Quest headset, complete these settings and follow this steps:

  1. From Meta Quest Link, select Settings>Beta, browse to find DeveloperRuntime Features. Must be "on" to enable Hand tracking. If not, toggle it "on".
  2. From XR Twin, in Edition mode, switch to a VR+Hands User before clicking Start. In VR-Simulation mode, just put your controllers down (they must be still).

With these settings in mind, you can run through the Quest setup sequence. See Meta Quest quick setup section.

How can I teleport ?

If you want to move around, you'll need your controllers from time to time. Take your controllers in hand and use teleport or any other navigation mode to move around the scene. When you've reached the desired position, place your controllers back down and still again.
A few seconds later, hand tracking resumes.

The same applies to the VR menu (to change navigation mode or use one of the VR-tools).

Use double-taps on headset to active/inactive Passthrough.

Troubleshooting#

If you encounter any problems with Quest hand tracking, see Headsets section for tips and links.

Hand tracking with Manus gloves#

See Hardware: Manus gloves chapter for hand and finger tracking using Manus gloves (detailed process for Manus Quantum Metagloves).

<!--------

Hand tracking with Leapmotion#

See Hardware: Leapmotion chapter for hand and finger tracking using Leapmotion hardware.