Hand Tracking and Gesture for WebVR

Ningxin Hu / Intel / @NingxinHu

“They watch some demos or play a game and walk away saying how impressive it is, but almost everyone makes a remark about how they wish they had hands.”

- Brandon Jones

Blog post on Oculus Rift and HTC Vive review

Depth Camera

Media Capture Depth Stream Extensions

Two Kinds of Hand Tracking

  • Skeleton Tracking
  • Cursor Tracking

Skeleton Tracking

Crosswalk experimental hand tracking Web API

Skeleton Tracking

  • For avatar hand tracking
  • More computing
  • WebSocket impl by HW vendors, Web APIs?

Cursor Tracking

Intel RealSense SDK

Cursor Tracking

  • For accurate UI control
  • Less computing
  • Pointer Event Extension?

Gestures Recognition

Source: HoloLens Use Gestures

Gestures Recognition

  • High-level events
  • HW acceleration
  • Patents?
“So my prediction is that in five years we'll see good avatar hand tracking and gesture-based simple interface control”

- Michael Abrash

Oculus Connect 3 Keynote Speech