VRHow / XR Technology / VR tracking explained

VR tracking explained

Last checked October 11, 2026. This is an evidence-led explainer, not a hands-on test; VRHow has not independently tested devices or verified current product availability.

What the headset is actually measuring

A headset needs a pose: three rotational values (looking left/right, up/down and tilting) and, on a positional system, three translation values (moving forward/back, side to side, and up/down). That is the familiar six degrees of freedom (6DoF). A 3DoF device can know which way you are facing but cannot reliably tell that you leaned toward a virtual desk.

The fast part is the inertial measurement unit (IMU), normally accelerometers and gyroscopes. It reports changes at high frequency, so the system can respond quickly. But integrating those tiny measurements causes drift: a small error accumulated over time becomes a meaningful position error. Cameras provide slower, external-looking evidence. The tracking software matches visible features—corners, texture and contrast—from frame to frame, estimates how the headset moved, and fuses that estimate with the IMU. Microsoft describes this camera-plus-IMU approach as the basis of inside-out tracking. Microsoft’s tracking-system documentation explains the camera/IMU fusion.

That division of labor explains a common symptom: a brief camera obstruction may not instantly freeze the view because the IMU can bridge the gap, but confidence falls if visual correction does not return. The result can be drift, a jump, or a “lost tracking” warning—not necessarily a faulty controller.

Inside the tracking loop

Every frame, the runtime predicts where the headset will be when an image is displayed, then supplies the application with poses in a coordinate system. This prediction matters because rendering and display take time; the pose used for a frame must be close to the wearer’s pose at display time, not merely where they were when the camera frame arrived.

Applications do not need one universal map of Earth. They ask for relationships between spaces. The Khronos OpenXR specification defines common reference spaces such as VIEW, LOCAL, LOCAL_FLOOR and STAGE; a runtime may adjust an origin as its understanding improves, and not every system supports every space. The OpenXR XrSpace reference is the authoritative description. In practical terms, a virtual mug stays attached to a table because the runtime keeps updating the mug’s pose relative to a tracked reference, not because the game has perfect knowledge of the physical world.

Headset, controller and hand tracking are different problems

Head tracking asks “where is the camera rig?” Controller tracking asks “where is this handheld object?” A controller may contribute its own IMU and visible markers or LEDs; the headset’s cameras then infer its pose when it can see those signals. Hand tracking instead detects a hand’s shape and movement from camera images. These methods have different failure modes: a controller can be hidden behind your back, while a hand can be hard to identify in darkness, against a cluttered background, or when fingers overlap.

This is why a headset can still track your head while reporting that a controller is lost. It is also why “tracking quality” is not one universal number. Ask whether a claim refers to head pose, controller pose, hands, eye gaze, or an application’s own virtual boundary.

Why your room changes the result

Visual-inertial tracking needs stable, distinctive information. A decorated room gives the cameras landmarks; a blank white wall gives them little to match. Microsoft’s environment guidance also identifies very dark scenes, overexposure from direct sunlight, reflective surfaces, repetitive areas, moving objects and sudden lighting changes as potential causes of instability. Microsoft’s environment considerations document lists these lighting, feature and surface limitations. Those are engineering constraints, not a promise that every headset behaves identically: camera hardware, firmware and algorithms vary by device.

Practical setup follows directly from the mechanism: use even, comfortable lighting; avoid staring cameras at a window or mirror; give the system walls and objects with varied detail; keep cameras and lenses clean; and redraw the boundary after moving furniture or changing rooms. Do not treat a safety boundary as a physical barrier—it is software guidance, and you still need clear space to move.

Choosing and evaluating tracking: the questions that matter

Do not infer that a newer camera count, a higher refresh-rate claim, or a “room-scale” label guarantees better tracking in your room. Those specifications may be relevant inputs, but they do not replace evidence for the complete camera, sensor-fusion and runtime behavior. For a deeper distinction between camera-based and external systems, see inside-out vs outside-in tracking. Tracking also supports, but is not the same thing as, mixed reality; the broader categories are explained in VR vs AR vs mixed reality.

A safe, useful checklist

• Clear a play area and keep people, pets and obstacles out of arm’s reach.
• Use steady, moderate lighting rather than a dark room, harsh spotlight or direct sun.
• When tracking degrades, stop moving first; check camera obstructions, room features and boundary location before blaming the headset.
• Re-run room setup after a major furniture or lighting change.
• For work or training, test the actual room, controllers and software workflow; published tracking descriptions are not a substitute for site validation.

Sources and scope

The sources below were opened for this article. The Microsoft tracking page is a legacy Windows Mixed Reality document (last updated 2017); it is used for the mechanism it documents, not as a claim about every current headset. OpenXR describes an API abstraction, not a universal hardware accuracy guarantee.

• Microsoft Learn: Inside-out tracking
• Microsoft Learn: HoloLens environment considerations
• Khronos Group: OpenXR XrSpace reference
• Khronos Group: OpenXR 1.1 specification