Methodology

Every experiment participant used only one headset. The used headset was picked according to a pre-questionnaire, where the participant told about their age, current mood, tiredness and prior experience with 3D games. That way we avoided the biased situations, where randomly one device would get more ill-mood participants for example.

All participants played the Half-Life: Alyx game from the beginning until the end of the first chapter or until they had played for about 30 minutes. In the Audioshield game, participants played 3 songs and that took about 20 minutes. The film Henry was watched in full and that took about 12 minutes. Between every experience, the participants were asked for feedback about the experience and that feedback was used for comparing the devices.

Results

Here we present the found significant results. All results can be found from the Bachelor's thesis, which is available (in Estonian) here.

The data collected during the experiments was analyzed with the Kruskal-Wallis test, which determined if the ratings for one device were statistically different from the other devices. The significance threshold was chosen p ≤ 0.05.

Statistically significant difference was found with tracking during the Henry movie. HTC Vive Pro and Varjo VR-1 were found to be different from other devices. The reason for this difference could be the use of Revive software. Because Henry was initially created for only Oculus platforms, then that compatibility software was needed to view it with other devices.

Another statistical difference was found in the positioning of headphones. From the chart below it is visible that all the participants with Oculus Rift CV1 found the headphones comfortable. With the HTC Vive Pro headset 3 of the 14 and with the Varjo VR-1 headset 7 out of the 15 participants did not find it comfortable.

With Varjo VR-1 external headphones Audio‑Technica ATH-M50x were used and that could have caused such results. Because of the wide headband (on both sides of the head) with Varjo VR-1 both the left and right headphones laid halfway on top of the headband and many thought this was not comfortable.

One important discovery was the positioning of the tracking devices. With Oculus Rift CV1 the two tracking devices were on a table in front of the participant. When playing Half-Life: Alyx, the participants often turned their back towards the tracking devices and such tracking was lost for both the headset and controllers. Because of this, the participants were often instructed to turn around again.

The last important finding was with the device's field of view (FoV). In the Audioshield game the balls did not always come from the front, but also from the side. Several Varjo VR-1 participants said that because of the small FoV they did not notice the balls coming from the side and had to move their head more, which they thought hindered their experience.

No other significant results were found from the data collected from the experiments. The graphics below compare the data for each device. One data point on the graph is the average rating from one test subject across all three experiences.

Varjo VR-1 was rated to have the sharpest screen. This was followed by Oculus Rift CV1, Oculus Quest and HTC Vive Pro. There was no statistical difference detected (p = 0.488).

The device with the best tracking was the HTC Vive Pro. It was followed by Varjo VR-1, Oculus Rift CV1 and Oculus Quest. There was no statistical difference found (p = 0.164).

The best-tracked controllers were the Oculus Rift CV1 controllers (Oculus Touch Controller), which were followed by controllers used with Varjo VR-1 (HTC Vive Pro Controller), HTC Vive Pro (HTC Vive Pro Controller) and Oculus Quest (Oculus Touch Controller). There was no statistical difference found (p = 0.669).

The most comfortable controllers were rated the Varjo VR-1 controllers (HTC Vive Pro Controller). This was followed by Oculus Rift CV1 (Oculus Touch Controller), Oculus Quest (Oculus Touch Controller) and HTC Vive Pro controllers (HTC Vive Pro Controller). The controllers used with Varjo VR-1 were the same ones as those used with HTC Vive Pro. There was no statistical difference found (p = 0.976).

HTC Vive Pro was rated the most enjoyable (ie least bothersome) headset. It was followed by Oculus Quest, Oculus Rift CV1 and Varjo VR-1. There was no statistical difference found (p = 0.086).

The least hot headset was rated to be Varjo VR-1, which was followed by HTC Vive Pro, Oculus Rift CV1 and Oculus Quest. There was no statistical difference found (p = 0.242).

Although in some of these cases there seemed to be a difference, it cannot be stated for certain based on the data collected in our experiments. These results are further described in the Bachelor's thesis.

The device with the best retail price and first-time user experience ratio was Oculus Quest. It was followed by Oculus Rift, HTC Vive Pro and the worst ratio was with Varjo VR-1.