Agricultural Training System with Gazing-Point Detection Function using Head-Mounted Display: HTC Vive Pro Eye and Virtual Reality–based Unity System

In this study, we develop a virtual reality (VR)–based agricultural (hereinafter, agri-) support technique to assist in training newcomers and trainees in agricultural work. The system consists of a head-mounted display (HMD)—a HTC Vive Pro Eye—gaming personal computer, and peripheral components. The HMD-based system consists of the following: (1) a precision eye-tracking system for tracking and interpreting eye movements to enable lifelike interactions, better manage GPU workload, and simplify input and navigation; (2) a room area tracker up to 100 m²; and (3) a VR-space experience with unmatched tracking accuracy with SteamVR™ Tracking. The agri-situation considered in this study is non-specific tomato harvesting performed by inexperienced agri-workers. The study aims to (1) measure and analyze persons’ cognition and behavior indicators in VR-based environments that simulate the work site, (2) provide suggestions regarding analyses of processes from the cognition of targets to specific behaviors using objective indexes; and (3) realize in advance the verification and comparison of improvements of measures related to specific agri-works without the need to utilize actual agri-work sites. Specifically, we utilize and apply the eye-tracking function incorporated in the HMD, in addition, we develop a Unity-based VR system with sound notification to indicate the validity of eye-tracking and the motion of manual agri-workers and managers. Subsequently, we conduct experimental trials in a non-specific room. In the trials, subjects wearing the HMD system sequentially gaze at figures of small red tomatoes in the VR spaces. In the process, the VR system plays alerts to notify subjects when they gaze at or miss the target tomatoes. The system provides quick and accurate operations, and the eye-tracking function of the system differs from existing agri-training-oriented techniques and products. The system has several advantages such as lower cost compared with existing similar mechanical systems found in the literature and similar commercial products. In addition, the Unity-based system is a minimal and flexibly scalable system that can be adjusted to suit future studies and expansion for different agri-situations. This study consists of five phases: (1) designing and confirming the validity of the system; (2) accumulating image data on outdoor farmland; (3) constructing the entire system and tuning various minor system settings like program parameters and other specifications; (4) executing experiments in an indoor room; and (5) assessing and discussing the results and gathering comments from the subjects and presenting the characteristics of these trials. We consider that, from the limited trials, the system can be judged to be valid to some extent in certain situations. However, we could not perform broader or more generalizable experiments using the system. We present experimental characteristics and numerical ranges related to the trials, particularly noting speed and likelihood of mistakes concerning the system’s practical operations. The novel achievements of this study lie in the fusion of the latest HMD and Unity-based agricultural training facilities. In future, agri-workers and managers can use the proposed system for training, particularly for eye movement. Furthermore, we believe that, by combining this system with other existing systems, agriculture can be greatly improved.