Estimating the Visual Attention of Construction Workers from Head Pose Using Convolutional Neural Network-Based Multi-Task Learning

The visual attention of construction workers is an important indicator to assess their situational awareness and infer their intention for reducing construction injuries and improving construction site safety. The eye-tracking technology has been adopted in several studies to directly measure the gaze direction and determine workers’ visual attention. However, eye-trackers are expensive and wearing them may disturb normal operations. Considering the increasing use of surveillance videos and the availability of construction images, it is of great potential to estimate workers’ visual attention from imagery data, which, however, has not been well exploited by existing studies. This paper presents a convolutional neural network (CNN)-based multi-task learning framework to estimate the visual attention of construction workers from head pose using low-resolution images. Visual attention is approximated by head yaw and pitch orientation. The problem is formulated as a multi-task image classification problem, where the first task is head yaw classification, and the second task is head pitch classification. A CNN-based multi-task learning framework is designed to jointly learn two tasks, with shared layers capturing the commonality between tasks, and task-specific layers modeling the uniqueness of individual tasks. Compared to traditional single-task learning mechanism that trains different classifiers for each task, the proposed approach leverages the commonality of relevant tasks and captures the shared representation, which can significantly improve the efficiency and performance. The results suggest the proposed multi-learning framework can achieve an accuracy of 76.5% for head yaw estimation and 88.7% for head pitch estimation, better than the performance obtained using conventional single task learning.