Technical Papers
Oct 29, 2020

Physiological Data Models to Understand the Effectiveness of Drone Operation Training in Immersive Virtual Reality

Publication: Journal of Computing in Civil Engineering
Volume 35, Issue 1

Abstract

Data collection using unmanned aerial vehicles (UAVs) in construction and heavy civil projects is subject to compliance with strict operational rules and safety regulations. Both the US Federal Aviation Administration (FAA) and the European Union Aviation Safety Agency (EASA) require drone operators to keep the drone in sight and avoid flying near people or other objects. From the perspective of the operator, remaining in standing or sitting position while always looking up to monitor the drone movements can cause awkward body postures, stress, and fatigue. Coupled with the mental load resulting from delegated tasks, this could potentially put the drone mission, people, and property at risk. This research investigates the reliability of using the drone operator’s physiological indexes and self-assessments to predict performance, mental workload (MWL), and stress in immersive virtual reality training and outdoor deployment. A user study was carried out to collect physiological data using wearable devices and design general population and group-specific prediction models. Results show that in 83% of cases, these models can predict performance, MWL, and stress levels accurately or within one level. This paper contributes to the core body of knowledge by providing a scalable approach to objectively quantifying performance, MWL, and stress that can be used to design adaptive training systems for drone operators. Personalized models of physiological signals are presented as reliable indexes to describing the outcome of interest. Scalability is achieved through the application of generalizable machine learning models that learn the interdependencies between physiological and self-assessment inputs and their association with corresponding outcomes.

Get full access to this article

View all available purchase options and get full access to this article.

Data Availability Statement

All data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.

Acknowledgments

Some of the experiments described in this paper were conducted using an Anafi drone that was generously provided by Parrot, Inc. We would like to thank Mr. Jerome Bouvard, Director of Strategic Partnerships at Parrot Inc. Any opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not represent the views of Parrot Inc.

References

Acharya, R., N. Kannathal, and S. M. Krishnan. 2004. “Comprehensive analysis of cardiac health using heart rate signals.” Physiol. Meas. 25 (5): 1139. https://doi.org/10.1088/0967-3334/25/5/005.
Akhavian, R., and A. H. Behzadan. 2015. “Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers.” Adv. Eng. Inf. 29 (4): 867–877. https://doi.org/10.1016/j.aei.2015.03.001.
Akhavian, R., and A. H. Behzadan. 2016. “Smartphone-based construction workers’ activity recognition and classification.” Autom. Constr. 71 (2): 198–209. https://doi.org/10.1016/j.autcon.2016.08.015.
Anwar, N., M. Amir, and F. Ahmed. 2018. “Construction monitoring and reporting using drones and unmanned aerial vehicles (UAVs).” In Proc., 10th Int. Conf. on Construction in the 21st Century (CITC-10). Greenville, NC: East Carolina Univ.
Arellano, J. L. H., J. A. C. Martinez, J. N. Serratos Pérez, and J. L. G. Alcaraz. 2015. “Relationship between workload and fatigue among Mexican assembly operators.” Int. J. Phys. Med. Rehabil. 3 (6): 1000315. https://doi.org/10.4172/2329-9096.1000315.
Ayaz, H., S. Bunce, P. Shewokis, K. Izzetoglu, B. Willems, and B. Onaral. 2012. “Using brain activity to predict task performance and operator efficiency.” In Proc., Int. Conf. on Brain Inspired Cognitive Systems, 147–155. New York: Springer.
Ballabio, D. 2015. “A MATLAB toolbox for principal component analysis and unsupervised exploration of data structure.” Chemom.Intell. Lab. Syst. 149 (Part B): 1–9. https://doi.org/10.1016/j.chemolab.2015.10.003.
Begum, S., S. Barua, R. Filla, and M. U. Ahmed. 2014. “Classification of physiological signals for wheel loader operators using multi-scale entropy analysis and case-based reasoning.” Expert Syst. Appl. 41 (2): 295–305. https://doi.org/10.1016/j.eswa.2013.05.068.
Benedek, M., and C. Kaernbach. 2010. “A continuous measure of phasic electrodermal activity.” J. Neurosci. Methods 190 (1): 80–91. https://doi.org/10.1016/j.jneumeth.2010.04.028.
Bevis, F. W., F. W. Bevis, D. R. Towill, and C. Finnlear. 1970. “Prediction of operator performance during learning of repetitive tasks.” Int. J. Prod. Res. 8 (4): 293–305. https://doi.org/10.1080/00207547008929848.
Bliss, J. P., P. D. Tidwell, and M. A. Guest. 1997. “The effectiveness of virtual reality for administering spatial navigation training to firefighters.” Presence Teleoperators Virtual Environ. 6 (1): 73–86. https://doi.org/10.1162/pres.1997.6.1.73.
Boksem, M. A. S., T. F. Meijman, and M. M. Lorist. 2005. “Effects of mental fatigue on attention: An ERP study.” Cognit. Brain Res. 25 (1): 107–116. https://doi.org/10.1016/j.cogbrainres.2005.04.011.
Bowman, D. A., and R. P. McMahan. 2007. “Virtual reality: How much immersion is enough?” Computer 40 (7): 36–43. https://doi.org/10.1109/MC.2007.257.
Cacioppo, J. T., and L. G. Tassinary. 1990. Principles of psychophysiology: Physical, social, and inferential elements. Cambridge, MA: Cambridge University Press.
Carreiras, C., A. P. Alves, A. Lourenço, F. Canento, H. Silva, and A. Fred. 2018. “BioSPPy: Biosignal processing in Python (2015–).” Accessed February 24, 2020. https://github.com/PIA-Group/BioSPPy.
Chellali, R., and S. Hennig. 2013. “Is it time to rethink motion artifacts? Temporal relationships between electrodermal activity and body movements in real-life conditions.” In Proc., Humaine Association Conf. on Affective Computing and Intelligent Interaction, 330–335. New York: IEEE.
Contreras, P., R. Canetti, and E. R. Migliaro. 2006. “Correlations between frequency-domain HRV indices and lagged Poincaré plot width in healthy and diabetic subjects.” Physiol. Meas. 28 (1): 85. https://doi.org/10.1088/0967-3334/28/1/008.
Corner, G. W., D. E. Saxbe, T. Chaspari, H. F. Rasmussen, L. Perrone, C. Pettit, M. Friendly, A. C. Timmons, and G. Margolin. 2019. “Compassion in a heartbeat: Physiology during couples’ loss discussions.” J. Social Pers. Relat. 36 (6): 1671–1694. https://doi.org/10.1177/0265407518770267.
Cribbet, M. R., T. W. Smith, B. N. Uchino, B. R. W. Baucom, and J. B. Nealey-Moore. 2020. “Autonomic influences on heart rate during marital conflict: Associations with high frequency heart rate variability and cardiac pre-ejection period.” Bio. Psychol. 151 (Mar): 107847. https://doi.org/10.1016/j.biopsycho.2020.107847.
Dasu, T., Y. Kanza, and D. Srivastava. 2018. “Geofences in the sky: Herding drones with blockchains and 5G.” In Proc., 26th ACM SIGSPATIAL Int. Conf. on Advances in Geographic Information Systems, 73–76. New York: Association for Computing Machinery.
Dawson, M., A. Shell, and D. Fillion. 2016. “The electrodermal system.” In Handbook of psychophysiology, edited by J. Cacioppo, L. Tassinary, and G. Berntson, 217–243. Cambridge, UK: Cambridge University Press. https://doi.org/10.1017/9781107415782.010.
De Winter, J., P. M. Van Leeuwen, and R. Happee. 2012. “Advantages and disadvantages of driving simulators: A discussion.” In Proc., Measuring behavior, 47–50. Wageningen, Netherlands: Noldus Information Technology.
Enders, C. K. 2003. “Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data.” Psychol. Methods 8 (3): 322. https://doi.org/10.1037/1082-989X.8.3.322.
European Union Aviation Safety Agency. 2019. “AMC & GM to commission implementing regulation (EU) 2019-947—Issue 1.” Accessed April 26, 2020. https://www.easa.europa.eu/document-library/acceptable-means-of-compliance-and-guidance-materials/amc-gm-commission.
Federal Aviation Administration. 2018. FAA aerospace forecast, FY2018-38. Washington, DC: USDOT.
Federal Aviation Administration. 2019. “Part 107 waivers.” Accessed April 17, 2020. https://www.faa.gov/uas/commercial_operators/part_107_waivers/.
Fowles, D. C., M. J. Christie, R. Edelberg, W. W. Grings, D. T. Lykken, and P. H. Venables. 1981. “Publication recommendations for electrodermal measurements.” Psychophysiology 18 (3): 232–239. https://doi.org/10.1111/j.1469-8986.1981.tb03024.x.
Gheisari, M., E. P. Karan, H. C. Christmann, J. Irizarry, and E. N. Johnson. 2015. “Investigating unmanned aerial system (UAS) application requirements within a department of transportation.” In Proc., Transportation Research Board 94th Annual Meeting. Washington, DC: Transportation Research Board.
Girard, J. M. 2014. “CARMA: Software for continuous affect rating and media annotation.” J. Open Res. Software 2 (1). https://doi.org/10.5334/jors.ar.
Goldstein, H., W. Browne, and J. Rasbash. 2002. “Multilevel modelling of medical data.” Stat. Med. 21 (21): 3291–3315. https://doi.org/10.1002/sim.1264.
Haneuse, S., and S. Bartell. 2011. “Designs for the combination of group-and individual-level data.” Epidemiol. (Cambridge, Mass.) 22 (3): 382. https://doi.org/10.1097/EDE.0b013e3182125cff.
Hart, S. G. 2006. “NASA-task load index (NASA-TLX); 20 years later.” In Proc., Human Factors and Ergonomics Society Annual Meeting, 904–908. Los Angeles: SAGE.
Hart, S. G., and C. D. Wickens. 1990. “Workload assessment and prediction.” In Manprint, 257–296. Dordrecht, Netherlands: Springer.
Hjortskov, N., D. Rissén, A. K. Blangsted, N. Fallentin, U. Lundberg, and K. Søgaard. 2004. “The effect of mental stress on heart rate variability and blood pressure during computer work.” Eur. J. Appl. Physiol. 92 (1–2): 84–89. https://doi.org/10.1007/s00421-004-1055-z.
Howse, W. R. 2011. Knowledge, skills, abilities, and other characteristics for remotely piloted aircraft pilots and operators. Gurnee, IL: Damos Aviation Services.
Hutter, F., H. H. Hoos, and K. Leyton-Brown. 2013. “Identifying key algorithm parameters and instance features using forward selection.” In Proc., Int. Conf. on Learning and Intelligent Optimization, 364–381. New York: Springer.
Irizarry, J., M. Gheisari, and B. N. Walker. 2012. “Usability assessment of drone technology as safety inspection tools.” J. Inf. Technol. Constr. 17 (Sep): 194–212.
Israel, K., and R. Nesbit. 2004. Defense science board study on unmanned aerial vehicles and uninhabited combat aerial vehicles. Washington, DC: Defense Science Board.
Kantor, L., N. S. Endler, R. J. Heslegrave, and N. L. Kocovski. 2001. “Validating self-report measures of state and trait anxiety against a physiological measure.” Curr. Psychol. 20 (3): 207–215. https://doi.org/10.1007/s12144-001-1007-2.
Khuwaja, K. S. A., B. S. Chowdhry, K. F. Khuwaja, V. O. Mihalca, and R. C. Ţarcă. 2018. “Virtual reality based visualization and training of a quadcopter by using RC remote control transmitter.” In Proc., IOP Conf. Series: Materials Science and Engineering, 52008. London: Institute of Physics Publishing.
Kim, K. H., S. W. Bang, and S. R. Kim. 2004. “Emotion recognition system using short-term monitoring of physiological signals.” Med. Biol. Eng. Comput. 42 (3): 419–427. https://doi.org/10.1007/BF02344719.
Kim, S., and J. Irizarry. 2019. “Human performance in UAS operations in construction and infrastructure environments.” J. Manage. Eng. 35 (6): 04019026. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000715.
Kwak, S. K., and J. H. Kim. 2017. “Statistical data preparation: Management of missing values and outliers.” Korean J. Anesthesiology 70 (4): 407–411. https://doi.org/10.4097/kjae.2017.70.4.407.
Lalak, N. 2010. “Restorative benefits of the natural environment: Enhancing restoration of directed attention & stress reduction through raising levels of cognitive awareness & physical encounter.” Ph.D. thesis, Faculty of Architecture, Design, and Planning, Univ. of Sydney.
Latteur, P., S. Goessens, and C. Mueller. 2016. “Masonry construction with drones.” In Proc., IASS Annual Symposia, 1–10. Madrid, Spain: International Association for Shell and Spatial Structures.
Ledalab. 2010. “Documentation.” Accessed April 26, 2020. http://www.ledalab.de/documentation.htm.
Mantovani, F., G. Castelnuovo, A. Gaggioli, and G. Riva. 2003. “Virtual reality training for health-care professionals.” Cyberpsychol. Behav. 6 (4): 389–395. https://doi.org/10.1089/109493103322278772.
Mehta, R. K., and M. J. Agnew. 2012a. “Effects of physical and mental demands on shoulder muscle fatigue.” Supplement, Work 41 (S1): 2897–2901. https://doi.org/10.3233/WOR-2012-0541-2897.
Mehta, R. K., and M. J. Agnew. 2012b. “Influence of mental workload on muscle endurance, fatigue, and recovery during intermittent static work.” Eur. J. Appl. Physiol. 112 (8): 2891–2902. https://doi.org/10.1007/s00421-011-2264-x.
Murphy, R. R. 2020. “Humans and robots in off-normal applications and emergencies.” In Vol. of 962 Proc. Advances in Human Factors in Robots and Unmanned Systems. Advances in Intelligent Systems and Computing, edited by J. Chen, 171–180. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-20467-9_16.
Pan, Z., A. D. Cheok, H. Yang, J. Zhu, and J. Shi. 2006. “Virtual reality and mixed reality for virtual learning environments.” Comput. Graphics (Pergamon) 30 (1): 20–28. https://doi.org/10.1016/j.cag.2005.10.004.
Papakonstantinou, G., and F. Gritzali. 1981. “Syntactic filtering of ECG waveforms.” Comput. Biomed. Res. 14 (2): 158–167. https://doi.org/10.1016/0010-4809(81)90033-1.
Papakonstantinou, G., E. Skordalakis, and F. Gritzali. 1986. “An attribute grammar for QRS detection.” Pattern Recognit. 19 (4): 297–303. https://doi.org/10.1016/0031-3203(86)90055-5.
Pi, Y., N. D. Nath, and A. H. Behzadan. 2020. “Convolutional neural networks for object detection in aerial imagery for disaster response and recovery.” Adv. Eng. Inf. 43 (Jan): 101009. https://doi.org/10.1016/j.aei.2019.101009.
Platania-Solazzo, A., T. M. Field, J. Blank, and F. Seligman. 1992. “Relaxation therapy reduces anxiety in child and adolescent psychiatric patients.” Acta Paedopsychiatrica: Int. J. Child Adolescent Psychiatry. 55 (2): 115–120.
Poh, M.-Z., N. C. Swenson, and R. W. Picard. 2010. “A wearable sensor for unobtrusive, long-term assessment of electrodermal activity.” IEEE Trans. Biomed. Eng. 57 (5): 1243–1252. https://doi.org/10.1109/TBME.2009.2038487.
Postal, G. R., W. Pavan, and R. Rieder. 2016. “A virtual environment for drone pilot training using VR devices.” In Proc., XVIII Symp. on Virtual and Augmented Reality (SVR), 183–187. New York: IEEE.
Rigas, G., Y. Goletsis, and D. I. Fotiadis. 2011. “Real-time driver’s stress event detection.” IEEE Trans. Intell. Transp. Syst. 13 (1): 221–234. https://doi.org/10.1109/TITS.2011.2168215.
Robar, D. E. 1978. “The comparative effectiveness of live, video-taped and audio-taped group relaxation training on the ability to reduce physiological arousal and self-report measures of anxiety.” Ph.D. dissertation, Dept. of Education, Univ. of Massachusetts Amherst.
Robnik-Šikonja, M., and I. Kononenko. 2003. “Theoretical and empirical analysis of ReliefF and RReliefF.” Mach. Learn. 53 (1–2): 23–69. https://doi.org/10.1023/A:1025667309714.
Sakib, M. N. 2019. “Wearable technology to assess the effectiveness of virtual reality training for drone operators.” Master’s thesis, Dept. of Construction Science, Texas A&M Univ.
Sakib, M. N., T. Chaspari, C. Ahn, and A. Behzadan. 2020. “An experimental study of wearable technology and immersive virtual reality for drone operator training.” In Proc., 27th Int. Workshop on Intelligent Computing in Engineering, edited by T. Ungureanu and L. Constantin, 154–163. Berlin: Hartmann.
Sakib, M. N., M. Yadav, T. Chaspari, and A. H. Behzadan. 2019. “Coupling virtual reality and physiological markers to improve public speaking performance.” In Proc., 19th Int. Conf. on Construction Applications of Virtual Reality (CONVR2019), 171–180. Middlesbrough, UK: Teesside Univ.
Schuemie, M. J., P. van der Straaten, M. Krijn, and C. A. P. G. van der Mast. 2001. “Research on presence in virtual reality: A survey.” CyberPsychol. Behav. 4 (2): 183–201. https://doi.org/10.1089/109493101300117884.
Shi, Y., M. H. Nguyen, P. Blitz, B. French, S. Fisk, F. De la Torre, A. Smailagic, D. P. Siewiorek, M. Al’Absi, and E. Ertin. 2010. “Personalized stress detection from physiological measurements.” In Proc., Int. Symp. on Quality of Life Technology, 28–29. Pittsburgh: Carnegie Mellon Univ.
Sinharay, S. 2010. “International encyclopedia of education.” In An overview of statistics in education, edited by P. Peterson, E. Baker, and B. McGaw. Amsterdam, Netherlands: Elsevier.
Spielberger, C. D. 2010. “State-trait anxiety inventory.” In The Corsini encyclopedia of psychology, edited by I. B. Weiner, and W. E. Craighead, 1. https://doi.org/10.1002/9780470479216.corpsy0943.
Taelman, J., S. Vandeput, A. Spaepen, and S. Van Huffel. 2009. “Influence of mental stress on heart rate and heart rate variability.” In Proc., 4th European Conf. of the Int. Federation for Medical and Biological Engineering, 1366–1369. Berlin: Springer.
Thayer, J. F., S. S. Yamamoto, and J. F. Brosschot. 2010. “The relationship of autonomic imbalance, heart rate variability and cardiovascular disease risk factors.” Int. J. Cardiol. 141 (2): 122–131. https://doi.org/10.1016/j.ijcard.2009.09.543.
Trahanias, P., and E. Skordalakis. 1990. “Syntactic pattern recognition of the ECG.” IEEE Trans. Pattern Anal. Mach. Intell. 12 (7): 648–657. https://doi.org/10.1109/34.56207.
Trinon, H. 2019. Immersive technologies for virtual reality-case study: Flight simulator for pilot training. Liège, Belgique: Université de Liège.
Vine, S. J., L. Uiga, A. Lavric, L. J. Moore, K. Tsaneva-Atanasova, and M. R. Wilson. 2015. “Individual reactions to stress predict performance during a critical aviation incident.” Anxiety Stress Coping 28 (4): 467–477. https://doi.org/10.1080/10615806.2014.986722.
Witmer, B. G., and M. J. Singer. 1998. “Measuring presence in virtual environments: A presence questionnaire.” Presence 7 (3): 225–240. https://doi.org/10.1162/105474698565686.
Yadav, M., T. Chaspari, J. Kim, and C. R. Ahn. 2018. “Capturing and quantifying emotional distress in the built environment.” In Proc., Workshop on Human-Habitat for Health (H3): Human-Habitat Multimodal Interaction for Promoting Health and Well-Being in the Internet of Things Era, 9. New York: Association for Computing Machinery (ACM).
Yadav, M., M. N. Sakib, K. Feng, T. Chaspari, and A. Behzadan. 2019. “Virtual reality interfaces and population-specific models to mitigate public speaking anxiety.” In Proc., 8th Int. Conf. on Affective Computing and Intelligent Interaction (ACII), 1–7. New York: IEEE.
Yu, L., and H. Liu. 2003. “Feature selection for high-dimensional data: A fast correlation-based filter solution.” In Proc., 20th Int. Conf. on Machine Learning (ICML-03), 856–863. Palo Alto, CA: Association for the Advancement of Artificial Intelligence.
Zhang, X., Z. Ye, J.-H. Zhu, and S. Li. 2002. “Unmanned aerial vehicle flight simulation and training system based on virtual reality.” Acta Ssmulata Systematica Sinica 2002 (8): 8.

Information & Authors

Information

Published In

Go to Journal of Computing in Civil Engineering
Journal of Computing in Civil Engineering
Volume 35Issue 1January 2021

History

Received: Apr 29, 2020
Accepted: Aug 5, 2020
Published online: Oct 29, 2020
Published in print: Jan 1, 2021
Discussion open until: Mar 29, 2021

Permissions

Request permissions for this article.

Authors

Affiliations

Ph.D. Student, Dept. of Multidisciplinary Engineering, Texas A&M Univ., 3127 TAMU, College Station, TX 77843. ORCID: https://orcid.org/0000-0003-0390-792X. Email: [email protected]
Theodora Chaspari [email protected]
Assistant Professor, Dept. of Computer Science and Engineering, Texas A&M Univ., 3127 TAMU, College Station, TX 77843. Email: [email protected]
Amir H. Behzadan, M.ASCE [email protected]
Associate Professor, Dept. of Construction Science, Texas A&M Univ., 3137 TAMU, College Station, TX 77843 (corresponding author). Email: [email protected]

Metrics & Citations

Metrics

Citations

Download citation

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited by

View Options

Get Access

Access content

Please select your options to get access

Log in/Register Log in via your institution (Shibboleth)
ASCE Members: Please log in to see member pricing

Purchase

Save for later Information on ASCE Library Cards
ASCE Library Cards let you download journal articles, proceedings papers, and available book chapters across the entire ASCE Library platform. ASCE Library Cards remain active for 24 months or until all downloads are used. Note: This content will be debited as one download at time of checkout.

Terms of Use: ASCE Library Cards are for individual, personal use only. Reselling, republishing, or forwarding the materials to libraries or reading rooms is prohibited.
ASCE Library Card (5 downloads)
$105.00
Add to cart
ASCE Library Card (20 downloads)
$280.00
Add to cart
Buy Single Article
$35.00
Add to cart

Get Access

Access content

Please select your options to get access

Log in/Register Log in via your institution (Shibboleth)
ASCE Members: Please log in to see member pricing

Purchase

Save for later Information on ASCE Library Cards
ASCE Library Cards let you download journal articles, proceedings papers, and available book chapters across the entire ASCE Library platform. ASCE Library Cards remain active for 24 months or until all downloads are used. Note: This content will be debited as one download at time of checkout.

Terms of Use: ASCE Library Cards are for individual, personal use only. Reselling, republishing, or forwarding the materials to libraries or reading rooms is prohibited.
ASCE Library Card (5 downloads)
$105.00
Add to cart
ASCE Library Card (20 downloads)
$280.00
Add to cart
Buy Single Article
$35.00
Add to cart

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share