Technical Papers
May 8, 2020

Robust Hybrid Approach of Vision-Based Tracking and Radio-Based Identification and Localization for 3D Tracking of Multiple Construction Workers

Publication: Journal of Computing in Civil Engineering
Volume 34, Issue 4

Abstract

Information of construction entity identity and real-time location reveals where specific construction resources are at any given time and thus is a critical prerequisite to the context-aware jobsite safety management. Most existing studies use either a vision-based or radio-based approach to automatically track construction entities, which, however, are bounded by the limitations of the applied technology. Vision-based tracking can achieve high localization accuracy but suffers from identity (ID) switch and fragmentation errors when multiple workers are in close proximity or occluded. In contrast, radio-based tracking is reliable in object detection and identification but less accurate in localization. This study proposes a hybrid framework that fuses results obtained from vision-based tracking and radio-based identification and localization for the 3D tracking of multiple construction workers. Compared to traditional fusion approaches that directly fuse locations extracted from these two approaches, the proposed method treats vision-based tracking as the main source to extract the object trajectory. Radio-based identification and localization results are used as a supplementary source to augment anonymous visual tracks with identity information and correct errors (e.g., false positives) in vision-based object detection, resulting in ID-linked 3D trajectories. In addition, a searching algorithm is introduced to recover possible missed detections in one camera view from the corresponding observations in the other view by applying a sliding window to search for regions with the most similar appearance along epipolar line. The newly created method has been validated using two indoor experiments. The results show that the new approach for fusing vision- and radio-based results increases the overall accuracy from 88% and 87% to 95% and 90%, compared to using a vision-based approach alone. The integration of radio-based identification is much more robust than using a vision system alone, as it allows the recovery of the same entity ID after the trajectory is fragmented and results in fewer fragmentations that last longer than 0.2 s.

Get full access to this article

View all available purchase options and get full access to this article.

Data Availability Statement

Some or all data, models, or code generated or used during the study are available from the corresponding author by request, including videos and radio data from two experiments, annotated locations, and calibration information.

References

Ahn, J. W., T. W. Chang, S. H. Lee, and Y. W. Seo. 2016. “Two-phase algorithm for optimal camera placement.” Sci. Program. https://doi.org/10.1155/2016/4801784.
Altahir, A. A., V. S. Asirvadam, N. H. Hamid, P. Sebastian, N. Saad, R. Ibrahim, and S. C. Dass. 2017. “Modeling multicamera coverage for placement optimization.” IEEE Sens. Lett. 1 (6): 1–4. https://doi.org/10.1109/LSENS.2017.2758371.
Asadi, K., H. Ramshankar, M. Noghabaei, and K. Han. 2019. “Real-time image localization and registration with BIM using perspective alignment for indoor monitoring of construction.” J. Comput. Civ. Eng. 33 (5). https://doi.org/10.1061/(ASCE)CP.1943-5487.0000847.
Battacharyya, A. 1943. “On a measure of divergence between two statistical populations defined by their probability distributions.” Bull. Calcutta Math. Soc. 35: 99–109. https://doi.org/10.1038/157869b0.
Brilakis, I., M. W. Park, and G. Jog. 2011. “Automated vision tracking of project related entities.” Adv. Eng. Inf. 25 (4): 713–724. https://doi.org/10.1016/j.aei.2011.01.003.
Cai, H., A. R. Andoh, X. Su, and S. Li. 2014. “A boundary condition based algorithm for locating construction site objects using RFID and GPS.” Adv. Eng. Inf. 28 (4): 455–468. https://doi.org/10.1016/j.aei.2014.07.002.
Cai, J., Y. Zhang, and H. Cai. 2019a. “Integrating positional and attentional cues for construction working group identification: A long short-term memory based machine learning approach.” In Computing in Civil Engineering 2019: Data, Sensing, and Analytics, 35–42. Reston, VA: ASCE.
Cai, J., Y. Zhang, and H. Cai. 2019b. “Two-step long short-term memory method for identifying construction activities through positional and attentional cues.” Autom. Constr. 106 (Oct): 102886. https://doi.org/10.1016/j.autcon.2019.102886.
Chen, H., X. Luo, and J. Ke. 2018. “Multisource fusion framework for environment learning–free indoor localization.” J. Comput. Civ. Eng. 32 (5): 04018040. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000782.
Costin, A. M., and J. Teizer. 2015. “Fusing passive RFID and BIM for increased accuracy in indoor localization.” Visualization Eng. 3 (1): 17. https://doi.org/10.1186/s40327-015-0030-6.
Fang, Q., H. Li, X. Luo, L. Ding, T. M. Rose, W. An, and Y. Yu. 2018. “A deep learning-based method for detecting non-certified work on construction sites.” Adv. Eng. Inf. 35 (Jan): 56–68. https://doi.org/10.1016/j.aei.2018.01.001.
Gao, X. S., X. R. Hou, J. Tang, and H. F. Cheng. 2003. “Complete solution classification for the perspective-three-point problem.” IEEE Trans. Pattern Anal. Mach. Intell. 25 (8): 930–943. https://doi.org/10.1109/TPAMI.2003.1217599.
Hartley, R., and A. Zisserman. 2003. Multiple view geometry in computer vision. Cambridge, UK: Cambridge University Press.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Jeelani, I., H. Ramshankar, K. Han, A. Albert, and K. Asadi. 2019. “Real-time hazard proximity detection—Localization of workers using visual data.” In Computing in civil engineering 2019: Data, sensing, and analytics, 281–289. Reston, VA: ASCE.
Jung, D., T. Teixeira, and A. Savvides. 2010. “Towards cooperative localization of wearable sensors using accelerometers and cameras.” In Proc., IEEE INFOCOM. New York: IEEE.
Kim, J., Y. Ham, Y. Chung, and S. Chi. 2019. “Systematic camera placement framework for operation-level visual monitoring on construction jobsites.” J. Constr. Eng. Manage. 145 (4): 04019019. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001636.
Konstantinou, E., and I. Brilakis. 2018. “Matching construction workers across views for automated 3D vision tracking on-site.” J. Constr. Eng. Manage. 144 (7): 04018061 https://doi.org/10.1061/(ASCE)CO.1943-7862.0001508.
Lee, Y. J., and M. W. Park. 2019. “3D tracking of multiple onsite workers based on stereo vision.” Autom. Constr. 98 (Feb): 146–159. https://doi.org/10.1016/j.autcon.2018.11.017.
Li, H., G. Chan, J. K. W. Wong, and M. Skitmore. 2016. “Real-time locating systems applications in construction.” Autom. Constr. 63 (Mar): 37–47. https://doi.org/10.1016/j.autcon.2015.12.001.
Liang, C. J., K. M. Lundeen, W. McGee, C. C. Menassa, S. H. Lee, and V. R. Kamat. 2019. “A vision-based marker-less pose estimation system for articulated construction robots.” Autom. Constr. 104 (Aug): 80–94. https://doi.org/10.1016/j.autcon.2019.04.004.
Liu, D., J. Chen, S. Li, and W. Cui. 2018. “An integrated visualization framework to support whole-process management of water pipeline safety.” Autom. Constr. 89 (May): 24–37. https://doi.org/10.1016/j.autcon.2018.01.010.
Liu, D., Y. Wu, S. Li, and Y. Sun. 2016. “A real-time monitoring system for lift-thickness control in highway construction.” Autom. Constr. 63 (Mar): 27–36. https://doi.org/10.1016/j.autcon.2015.12.004.
Luo, X., H. Li, H. Wang, Z. Wu, F. Dai, and D. Cao. 2019. “Vision-based detection and visualization of dynamic workspaces.” Autom. Constr. 104 (Aug): 1–13. https://doi.org/10.1016/j.autcon.2019.04.001.
Mahalanobis, P. C. 1936. “On the generalized distance in statistics.” Proc. Nat. Inst. Sci. 2: 49–55. https://doi.org/10.1145/1390156.1390302.
Mandeljc, R., S. Kovačič, M. Kristan, and J. Perš. 2013. “Tracking by identification using computer vision and radio.” Sensors (Switzerland) 13 (1): 241–273. https://doi.org/10.3390/s130100241.
MathWorks. 2019. “Computer vision toolbox.” Accessed June 4, 2019. https://www.mathworks.com/products/computer-vision.html.
Memarzadeh, M., M. Golparvar-Fard, and J. C. Niebles. 2013. “Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors.” Autom. Constr. 32 (Jul): 24–37. https://doi.org/10.1016/j.autcon.2012.12.002.
Milan, A., L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler. 2016. “MOT16: A benchmark for multi-object tracking.” Preprint, submitted March 2, 2016. http://arxiv.org/abs/1603.00831.
Mohebbi, P., E. Stroulia, and I. Nikolaidis. 2017. “Sensor-data fusion for multi-person indoor location estimation.” Sensors (Switzerland) 17 (10): 2377. https://doi.org/10.3390/s17102377.
Munkres, J. 1957. “Algorithms for the assignment and transportation problems.” J. Soc. Indus. Appl. Math. 5 (1): 32–38. https://doi.org/10.1137/0105003.
Papaioannou, S., A. Markham, and N. Trigoni. 2017. “Tracking people in highly dynamic industrial environments.” IEEE Trans. Mob. Comput. 16 (8): 2351–2365. https://doi.org/10.1109/TMC.2016.2613523.
Papaioannou, S., H. Wen, A. Markham, and N. Trigoni. 2015. “Fusion of radio and camera sensor data for accurate indoor positioning.” In Proc., 11th IEEE Int. Conf. on Mobile Ad Hoc and Sensor Systems, MASS 2014, 109–117. New York: IEEE.
Park, J. W., K. Kim, and Y. K. Cho. 2016. “Framework of automated construction-safety monitoring using cloud-enabled BIM and BLE mobile tracking sensors.” J. Constr. Eng. Manage. 143 (2): 05016019. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001223.
Park, J. W., X. Yang, Y. K. Cho, and J. Seo. 2017. “Improving dynamic proximity sensing and processing for smart work-zone safety.” Autom. Constr. 84 (Dec): 111–120. https://doi.org/10.1016/j.autcon.2017.08.025.
Park, M. W., and I. Brilakis. 2012. “Construction worker detection in video frames for initializing vision trackers.” Autom. Constr. 28 (Dec): 15–25. https://doi.org/10.1016/j.autcon.2012.06.001.
Park, M. W., and I. Brilakis. 2016. “Continuous localization of construction workers via integration of detection and tracking.” Autom. Constr. 72 (Dec): 129–142. https://doi.org/10.1016/j.autcon.2016.08.039.
Park, M. W., C. Koch, and I. Brilakis. 2011. “Three-dimensional tracking of construction resources using an on-site camera system.” J. Comput. Civ. Eng. 26 (4): 541–549. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000168.
Ren, S., K. He, R. Girshick, and J. Sun. 2017. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Roberts, D., and M. Golparvar-Fard. 2019. “End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level.” Autom. Constr. 105 (Sep): 102811. https://doi.org/10.1016/j.autcon.2019.04.006.
Son, H., H. Seong, H. Choi, and C. Kim. 2019. “Real-time vision-based warning system for prevention of collisions between workers and heavy equipment.” J. Comput. Civ. Eng. 33 (5): 04019029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000845.
Su, X., S. Li, C. Yuan, H. Cai, and V. R. Kamat. 2014. “Enhanced boundary condition–based approach for construction location sensing using RFID and RTK GPS.” J. Constr. Eng. Manage. 140 (10): 04014048 https://doi.org/10.1061/(ASCE)CO.1943-7862.0000889.
Thaljaoui, A., T. Val, N. Nasri, and D. Brulin. 2015. “BLE localization using RSSI measurements and iRingLA.” In Proc., IEEE Int. Conf. on Industrial Technology. New York: IEEE.
Topak, F., M. K. Pekeriçli, and A. M. Tanyer. 2018. “Technological viability assessment of Bluetooth low energy technology for indoor localization.” J. Comput. Civ. Eng. 32 (5): 04018034. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000778.
Xiao, Y., V. R. Kamat, and C. C. Menassa. 2019. “Human tracking from single RGB-D camera using online learning.” Image Vision Comput. 88 (Aug): 67–75. https://doi.org/10.1016/j.imavis.2019.05.003.
Yan, X., H. Zhang, and H. Li. 2019. “Estimating worker-centric 3D spatial crowdedness for construction safety management using a single 2D camera.” J. Comput. Civ. Eng. 33 (5): 04019030 https://doi.org/10.1061/(ASCE)CP.1943-5487.0000844.
Yang, J., O. Arif, P. A. Vela, J. Teizer, and Z. Shi. 2010. “Tracking multiple workers on construction sites using video cameras.” Adv. Eng. Inf. 24 (4): 428–434. https://doi.org/10.1016/j.aei.2010.06.008.
Yang, X., H. Li, T. Huang, X. Zhai, F. Wang, and C. Wang. 2018. “Computer-aided optimization of surveillance cameras placement on construction sites.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1110–1126. https://doi.org/10.1111/mice.12385.
Yu, X., and A. Ganz. 2010. “Global identification of tracklets in video using long range identity sensors.” In Proc., IEEE Int. Conf. on Advanced Video and Signal Based Surveillance, AVSS 2010, 361–368. New York: IEEE.
Yuan, C., S. Li, and H. Cai. 2016. “Vision-based excavator detection and tracking using hybrid kinematic shapes and key nodes.” J. Comput. Civ. Eng. 31 (1): 04016038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000602.
Zhang, B., Z. Zhu, A. Hammad, and W. Aly. 2018a. “Automatic matching of construction onsite resources under camera views.” Autom. Constr. 91 (Jul): 206–215. https://doi.org/10.1016/j.autcon.2018.03.011.
Zhang, H., X. Yan, and H. Li. 2018b. “Ergonomic posture recognition using 3D view-invariant features from single ordinary camera.” Autom. Constr. 94 (Oct): 1–10. https://doi.org/10.1016/j.autcon.2018.05.033.
Zhang, Z. 2000. “A flexible new technique for camera calibration.” IEEE Trans. Pattern Anal. Mach. Intell. 22 (11): 1330–1334. https://doi.org/10.1109/34.888718.
Zhao, J., O. Seppänen, A. Peltokorpi, B. Badihi, and H. Olivieri. 2019. “Real-time resource tracking for analyzing value-adding time in construction.” Autom. Constr. 104 (Aug): 52–65. https://doi.org/10.1016/j.autcon.2019.04.003.
Zhao, J., R. Yoshida, S. S. Cheung, and D. Haws. 2013. “Approximate techniques in solving optimal camera placement problems.” Int. J. Distrib. Sens. Netw. 9 (11): 241913. https://doi.org/10.1155/2013/241913.
Zhu, Z., M. W. Park, C. Koch, M. Soltani, A. Hammad, and K. Davari. 2016a. “Predicting movements of onsite workers and mobile equipment for enhancing construction site safety.” Autom. Constr. 68 (Aug): 95–101. https://doi.org/10.1016/j.autcon.2016.04.009.
Zhu, Z., X. Ren, and Z. Chen. 2016b. “Visual tracking of construction jobsite workforce and equipment with particle filtering.” J. Comput. Civ. Eng. 30 (6): 04016023. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000573.
Zhu, Z., X. Ren, and Z. Chen. 2017. “Integrated detection and tracking of workforce and equipment from construction jobsite videos.” Autom. Constr. 81 (Sep): 161–171. https://doi.org/10.1016/j.autcon.2017.05.005.
Zhuang, Y., J. Yang, Y. Li, L. Qi, and N. El-Sheimy. 2016. “Smartphone-based indoor localization with bluetooth low energy beacons.” Sensors (Switzerland) 16 (5): 596. https://doi.org/10.3390/s16050596.
Zou, H., L. Xie, Q.-S. Jia, and H. Wang. 2014. “Platform and algorithm development for a RFID-based indoor positioning system.” Unmanned Syst. 2 (3): 279–291. https://doi.org/10.1142/S2301385014400068.

Information & Authors

Information

Published In

Go to Journal of Computing in Civil Engineering
Journal of Computing in Civil Engineering
Volume 34Issue 4July 2020

History

Received: Sep 10, 2019
Accepted: Feb 7, 2020
Published online: May 8, 2020
Published in print: Jul 1, 2020
Discussion open until: Oct 8, 2020

Permissions

Request permissions for this article.

Authors

Affiliations

Jiannan Cai [email protected]
Ph.D. Candidate, Lyles School of Civil Engineering, Purdue Univ., 550 Stadium Mall Dr., West Lafayette, IN 47907. Email: [email protected]
Hubo Cai, M.ASCE [email protected]
Associate Professor, Lyles School of Civil Engineering, Purdue Univ., 550 Stadium Mall Dr., West Lafayette, IN 47907 (corresponding author). Email: [email protected]

Metrics & Citations

Metrics

Citations

Download citation

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited by

View Options

Get Access

Access content

Please select your options to get access

Log in/Register Log in via your institution (Shibboleth)
ASCE Members: Please log in to see member pricing

Purchase

Save for later Information on ASCE Library Cards
ASCE Library Cards let you download journal articles, proceedings papers, and available book chapters across the entire ASCE Library platform. ASCE Library Cards remain active for 24 months or until all downloads are used. Note: This content will be debited as one download at time of checkout.

Terms of Use: ASCE Library Cards are for individual, personal use only. Reselling, republishing, or forwarding the materials to libraries or reading rooms is prohibited.
ASCE Library Card (5 downloads)
$105.00
Add to cart
ASCE Library Card (20 downloads)
$280.00
Add to cart
Buy Single Article
$35.00
Add to cart

Get Access

Access content

Please select your options to get access

Log in/Register Log in via your institution (Shibboleth)
ASCE Members: Please log in to see member pricing

Purchase

Save for later Information on ASCE Library Cards
ASCE Library Cards let you download journal articles, proceedings papers, and available book chapters across the entire ASCE Library platform. ASCE Library Cards remain active for 24 months or until all downloads are used. Note: This content will be debited as one download at time of checkout.

Terms of Use: ASCE Library Cards are for individual, personal use only. Reselling, republishing, or forwarding the materials to libraries or reading rooms is prohibited.
ASCE Library Card (5 downloads)
$105.00
Add to cart
ASCE Library Card (20 downloads)
$280.00
Add to cart
Buy Single Article
$35.00
Add to cart

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share