Abstract
This paper presents a time- and cost-effective elevation determination method for earthwork operations using ready-to-fly imaging drones and deep learning technologies. The proposed method is named the fast pixel grid/group matching and elevation determination (Fast-PGMED) algorithm. The input data are a pair of approximate 2:1-scale top-view images, and the output is the determined elevation map for the scanned station. Feature matching of the two multiscale images is conducted by calculating correlations between target patch predictions (via DeepMatchNet, a fully convolutional network) and potential target patches (via virtual elevation model). The overall processing time is about 21 s (including 5 s for low-high orthoimage assembly, 3 s for patch feature generation, and 13 s for pixel matching) to process a 2,500-pixel grid, and the generated elevation values are as accurate as photogrammetry (within 5-cm error) but took much less time. Moreover, the developed method has been evaluated with two different drones. Volume measurement was quickly conducted via 2D elevation maps and accurately estimated via dense point clouds and Civil 3D.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The training and testing data sets are available in (Jiang 2021a). The Python codes are available from the corresponding author upon reasonable request.
Acknowledgments
This research was financially supported by the McShane Endowment fund at Marquette University. The authors are thankful to Mr. Weijing Nie (graduate student at Guangzhou University) for his help in capturing images at experimental site C. In addition, the authors are grateful to the reviewers for their valuable comments and feedback.
References
Aguilar, R., M. F. Noel, and L. F. Ramos. 2019. “Integration of reverse engineering and non-linear numerical analysis for the seismic assessment of historical adobe buildings.” Autom. Constr. 98 (Feb): 1–15. https://doi.org/10.1016/j.autcon.2018.11.010.
Arias, P., J. Herráez, H. Lorenzo, and C. Ordóñez. 2005. “Control of structural problems in cultural heritage monuments using close-range photogrammetry and computer methods.” Comput. Struct. 83 (21–22): 1754–1766. https://doi.org/10.1016/j.compstruc.2005.02.018.
Barazzetti, L., F. Remondino, and M. Scaioni. 2010. “Automation in 3D reconstruction: Results on different kinds of close-range blocks.” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXVIII (5): 55–61.
Bay, H., A. Ess, T. Tuytelaars, and L. Van Gool. 2008. “Speeded-up robust features (SURF).” Comput. Vision Image Understanding 110 (3): 346–359. https://doi.org/10.1016/j.cviu.2007.09.014.
Bethmann, F., and T. Luhmann. 2015. “Semi-global matching in object space.” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XL-3/W2 (3W2): 23–30, https://doi.org/10.5194/isprsarchives-XL-3-W2-23-2015.
Chen, K., W. Lu, F. Xue, P. Tang, and L. H. Li. 2018. “Automatic building information model reconstruction in high-density urban areas: Augmenting multi-source data with architectural knowledge.” Autom. Constr. 93: 22–34. https://doi.org/10.1016/j.autcon.2018.05.009.
Choe, J., K. Joo, F. Rameau, and I. S. Kweon. 2021. Stereo object matching network. New York: IEEE. https://doi.org/10.1109/ICRA48506.2021.9562027.
Chollet, F. 2015. “Keras: The python deep learning library.” Accessed March 7, 2020. https://keras.io/.
Chollet, F. 2021. “Dropout layer.” Accessed October 10, 2021. https://keras.io/api/layers/regularization_layers/dropout.
DJI. 2020. “Phantom 4 Pro V2.0.” Accessed March 7, 2020. https://www.dji.com/phantom-4-pro-v2/specs.
DJI. 2021. “Mavic 2.” Accessed April 27, 2021. https://www.dji.com/mavic-2/info#specs.
Du, J.-C., and H.-C. Teng. 2007. “3D laser scanning and GPS technology for landslide earthwork volume estimation.” Autom. Constr. 16 (5): 657–663. https://doi.org/10.1016/j.autcon.2006.11.002.
Furukawa, Y., and J. Ponce. 2010. “Accurate, dense, and robust multiview stereopsis.” IEEE Trans. Pattern Anal. Mach. Intell. 32 (8): 1362–1376. https://doi.org/10.1109/TPAMI.2009.161.
Haur, C. J., L. S. Kuo, C. P. Fu, Y. L. Hsu, and C. Da Heng. 2018. “Feasibility study on UAV-assisted construction surplus soil tracking control and management technique.” IOP Conf. Ser. Mater. Sci. Eng. 301: 012145. https://doi.org/10.1088/1757-899X/301/1/012145.
Hughes, L. H., D. Marcos, S. Lobry, D. Tuia, and M. Schmitt. 2020. “A deep learning framework for matching of SAR and optical imagery.” ISPRS J. Photogramm. Remote Sens. 169: 166–179. https://doi.org/10.1016/j.isprsjprs.2020.09.012.
Hughes, L. H., N. Merkle, T. Burgmann, S. Auer, and M. Schmitt. 2019. “Deep learning for SAR-optical image matching.” In Proc., IGARSS 2019-2019 IEEE Int. Geoscience and Remote Sensing Symp., 4877–4880. New York: IEEE.
Inzerillo, L., G. Di Mino, and R. Roberts. 2018. “Image-based 3D reconstruction using traditional and UAV datasets for analysis of road pavement distress.” Autom. Constr. 96: 457–469. https://doi.org/10.1016/j.autcon.2018.10.010.
Jiang, Y. 2021a. “Automated ortho-image and elevation-map stitching.” Accessed May 11, 2021. https://www.yuhanjiang.com/research/DT/PGMED/Stitching.
Jiang, Y. 2021b. “Dataset.” Accessed May 5, 2021. https://www.yuhanjiang.com/dataset.
Jiang, Y., and Y. Bai. 2020a. “Determination of construction site elevations using drone technology.” In Proc., Construction Research Congress 2020, 296–305. Reston, VA: ASCE.
Jiang, Y., and Y. Bai. 2020b. “Estimation of construction site elevations using drone-based orthoimagery and deep learning.” J. Constr. Eng. Manage. 146 (8): 04020086. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001869.
Jiang, Y., and Y. Bai. 2021. “Low–high orthoimage pairs-based 3d reconstruction for elevation determination using drone.” J. Constr. Eng. Manage. 147 (9): 04021097. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002067.
Jiang, Y., Y. Bai, and S. Han. 2020. “Determining ground elevations covered by vegetation on construction sites using drone-based orthoimage and convolutional neural network.” J. Comput. Civ. Eng. 34 (6): 04020049. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000930.
Kaehler, A., and G. Bradski. 2016. Learning OpenCV 3: Computer vision in C++ with the OpenCV library. Sebastopol, CA: O’Reilly Media.
Kim, H., and H. Kim. 2018. “3D reconstruction of a concrete mixer truck for training object detectors.” Autom. Constr. 88: 23–30. https://doi.org/10.1016/j.autcon.2017.12.034.
Knyaz, V. A., O. Vygolov, V. V. Kniaz, Y. Vizilter, V. Gorbatsevich, T. Luhmann, and N. Conen. 2017. “Deep learning of convolutional auto-encoder for image matching and 3D object reconstruction in the infrared range.” In Proc., 2017 IEEE Int. Conf. on Computer Vision Workshops (ICCVW), 2155–2164. New York: IEEE.
Kwon, S., J.-W. Park, D. Moon, S. Jung, and H. Park. 2017. “Smart merging method for hybrid point cloud data using UAV and LIDAR in earthwork construction.” Procedia Eng. 196: 21–28. https://doi.org/10.1016/j.proeng.2017.07.168.
Lewis, J. P. 1995. “Fast template matching.” In Proc., Vision Interface 95, 120–123. Quebec City: Canadian Image Processing and Pattern Recognition Society. http://scribblethink.org/Work/nvisionInterface/vi95_lewis.pdf.
Li, D., and M. Lu. 2018. “Integrating geometric models, site images and GIS based on Google earth and keyhole markup language.” Autom. Constr. 89: 317–331. https://doi.org/10.1016/j.autcon.2018.02.002.
Li, G., L. Yu, and S. Fei. 2021. “A deep-learning real-time visual SLAM system based on multi-task feature extraction network and self-supervised feature points.” Measurement 168: 108403. https://doi.org/10.1016/j.measurement.2020.108403.
Liu, Y., Y. Li, L. Dai, C. Yang, L. Wei, T. Lai, and R. Chen. 2021. “Robust feature matching via advanced neighborhood topology consensus.” Neurocomputing 421: 273–284. https://doi.org/10.1016/j.neucom.2020.09.047.
Luo, W., A. G. Schwing, and R. Urtasun. 2016. “Efficient deep learning for stereo matching.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 5695–5703. New York: IEEE.
Memarzadeh, M., M. Golparvar-Fard, and J. C. Niebles. 2013. “Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors.” Autom. Constr. 32: 24–37. https://doi.org/10.1016/j.autcon.2012.12.002.
Moon, D., S. Chung, S. Kwon, J. Seo, and J. Shin. 2019. “Comparison and utilization of point cloud generated from photogrammetry and laser scanning: 3D world model for smart heavy equipment planning.” Autom. Constr. 98: 322–331, https://doi.org/10.1016/j.autcon.2018.07.020.
Mughal, M. H., M. J. Khokhar, and M. Shahzad. 2021. “Assisting UAV localization via deep contextual image matching.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 14: 2445–2457. https://doi.org/10.1109/JSTARS.2021.3054832.
Nair, V., and G. Hinton. 2010. “Rectified linear units improve restricted Boltzmann machines.” In Proc., 27th Int. Conf. on Machine Learning, 807–814. Madison, WI: Omnipress. https://doi.org/doi/10.5555/3104322.3104425.
Nassar, K., and Y.-H. Jung. 2012. “Structure-from-motion approach to the reconstruction of surfaces for earthwork planning.” J. Constr. Eng. Project Manage. 2 (3): 1–7. https://doi.org/10.6106/JCEPM.2012.2.3.001.
Nex, F., and F. Remondino. 2014. “UAV for 3D mapping applications: A review.” In Applied geomatics. New York: Springer.
OpenCV. 2018a. “Feature matching + homography to find objects.” Accessed March 7, 2020. https://docs.opencv.org/3.4.12/d1/de0/tutorial_py_feature_homography.html.
OpenCV. 2018b. “Template matching.” Accessed March 7, 2020. https://docs.opencv.org/3.4.2/d8/dd1/tutorial_js_template_matching.html.
Park, J., P. Kim, Y. K. Cho, and J. Kang. 2019. “Framework for automated registration of UAV and UGV point clouds using local features in images.” Autom. Constr. 98: 175–182. https://doi.org/10.1016/j.autcon.2018.11.024.
Rublee, E., V. Rabaud, K. Konolige, and G. Bradski. 2011. “ORB: An efficient alternative to SIFT or SURF.” In Proc., 2011 Int. Conf. on Computer Vision, 2564–2571. New York: IEEE.
Shang, Z., and Z. Shen. 2018. “Real-time 3D reconstruction on construction site using visual SLAM and UAV.” In Proc., Construction Research Congress 2018, 305–315. Reston, VA: ASCE.
Siebert, S., and J. Teizer. 2014. “Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system.” Autom. Constr. 41: 1–14. https://doi.org/10.1016/j.autcon.2014.01.004.
Snavely, N. 2010. “Bundler: Structure from motion (SfM) for unordered image collections.” Accessed March 7, 2020. http://www.cs.cornell.edu/∼snavely/bundler/#S1.
Sung, C., and P. Y. Kim. 2016. “3D terrain reconstruction of construction sites using a stereo camera.” Autom. Constr. 64: 65–77. https://doi.org/10.1016/j.autcon.2015.12.022.
Takahashi, N., R. Wakutsu, T. Kato, T. Wakaizumi, T. Ooishi, and R. Matsuoka. 2017. “Experiment on UAV photogrammetry and terrestrial laser scanning for ICT-integrated construction.” In Proc., ISPRS—Int. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 371–377. Hannover, Germany: International Society for Photogrammetry and Remote Sensing. https://doi.org/10.5194/isprs-archives-XLII-2-W6-371-2017.
Wang, G., and Y. Chen. 2021. “Robust feature matching using guided local outlier factor.” Pattern Recogn. 117 (Sep): 107986. https://doi.org/10.1016/j.patcog.2021.107986.
Wu, C. 2007. “SiftGPU: A GPU implementation of Scale Invariant Feature Transform (SIFT).” Accessed March 7, 2020. http://cs.unc.edu/∼ccwu/siftgpu.
Wu, C. 2011. “‘VisualSFM: A visual structure from motion system.” Accessed March 22, 2021. http://ccwu.me/vsfm/index.html.
Wu, C., S. Agarwal, B. Curless, and S. M. Seitz. 2011. “Multicore bundle adjustment.” In Proc., CVPR 2011, 3057–3064. New York: IEEE.
Yang, C.-H., M.-H. Tsai, S.-C. Kang, and C.-Y. Hung. 2018. “UAV path planning method for digital terrain model reconstruction—A debris fan example.” Autom. Constr. 93: 214–230, https://doi.org/10.1016/j.autcon.2018.05.024.
Zhu, H., L. Jiao, W. Ma, F. Liu, and W. Zhao. 2019. “A novel neural network for remote sensing image matching.” IEEE Trans. Neural Networks Learn. Syst. 30 (9): 2853–2865. https://doi.org/10.1109/TNNLS.2018.2888757.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 148 • Issue 4 • April 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Jul 4, 2021
Accepted: Dec 16, 2021
Published online: Feb 9, 2022
Published in print: Apr 1, 2022
Discussion open until: Jul 9, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Ilpo Niskanen, Matti Immonen, Lauri Hallman, Martti Mikkonen, Visa Hokkanen, Takeshi Hashimoto, Juha Kostamovaara, Rauno Heikkilä, Using a 2D Profilometer to Determine Volume and Thickness of Stockpiles and Ground Layers of Roads, Journal of Transportation Engineering, Part B: Pavements, 10.1061/JPEODX.PVENG-1149, 149, 1, (2023).
- Sisi Han, Yuhan Jiang, Yilei Huang, Mingzhu Wang, Yong Bai, Andrea Spool-White, Scan2Drawing: Use of Deep Learning for As-Built Model Landscape Architecture, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13077, 149, 5, (2023).
- Samira Badrloo, Masood Varshosaz, Saied Pirasteh, Jonathan Li, Image-Based Obstacle Detection Methods for the Safe Navigation of Unmanned Vehicles: A Review, Remote Sensing, 10.3390/rs14153824, 14, 15, (3824), (2022).
- Diogo Gonçalves, Gil Gonçalves, Juan Antonio Pérez-Alvávez, Umberto Andriolo, On the 3D Reconstruction of Coastal Structures by Unmanned Aerial Systems with Onboard Global Navigation Satellite System and Real-Time Kinematics and Terrestrial Laser Scanning, Remote Sensing, 10.3390/rs14061485, 14, 6, (1485), (2022).
- Yuhan Jiang, Yilei Huang, Jingkuang Liu, Dapeng Li, Shuiyuan Li, Weijing Nie, In-Hun Chung, Automatic Volume Calculation and Mapping of Construction and Demolition Debris Using Drones, Deep Learning, and GIS, Drones, 10.3390/drones6100279, 6, 10, (279), (2022).
- Yuhan Jiang, Sisi Han, Yong Bai, Scan4Façade: Automated As-Is Façade Modeling of Historic High-Rise Buildings Using Drones and AI, Journal of Architectural Engineering, 10.1061/(ASCE)AE.1943-5568.0000564, 28, 4, (2022).