Three-Dimensional Field-Scale Coupled Thermo-Hydro-Mechanical Modeling: Parallel Computing Implementation
Publication: International Journal of Geomechanics
Volume 11, Issue 2
Abstract
An approach for the simulation of three-dimensional field-scale coupled thermo-hydro-mechanical problems is presented, including the implementation of parallel computation algorithms. The approach is designed to allow three-dimensional large-scale coupled simulations to be undertaken in reduced time. Owing to progress in computer technology, existing parallel implementations have been found to be ineffective, with the time taken for communication dominating any reduction in time gained by splitting computation across processors. After analysis of the behavior of the solver and the architecture of multicore, nodal, parallel computers, modification of the parallel algorithm using a novel hybrid message passing interface/open multiprocessing (MPI/OpenMP) method was implemented and found to yield significant improvements by reducing the amount of communication required. This finding reflects recent enhancements of current high-performance computing architectures. An increase in performance of 500% over existing parallel implementations on current processors was achieved for the solver. An example problem involving the Prototype Repository experiment undertaken by the Swedish Nuclear Fuel and Waste Management Co. [Svensk Kärnbränslehantering AB (SKB)] in Äspö, Sweden, has been presented to demonstrate situations in which parallel computation is invaluable because of the complex, highly coupled nature of the problem.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
Support for the first writer from an Engineering and Physical Sciences Research Council (EPSRC) studentship and funding by the European Commission (EC) via the Prototype Repository Project (ECFIKW-2000-00055), along with access to SKB’s high quality data set related to the Prototype Repository project are gratefully acknowledged. Support and use of the computing facilities both at the High-Performance Computing Collaboratory () at Mississippi State University and at Advanced Research Computing @ Cardiff (ARCCA) at Cardiff University are also appreciatively acknowledged. The writers would also like to thank the National Science Foundation for its partial support of this work through the grants NSFNSF EPS #0556308 and NSF IIP #1034897.
References
Asanovic, A., et al. (2006). The landscape of parallel computing research: A view from Berkeley, Univ. of California, Berkeley, CA.
Barrett, R., et al. (1995). Templates for the solution of linear systems: Building blocks for iterative methods, Wiley, New York.
Cleall, P. J., Melhuish, T. A., and Thomas, H. R. (2006a). “Modeling the three-dimensional behavior of a prototype nuclear waste repository.” Eng. Geol., 85(1–2), 212–220.
Cleall, P. J., Thomas, H. R., Melhuish, T. A., and Owen, D. H. (2006b). “Use of parallel computing and visualization techniques in the simulation of large scale geoenvironmental engineering problems.” Future Gener. Comput. Syst., 22(4), 460–467.
Dahlström, L-O. (1998). “Äspö HRL—Test plan for the prototype repository.” Progress Rep. No. HRL-98-24, Swedish Nuclear Fuel and Waste Management (SKB), Stockholm, Sweden.
Duff, I. S., and van der Vorst, H. A. (1999). “Developments and trends in the parallel solution of linear systems.” Technical Rep. TR/PA/99/10, Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique (CERFACS) (European Centre for Research and Advanced Training in Scientific Computation), Toulouse, France.
Flynn, M. (1972). “Some computer organizations and their effectiveness.” IEEE Trans. Comput., C-21(9), 948–960.
Goudarzi, R., and Johannesson, L.-E. (2006). “Äspö Hard Rock Laboratory, Prototype Repository—Sensor data report (Period 010917-061201).” 16, IPR-07-05, Swedish Nuclear Fuel and Waste Management (SKB), Stockholm, Sweden.
Hennessy, J., and Patterson, D. (2007). Computer architecture: A quantitative approach, 4th Ed., Morgan Kauffman, San Francisco.
Hockney, R. (1994). “The communication challenge for MPP: Intel Paragon and Meiko CS-2.” Parallel Comput., 20(3), 389–398.
Intel. (2006). “White paper—Intel architecture and silicon cadence. The catalyst for industry innovation.”
Jiayin, M, Bo, S., Yongwei, W., and Guangwen, Y. (2006). “Overlapping communication and computation in MPI by multithreading.” Proc., 2006 Int. Conf. on Parallel & Distributed Processing Techniques and Applications (PDPTA’06), CSREA Press, Bogart, GA, 52–57.
Johannesson, L.-E., Börgesson, L., Goudarzi, R., Sandén, T., Gunnarsson, D., and Svemar, C. (2007). “Prototype Repository: A full scale experiment at Äspö HRL.” Phys. Chem. Earth, 32(1–7), 58–76.
Neretnieks, I. (1993). “Solute transport in fractured rock—applications to radionuclide waste repositories.” Flow and contaminant transport in fractured rock, J. Bear, C.-F. Tsang, and G. de Marsily, eds., Academic Press, San Diego, 39–127.
Owen, D. H. (2000). “Preconditioned parallel iterative solution methods for coupled finite element analyses.” Ph.D. thesis, Univ. of Wales, Cardiff, UK.
Roosta, S. H. (2000). “Parallel processing and parallel algorithms.” Theory and computation, Springer-Verlag, New York.
Smith, I. M., and Griffiths, D. V. (2004). Programming the finite element method, 4th Ed., Wiley, Chichester, UK.
Svemar, C., and Pusch, R. (2000). “Äspö Hard Rock Laboratory.” Prototype Repository, project description, FIKW-CT-2000-00055, IPR-00-30, Swedish Nuclear Fuel and Waste Management (SKB), Stockholm, Sweden.
Thomas, H. R., Cleall, P. J., Chandler, N., Dixon, D., and Mitchell, H. P. (2003a). “Water infiltration into a large scale in situ experiment in an underground research laboratory—Physical measurements and numerical simulation.” Géotechnique, 53(2), 207–224.
Thomas, H. R., Cleall, P. J., Dixon, D., and Mitchell, H. P. (2009). “The coupled thermal-hydraulic-mechanical behavior of a large scale in situ heating experiment.” Géotechnique, 59(4), 401–413.
Thomas, H. R., and He, Y. (1997). “A coupled heat-moisture transfer theory for deformable unsaturated soil and its algorithmic implementation.” Int. J. Numer. Methods Eng., 40(18), 3421–3441.
Thomas, H. R., Yang, H. T., He, Y., and Cleall, P. J. (2003b). “A multilevel parallelized substructuring frontal solution for coupled thermo/hydro/mechanical problems in unsaturated soil.” Int. J. Numer. Anal. Meth. Geomech., 27(11), 951–965.
van de Steen, A. J., and Dongarra, J. J. (1996). “Overview of recent supercomputers.” Technical report, Dept. of Computer Science, 6th Ed., Univ. of Tennessee, Knoxville, TN.
White, J. B., and Bova, S. W. (1999). “Where’s the overlap? An analysis of popular MPI implementations.” Proc., Third MPI Developer’s and User’s Conf., MPIDC ’99, MPI Software Technology Press, Starkville, MS.
Information & Authors
Information
Published In
International Journal of Geomechanics
Volume 11 • Issue 2 • April 2011
Pages: 90 - 98
Copyright
© 2011 American Society of Civil Engineers.
History
Received: Jul 3, 2009
Accepted: Feb 24, 2010
Published online: Sep 18, 2010
Published in print: Apr 1, 2011
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.