Linear solver test problem

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • dsrawlins
    Developer
    • Dec 2008
    • 366

    Linear solver test problem

    This is a simple (but large) problem to compare the linear solvers Skyline, SuperLU and PSLDLT on an SGI Altix machine. It consists of a box with a linear-elastic material, constrained on the left end and with an applied pressure on the right end. All tests were limited to one cpu.

    Results:

    Skyline 6 min 47 sec
    SuperLU 3 min 29 sec
    PSLDLT 0 min 8 sec

    Because of the obvious dominance of PSLDLT over the other two solvers, we have decided not to implement the multi-threaded version of SuperLU at this time, but to implement another solver, Pardiso, which should be more comparable to the speed of PSLDLT.
    Department of Bioengineering, University of Utah
    Scientific Computing and Imaging institute, University of Utah
  • dsrawlins
    Developer
    • Dec 2008
    • 366

    #2
    Pardiso and WSMP

    The Pardiso linear solver included in the Intel MKL (math kernel library) is now implemented in FEBio for all platforms (Linux, Windows, Mac and Altix), and the WSMP linear solver is implemented in Linux and Altix. Here are the timings for this problem (box15.feb) on Altix:

    Pardiso 11 seconds
    WSMP 11 seconds

    For a larger problem (box40.feb, 806880 equations), I get the following timings:

    Solver Time (1cpu) Time (4 cpus)

    Pardiso 46:52 12:52
    PSLDLT 44:11 17:44
    WSMP 25:15 7:33

    Dave
    Department of Bioengineering, University of Utah
    Scientific Computing and Imaging institute, University of Utah

    Comment

    • Sam
      Junior Member
      • Jun 2008
      • 5

      #3
      Pardiso solver

      Dave,
      I'm trying to implement a parallel version of FEBio on our supercomputer.
      How did you go about implementing the pardiso solver?
      Could I use your source code?
      Thanks for your help,
      Sam

      Comment

      • dsrawlins
        Developer
        • Dec 2008
        • 366

        #4
        Pardiso Solver

        Sam,

        We have implemented the Pardiso solver by using the shared library available at http://www.pardiso-project.org/ and by using the version available in the Intel Math Kernel Library (MKL) available at http://software.intel.com/en-us/intel-mkl/. Let me know if you have any questions about getting it running.

        Dave
        Department of Bioengineering, University of Utah
        Scientific Computing and Imaging institute, University of Utah

        Comment

        • Sam
          Junior Member
          • Jun 2008
          • 5

          #5
          Pardiso solver

          Dave,
          Thanks.
          I'm working on compiling it at the moment and making some progress, although I am having the usual problems with library files etc. A couple of questions:-
          Do I need to tell FEBio how many processors to use, or do I just run it with mpirun?
          What is the format of the config file that the latest source code looks for? At the moment I am using -noconfig every time but it would be good to avoid that.
          Thanks for your help,
          Sam

          Comment

          • dsrawlins
            Developer
            • Dec 2008
            • 366

            #6
            Pardiso solver

            Sam,

            You can set the number of processors with the environment variable OMP_NUM_THREADS. I've attached a zip file with several different files that may be useful for installing and running febio. febio.xml is an example config file for specifying pardiso as the linear solver. Put it in the same directory as your executable or use the flag -cnf febio.xml.

            The includes.bash is a shell script that creates the files objects.mk and the dependencies.mk that are used in the Makefile. It needs to be run from the root febio directory. make.inc contains the information I used to link with the pardiso shared library. It also needs to be put in the root febio directory. Copy Makefile.main to Makefile in the root febio directory, Makefile.fecore to Makefile in the FEBio directory and Makefile.fecore to Makefile in the FECore directory.

            Hope this helps. Let me know if you have any other questions.

            Dave
            Department of Bioengineering, University of Utah
            Scientific Computing and Imaging institute, University of Utah

            Comment

            • Sam
              Junior Member
              • Jun 2008
              • 5

              #7
              Dave,
              Thanks! Very helpful.
              If I understand your email correctly, you have three directories:-
              ./febio/FEBio/FECore
              Is that right? Then the makefile in the top directory calls the other two and the include file?
              Cheers,
              Sam

              Originally posted by dsrawlins View Post
              Sam,

              You can set the number of processors with the environment variable OMP_NUM_THREADS. I've attached a zip file with several different files that may be useful for installing and running febio. febio.xml is an example config file for specifying pardiso as the linear solver. Put it in the same directory as your executable or use the flag -cnf febio.xml.

              The includes.bash is a shell script that creates the files objects.mk and the dependencies.mk that are used in the Makefile. It needs to be run from the root febio directory. make.inc contains the information I used to link with the pardiso shared library. It also needs to be put in the root febio directory. Copy Makefile.main to Makefile in the root febio directory, Makefile.fecore to Makefile in the FEBio directory and Makefile.fecore to Makefile in the FECore directory.

              Hope this helps. Let me know if you have any other questions.

              Dave

              Comment

              • dsrawlins
                Developer
                • Dec 2008
                • 366

                #8
                Febio file structure

                Sam,

                There should be two subdirectories under the root ./febio: ./febio/FEBio and ./febio/FECore. I was assuming that this was the directory structure you got from Steve. If not, let me know.

                Cheers,

                Dave
                Department of Bioengineering, University of Utah
                Scientific Computing and Imaging institute, University of Utah

                Comment

                • Sam
                  Junior Member
                  • Jun 2008
                  • 5

                  #9
                  No, I don't have the /febio/FEBio subdirectory, only /febio/FECore and /febio/lib.
                  If you let me know what goes in which directory I can move it around to match your structure.
                  Cheers,
                  Sam

                  Originally posted by dsrawlins View Post
                  Sam,

                  There should be two subdirectories under the root ./febio: ./febio/FEBio and ./febio/FECore. I was assuming that this was the directory structure you got from Steve. If not, let me know.

                  Cheers,

                  Dave

                  Comment

                  • dsrawlins
                    Developer
                    • Dec 2008
                    • 366

                    #10
                    Febio file structure

                    Sam,

                    If you move all the source code you have in your root febio directory into a subdirectory called FEBio, I think we will have the same directory structure. I don't have any source code in my root febio directory. Here is the listing of my FECore directory to make sure we have the same files in there:

                    dependencies.mk Makefile_steve.FECore PardisoSolver.cpp SuperLU_MT_Solver.o
                    FECore.cpp matrix.cpp PardisoSolver.h SuperLUSolver.cpp
                    FECore.h MatrixFactory.cpp PardisoSolver.o SuperLUSolver.o
                    FECore.o MatrixFactory.h ReadMe.txt targetver.h
                    FECore.vcproj MatrixFactory.o SparseMatrix.cpp vector.cpp
                    LinearSolver.cpp matrix.h SparseMatrix.h vector.h
                    LinearSolver.h matrix.o SparseMatrix.o vector.o
                    LinearSolver.o MatrixProfile.cpp stdafx.cpp WSMPSolver.cpp
                    lusolver.cpp MatrixProfile.h stdafx.h WSMPSolver.h
                    lusolver.o MatrixProfile.o stdafx.o WSMPSolver.o
                    Makefile objects.mk SuperLU_MT_Solver.cpp

                    Dave
                    Department of Bioengineering, University of Utah
                    Scientific Computing and Imaging institute, University of Utah

                    Comment

                    • dsrawlins
                      Developer
                      • Dec 2008
                      • 366

                      #11
                      Compiling febio

                      Sam,

                      I should note that I had problems compiling the source files in the FEBio subdirectory independently and then linking them on our ALTIX machines. I've attached the Makefile for the FEBio subdirectory that I use on our ALTIX machines.

                      Dave
                      Department of Bioengineering, University of Utah
                      Scientific Computing and Imaging institute, University of Utah

                      Comment

                      • Sam
                        Junior Member
                        • Jun 2008
                        • 5

                        #12
                        Dave,
                        I've installed your makefiles and changed the compiler and paths etc.
                        I haven't tried this makefile yet, but with the others I get an error:-
                        *** No rule to make target `assert.h', needed by `queue.o'. Stop.
                        Any suggestions?
                        Sam

                        Originally posted by dsrawlins View Post
                        Sam,

                        I should note that I had problems compiling the source files in the FEBio subdirectory independently and then linking them on our ALTIX machines. I've attached the Makefile for the FEBio subdirectory that I use on our ALTIX machines.

                        Dave

                        Comment

                        • dsrawlins
                          Developer
                          • Dec 2008
                          • 366

                          #13
                          dependency files

                          Sam,

                          It looks like my file includes.bash is working differently on your machine than it is on mine. The reference to assert.h should not have been included in the dependencies.mk file. The advantage of using the dependencies.mk file is that you don't have to recompile all the files if you are just making a change in one file, but since you need to compile everything, I would recommend using the second Makefile I sent to you.

                          Dave
                          Department of Bioengineering, University of Utah
                          Scientific Computing and Imaging institute, University of Utah

                          Comment

                          • siboles
                            Member
                            • Mar 2010
                            • 42

                            #14
                            Hi Dave,

                            It seems for this specific problem WSMP outperforms Pardiso. Have you compared the two for other models on single and multiple threads? Specifically, which one is better if running on a single thread or is that an impossible question to answer due to variance between problems?

                            Also, this may be answered elsewhere, but do you include the non-symmetric solvers for WSMP in the compile as well?

                            Thanks,

                            Scott

                            Originally posted by dsrawlins View Post
                            The Pardiso linear solver included in the Intel MKL (math kernel library) is now implemented in FEBio for all platforms (Linux, Windows, Mac and Altix), and the WSMP linear solver is implemented in Linux and Altix. Here are the timings for this problem (box15.feb) on Altix:

                            Pardiso 11 seconds
                            WSMP 11 seconds

                            For a larger problem (box40.feb, 806880 equations), I get the following timings:

                            Solver Time (1cpu) Time (4 cpus)

                            Pardiso 46:52 12:52
                            PSLDLT 44:11 17:44
                            WSMP 25:15 7:33

                            Dave

                            Comment

                            Working...
                            X
                            😀
                            😂
                            🥰
                            😘
                            🤢
                            😎
                            😞
                            😡
                            👍
                            👎