A restriction of mesh elements number in Febio

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • KaravaevAS
    Junior Member
    • Jan 2016
    • 21

    A restriction of mesh elements number in Febio

    Hello dear Febio users?

    I have a FEM model which consists over 2000000 hexahedral elements. Is it possible to compute dynamic simulation with such big computational mesh? because now i gor an error "ERROR during factorization: Not enough memory"

    with best regards Alexander
  • maas
    Lead Code Developer
    • Nov 2007
    • 3400

    #2
    Hi Alexander,

    FEBio is only limited by the available memory on your machine. I think you might be braking a record here in terms of number of elements. I suspect you'll need a lot of memory to solve this problem. Perhaps you can try another computer that has more memory?

    Cheers,

    Steve
    Department of Bioengineering, University of Utah
    Scientific Computing and Imaging institute, University of Utah

    Comment

    • KaravaevAS
      Junior Member
      • Jan 2016
      • 21

      #3
      Thank you for reply Steve

      ok i will try.

      so is it possible to use conjugate gradient method to solve linear system instead of pardiso?

      with best regards Alexander

      Comment

      • maas
        Lead Code Developer
        • Nov 2007
        • 3400

        #4
        Hi Alexander,

        FEBio has a conjugate gradient solver, although I'm not sure how good it is (I've never used it myself to be honest). You can select it in your configuration file (febio.xml). Just replace the pardiso default solver with

        <linear_solver type="rcicg"/>

        Please give it a try and let me know if that worked.

        In the upcoming release of FEBio 2.5 we will have added several other iterative solvers that we hope will be useful for solving really big models.

        Cheers,

        Steve
        Department of Bioengineering, University of Utah
        Scientific Computing and Imaging institute, University of Utah

        Comment

        • KaravaevAS
          Junior Member
          • Jan 2016
          • 21

          #5
          Hi Steve.

          So i decided to test RAM consumption on 24Gb server with simple uniaxial tension test of cube along OZ axis. Bottom face was fixed and the nodes of upper face were moved .
          These is the results for different number of hexahedral cells in the cube:

          1000000 cells – Febio crashed down with now messages

          512000 cells - Febio crashed down but write stiffness matrix parameters

          Nr of equations ........................... : 1574640
          Nr of nonzeroes in stiffness matrix ....... : 62992071

          The maximal occupied RAM before termination was around 30%

          238328 cells – Febio works, stiffness matrix parameters

          Nr of equations ........................... : 738234
          Nr of nonzeroes in stiffness matrix ....... : 29323449

          The maximal occupied RAM during 1-2 step was around 20%

          Also I tried “rcicg” option with 512000 cells but it didn’t work either.

          So any ideas?

          with best regards, Alexander

          Comment

          • maas
            Lead Code Developer
            • Nov 2007
            • 3400

            #6
            Hi Alexander,

            We'll try to do some experiments on our end to determine FEBio's behavior for really large problems and report back. I still suspect it's simply a memory allocation failure, but I am surprised to see FEBio behave differently for the 1M and 512K cases.

            Cheers,

            Steve
            Department of Bioengineering, University of Utah
            Scientific Computing and Imaging institute, University of Utah

            Comment

            • KaravaevAS
              Junior Member
              • Jan 2016
              • 21

              #7
              Hi Steve.

              Thank's for reply. I'll try to test Febio in Debug mode also.

              with best regards, Alexander

              Comment

              • avoorhees
                Junior Member
                • Jul 2015
                • 5

                #8
                Was this issue ever resolved?

                I have been running some very large models with FEBio myself, and I seem to keep running into an upper limit on the number of elements/dof's I can run. Similar to Alexander's problem, Febio crashes without error messages when I use a mesh with 2,000,000 linear tets or more, and I do not seem to be utilizing all of my memory (15-25 GB of 64 available on my system). The log file just stops writing after the LINEAR SOLVER DATA section. I'm running windows 7 64 bit if that makes a difference. I've also tried these large problems with both FEBIO 2.3 and 2.5 and I get the same result.

                I have successfully run models with ~167,000 linear tets and 340,000 nodes, but this seems to be pretty close to the maximum I can run. Moving forward, some of my projects would definitely benefit from the ability to solve larger models, so I'm hoping that there was a resolution. Thanks.

                --Andrew

                Comment

                • maas
                  Lead Code Developer
                  • Nov 2007
                  • 3400

                  #9
                  Hi Andrew,

                  I suspect that memory allocation is the issue. Keep in mind that during the initialization phase, FEBio only needs enough memory to store the global stiffness matrix, which usually will only be a small fraction of your available memory, even for relatively large problems. However, the factorization of the stiffness matrix requires a lot more memory, easily an order of magnitude larger. So, if you are already using 15-25 GB before hitting factorization, it is likely that the memory allocation for factorizing the matrix fails.

                  For such large problems, you may want to consider using FEBio's iterative solvers which do not require a full factorization of the stiffness matrix. You can set the linear solver in the configuration file as described above. As of FEBio 2.5 the following iterative solvers are available:

                  rcicg
                  fgmres
                  fgmres_ilut
                  fgmres_ilu0

                  The first two require the least amount of memory. The last two require a bit more memory since they use a preconditioner, however the memory requirement will still be much lower than the default direct solver (i.e. pardiso), and the convergence will probably be much better than the first two (although this is somewhat problem dependent). Please give this a try and let us know if this resolves the issue or not.

                  Cheers,

                  Steve
                  Department of Bioengineering, University of Utah
                  Scientific Computing and Imaging institute, University of Utah

                  Comment

                  • avoorhees
                    Junior Member
                    • Jul 2015
                    • 5

                    #10
                    Hi Steve,
                    Thanks for the reply. I tried changing the solver in the configuration file, but for everyone of these options I get the following error: "Fatal Error: The selected linear solver does not support the requested matrix format. Please select a different linear solver." I get this same error when I try to run a smaller model that I know works with the pardiso solver too. Are there limitations on the types of problems these solvers can run? I'm running a static analysis with a solid mixture materials consisting of a Mooney-Rivlin solid and fibers. My boundary conditions are all displacement driven. Any insight would be welcome. Thanks.

                    --Andrew

                    Comment

                    • maas
                      Lead Code Developer
                      • Nov 2007
                      • 3400

                      #11
                      Hi Andrew,

                      Some of these iterative solvers require a symmetric matrix, but the problem you are describing should have a symmetric matrix, so I'm not sure why you would get that error. I'll take a look at these solvers (I haven't used them in a while), but you can double-check your Control section and make sure that you don't have the symmetric_stiffness flag defined (or if it's defined its value should be 1). I'll let you know what I find.

                      Cheers,

                      Steve
                      Department of Bioengineering, University of Utah
                      Scientific Computing and Imaging institute, University of Utah

                      Comment

                      • avoorhees
                        Junior Member
                        • Jul 2015
                        • 5

                        #12
                        I did not have the symmetric_stiffness flag defined, and as expected, adding it did not resolve the problem. If you want any of my input files, let me know. Thanks for the help.

                        --Andrew

                        Comment

                        • dsrawlins
                          Developer
                          • Dec 2008
                          • 366

                          #13
                          Hi Andrew,

                          I need to add a flag to compile the iterative solvers into the executable. Which platform are you using?

                          Thanks,

                          Dave
                          Department of Bioengineering, University of Utah
                          Scientific Computing and Imaging institute, University of Utah

                          Comment

                          • avoorhees
                            Junior Member
                            • Jul 2015
                            • 5

                            #14
                            Hi Dave,
                            I'm running Windows 7 64 bit. Thanks.

                            Comment

                            • dsrawlins
                              Developer
                              • Dec 2008
                              • 366

                              #15
                              Andrew,

                              It looks like there is a bug in our current version of the 64bit Intel Math Kernel Library that affects the iterative solvers, so I'm going to do an upgrade. In the mean time the Win32 version of FEBio2 should work with the iterative solvers.

                              Best,

                              Dave
                              Department of Bioengineering, University of Utah
                              Scientific Computing and Imaging institute, University of Utah

                              Comment

                              Working...
                              X
                              😀
                              😂
                              🥰
                              😘
                              🤢
                              😎
                              😞
                              😡
                              👍
                              👎