OF23_IOPerformance_Analysis - wyldckat/wyldckat.github.io GitHub Wiki

Reason for this wiki page: http://www.cfd-online.com/Forums/openfoam/136983-binary-gives-significant-performance-advantage-mesh-solve.html

Machine specs:

  • AMD 1055T 6 core
  • DDR2 800MHz - 6GB of RAM, 2+2 paired in DDR2 and 1 in DDR2 single
  • 2TB hard-drive

Table of Contents

First approach

Using the tutorial "incompressible/simpleFoam/motorBike" from OF 2.3.x, changed to ascii and using 12 precision. Commented out the "functions" entry and changed write interval to 1000, out of 500. This means that no writing is done. This case runs in parallel.

Final iteration in "log.simpleFoam":

  • ascii
    smoothSolver:  Solving for Ux, Initial residual = 0.000466251003803, Final residual = 2.5796037839e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00951202893766, Final residual = 0.000533443325206, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00957570141229, Final residual = 0.000543443859518, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0282855308665, Final residual = 0.00013751222697, No Iterations 4
    time step continuity errors : sum local = 2.18333802209e-05, global = 4.66760310592e-07, cumulative = -0.000588122806168
    smoothSolver:  Solving for omega, Initial residual = 0.000119597612019, Final residual = 8.96118905641e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00213864598918, Final residual = 0.000189348333041, No Iterations 3
    ExecutionTime = 1020.96 s  ClockTime = 1043 s
  • binary
    smoothSolver:  Solving for Ux, Initial residual = 0.000320281, Final residual = 1.75648e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00603395, Final residual = 0.000336888, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00664659, Final residual = 0.000372021, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0228133, Final residual = 9.24127e-05, No Iterations 4
    time step continuity errors : sum local = 1.4661e-05, global = 1.70817e-07, cumulative = 0.000631401
    smoothSolver:  Solving for omega, Initial residual = 8.83011e-05, Final residual = 6.38391e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00129088, Final residual = 0.000111512, No Iterations 3
    ExecutionTime = 1025.57 s  ClockTime = 1055 s

Conclusion: Inconclusive.

Second approach

Since no writing didn't seem to affect the timing, then tried using the same binary mesh, using write interval of 10 for a 200 iteration run. 

    rm -rf processor*/[1-9]* && rm log.simpleFoam
    ./Allrun && tail -n 20 log.simpleFoam
  • ascii precision 12
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 424.63 s  ClockTime = 437 s
  • binary precision 6
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412, Final residual = 1.64279e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348, Final residual = 0.000290571, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574, Final residual = 0.000322775, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214552, Final residual = 8.91833e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901e-05, global = 4.52797e-08, cumulative = 0.00064491
    smoothSolver:  Solving for omega, Initial residual = 8.32191e-05, Final residual = 6.09269e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525, Final residual = 0.000116052, No Iterations 3
    ExecutionTime = 455.32 s  ClockTime = 463 s
  • ascii precision 6
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412, Final residual = 1.64279e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348, Final residual = 0.000290571, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574, Final residual = 0.000322775, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214552, Final residual = 8.91833e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901e-05, global = 4.52797e-08, cumulative = 0.00064491
    smoothSolver:  Solving for omega, Initial residual = 8.32191e-05, Final residual = 6.09269e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525, Final residual = 0.000116052, No Iterations 3
    ExecutionTime = 482.83 s  ClockTime = 502 s
  • ascii precision 8
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.00029641233, Final residual = 1.6427868e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.0053034827, Final residual = 0.00029057133, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574007, Final residual = 0.00032277536, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.021455166, Final residual = 8.9183286e-05, No Iterations 4
    time step continuity errors : sum local = 1.4090141e-05, global = 4.5279685e-08, cumulative = 0.00064490983
    smoothSolver:  Solving for omega, Initial residual = 8.321907e-05, Final residual = 6.0926876e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.0012852508, Final residual = 0.0001160519, No Iterations 3
    ExecutionTime = 487.92 s  ClockTime = 502 s
  • ascii precision 10
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.0002964123314, Final residual = 1.642786825e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.005303482689, Final residual = 0.0002905713327, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.005857400712, Final residual = 0.0003227753638, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.02145516631, Final residual = 8.918328611e-05, No Iterations 4
    time step continuity errors : sum local = 1.409014051e-05, global = 4.527968493e-08, cumulative = 0.0006449098309
    smoothSolver:  Solving for omega, Initial residual = 8.321906977e-05, Final residual = 6.092687649e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.001285250835, Final residual = 0.0001160519044, No Iterations 3
    ExecutionTime = 488.25 s  ClockTime = 500 s
  • ascii precision 12
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 492.03 s  ClockTime = 503 s
  • binary precision 12
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 490.31 s  ClockTime = 510 s
  • binary precision 6 
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412, Final residual = 1.64279e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348, Final residual = 0.000290571, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574, Final residual = 0.000322775, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214552, Final residual = 8.91833e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901e-05, global = 4.52797e-08, cumulative = 0.00064491
    smoothSolver:  Solving for omega, Initial residual = 8.32191e-05, Final residual = 6.09269e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525, Final residual = 0.000116052, No Iterations 3
    ExecutionTime = 472.03 s  ClockTime = 485 s
  • ascii precision 12
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 487.57 s  ClockTime = 502 s
  • ascii precision 16
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.0002964123313928318, Final residual = 1.642786825448099e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.005303482688829303, Final residual = 0.0002905713327031695, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.005857400711755268, Final residual = 0.0003227753638058325, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.02145516631122302, Final residual = 8.918328610648241e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073281e-05, global = 4.527968492618009e-08, cumulative = 0.000644909830931284
    smoothSolver:  Solving for omega, Initial residual = 8.321906976752431e-05, Final residual = 6.092687648942599e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.001285250834973667, Final residual = 0.0001160519044286107, No Iterations 3
    ExecutionTime = 498.47 s  ClockTime = 503 s

Note: ASCII precision 12 needs roughly 1.2 GB of disk space for the "processor*" folders, while binary needs around 650MB: 

    du -sh processor*

Conclusion: Seems to be a disk/IO cache issue. Output to the log file might also be affecting the run time, even if it's only 190kB. It's also possible that a plugin in my Firefox might have stollen some CPU/disk resources during these runs...

Third approach

Using the same strategy as in the second approach, but this time the write interval is set to 400, so that it never writes the solution fields to disk. 

    rm -rf processor*/[1-9]* && rm log.simpleFoam
    time ./Allrun && tail -n 20 log.simpleFoam
  • ascii precision 12
    real  8m33.360s
    user  50m1.440s
    sys 1m0.920s

    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 510.63 s  ClockTime = 513 s
  • binary precision 12
    real  8m36.866s
    user  50m13.072s
    sys 1m4.348s

    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 512.2 s  ClockTime = 515 s
  • binary precision 12 - with "solverPerformance 0" in "$WM_PROJECT_DIR/etc/controlDict", i.e. minimal output to log file
    real  8m40.344s
    user  50m31.585s
    sys 1m6.040s

    Time = 200

    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    ExecutionTime = 516.92 s  ClockTime = 519 s

Conclusion: Omiting solver output does not affect performance. Using "time" to track the time it takes to run a script, that seemed to eat up a bit.

Fourth approach

Using the same strategy as in the second approach, but this time we have symbolic links that preemptively point to "/dev/null" for each time snapshot. 

    rm -rf processor*/[1-9]* && rm log.simpleFoam
    for a in processor*; do cd $a; cp -r 0 10; cd 10; rm -rf include; for b in *; do rm $b; ln -s /dev/null $b; done; cd ..; for b in $(seq 20 10 200); do cp -r 10 $b; done;  cd ..; done
    time ./Allrun && tail -n 20 log.simpleFoam
  • binary precision 12
    real  8m40.371s
    user  50m32.718s
    sys 1m3.712s

    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 516.79 s  ClockTime = 519 s
  • binary precision 12 - without the symbolic links
    real  8m42.443s
    user  50m36.698s
    sys 1m6.056s

    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 516.68 s  ClockTime = 521 s
  • binary precision 12 - without the symbolic links and the use "time"
    Time = 200

    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 516.66 s  ClockTime = 520 s







  • binary precision 12 - without the symbolic links and the use "time" and after a reboot


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 599.66 s  ClockTime = 613 s








  • binary precision 12 - without the symbolic links and the use "time" and launched a bit after the previous one ended


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 436.35 s  ClockTime = 441 s







  • binary precision 12 - 3rd consecutive run


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 449.73 s  ClockTime = 452 s    







  • ascii precision 12 - 1st run


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 462.81 s  ClockTime = 466 s







  • ascii precision 12 - 2nd run


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 471.87 s  ClockTime = 474 s







  • binary precision 12 - consecutive run after the previous ascii one


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412331393, Final residual = 1.64278682545e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348268883, Final residual = 0.000290571332703, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00585740071176, Final residual = 0.000322775363806, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214551663112, Final residual = 8.91832861065e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901405073e-05, global = 4.52796849262e-08, cumulative = 0.000644909830931
    smoothSolver:  Solving for omega, Initial residual = 8.32190697675e-05, Final residual = 6.09268764894e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525083497, Final residual = 0.000116051904429, No Iterations 3
    ExecutionTime = 456.22 s  ClockTime = 459 s







  • ascii precision 6 - consecutive run after the previous binary one


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412, Final residual = 1.64279e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348, Final residual = 0.000290571, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574, Final residual = 0.000322775, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214552, Final residual = 8.91833e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901e-05, global = 4.52797e-08, cumulative = 0.00064491
    smoothSolver:  Solving for omega, Initial residual = 8.32191e-05, Final residual = 6.09269e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525, Final residual = 0.000116052, No Iterations 3
    ExecutionTime = 472.36 s  ClockTime = 475 s







  • ascii precision 6 - consecutive run after the previous ascii one, but with write interval at 1000, i.e. no writes


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412, Final residual = 1.64279e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348, Final residual = 0.000290571, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574, Final residual = 0.000322775, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214552, Final residual = 8.91833e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901e-05, global = 4.52797e-08, cumulative = 0.00064491
    smoothSolver:  Solving for omega, Initial residual = 8.32191e-05, Final residual = 6.09269e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525, Final residual = 0.000116052, No Iterations 3
    ExecutionTime = 466.91 s  ClockTime = 469 s







  • binary precision 6 - consecutive run after the previous ascii one, but with write interval at 1000, i.e. no writes


    Time = 200






    smoothSolver:  Solving for Ux, Initial residual = 0.000296412, Final residual = 1.64279e-05, No Iterations 4
    smoothSolver:  Solving for Uy, Initial residual = 0.00530348, Final residual = 0.000290571, No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.0058574, Final residual = 0.000322775, No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.0214552, Final residual = 8.91833e-05, No Iterations 4
    time step continuity errors : sum local = 1.40901e-05, global = 4.52797e-08, cumulative = 0.00064491
    smoothSolver:  Solving for omega, Initial residual = 8.32191e-05, Final residual = 6.09269e-06, No Iterations 3
    smoothSolver:  Solving for k, Initial residual = 0.00128525, Final residual = 0.000116052, No Iterations 3
    ExecutionTime = 466.62 s  ClockTime = 468 s






Conclusion:




    

  • The excess time that the ascii version takes does not seem directly related to the file size, at least not while it has disk cache to spare.
    • 

  • If it doesn't write, then there is no slow down due to using ascii.
    • 


Something seems to clog up the system's performance after each run... it's possible that the disk cache mechanism gets bogged down after a few thousand files come and go from the disk in a short period of time. The other possibility is that the problem is when the solver processes get pushed to the DDR2 single RAM module, which halves the performance on those accesses, which would explain it going from 450~75 to 520.


Continued investigations with another machine and detailed here: OF23_IOPerformance_Analysis_2

              


           
⚠️ **GitHub.com Fallback** ⚠️