Running AWSEM simulations in parallel - adavtyan/awsemmd GitHub Wiki

AWSEM-MD is implemented in LAMMPS in a parallel manner using MPI (Message Passing Interface). Thus, one can run simulations on multiple CPU cores to get acceleration for a large system.

Simulation setup

To run the simulations in parallel and to get the most optimal performance several modifications are needed to the standard AWSEM-MD input file. See the Getting started page for instructions on how to setup a typical AWSEM-MD simulations. One of the files that is generated in the process is a *.in file that has the LAMMPS instructions for simulation setup. Make the following changes to this *.in file:

  1. Increase the neighbor skin to 24 by modifying the neighbor command, i.e.

neighbor 24 bin

This will make sure that all the necessary atoms are visible to each other. If the simulation crashes increase the neighbor skin further or check that your system is correctly constructed.

  1. Turn on the dynamic load balancing. This can be done in several ways (see the LAMMPS instructions for balance or fix balance commands). For example, add one of the following segments to your input file:
fix             4 all balance 100 1.05 shift xyz 50 1.05

comm_style      tiled
fix             4 all balance 100 1.1 rcb

The first option uses a brick-like partitioning of the x, y, and z directions, while in the latter case tiled partitioning is used.

  1. Optionally choose the processor grid based on your system geometry by including the processors command (see LAMMPS manual). This may be helpful for an asymmetric system or when the processor number you are using has large prime numbers in it. In those cases you can direct the resources where they are most necessary by explicitly specifying the number of processors in each dimension. E.g. processors 3 2 4. If the number of processors in any of the dimensions is 1, than the fix balance command should be modified accordingly to exclude that dimension.

  2. In addition, make sure that the box dimensions in the data.* file are reasonable for your system. All the atoms must be inside the box boundary, while, at the same time, the box should not be considerably larger than the system, because that may lead to poor performance.

Running parallel simulations

To run parallel simulations you will first need to compile an MPI-enabled version of LAMMPS. Most of the time that may be as simple as typing make mpi in the LAMMPS src directory. However, additional libraries may be required to be installed or loaded on your system prior to the compilation.

After successful compilation a lmp_mpi executable will appear in the src folder. To run parallel simulations on a local machine simply type:

mpirun -n 4 $LAMMPS_dir/src/lmp_mpi -in project.in

Here, -n option for mpirun specifies the number of processors to use, while the -in option for lmp_mpi specifies the input file for your project.