By inserting the FoamToPbvr adapter that comes with pbvr into OpenFOAM, the OpenFOAM solver can generate particles for visualization, which is stored in the Filter directory and can be used by referencing and linking to the particle generation library like the particle sampler. The adapter converts the variable array of computed physical values into the vtkUnstructuredGrid class using the vtk library and passes it to the particle sampler. The following shows the operation procedure of the OpenFOAM wind around building sample code attached to PBVR.

In this case, the sample code is run on a supercomputer (SGI8600) owned by JAEA. When operating on a separate server, the module, etc. should be read separately.

[Module] GNU/9.5.0 openMPI/4.1.4

【OpenFOAM】　v2206（esi ver.）(pre-installed /home/app/OpenFOAM/OpenFOAM-v2206)

[Sample code name] wind around building

[vtk] v9.2 (pre-installed ''/home/center/viztools/VTK-9.2.2'')

【shell】 bash

The module loading command is as follows.

module purge; module load gnu/9.5.0　openmpi/4.1.4 

Here, the path of the installation directory is ($install_dir_pbvr).

In the installation directory, run git clone.

git clone [email protected]:CCSEPBVR/CS-IS-PBVR.git 

In order to create a library for OpenFOAM, change the compile settings of pbvr as shown in the figure as shown below. Change PBVR_MACHINE to gcc and VTK_PTAH to true. Also, if necessary, set the path to the pre-installed VTK library. (Default)

As soon as the configuration is complete, run the compilation.

make -j 40

In order to use the pbvr library in OpenFOAM, the compilation settings are matched with pbvr. In SGI8600, the pre-installed OpenFOAM is copied locally and the settings are changed.

If the path of the OpenFOAM installation directory is ($install_dir_foam), the copy command is as follows.

rsync -r /home/app/OpenFOAM/OpenFOAM-v2206  ($install_dir_foam)
rsync -r /home/app/OpenFOAM/ThirdParty-v2206  ($install_dir_foam)

Once copied, set the projectDir of ($install_dir_foam)/OpenFOAM-v2206/etc/bashrc to ($install_dir_foam) and change the environment variable WM_COMPIKER to Gcc as shown in the red line in the figure.

From here, enable the OpenFOAM command. First, run /etc/bashrc once.

source ($install_dir_foam)/OpenFOAM-v2206/etc/bashrc

Run ./allwmake in the OpenFOAM directory.

cd OpenFOAM-v2206
./Allwmake

Ready to enable by running /etc/bashrc again. When the help log appears with the following command, it is enabled.

source ($install_dir_foam)/OpenFOAM-v2206/etc/bashrc
icoFOAM –help 

In the compile configuration file (''($install_dir_foam)/wmake/rules/General/Gcc/c++''), compile command part ( CC = ... ) as shown in the figure.

CC          = mpic++$(COMPILER_VERSION) -std=c++14 -m64 -pthread -fopenmp

Go to the sample code directory for OpenFOAM (here, IS_DaemonAndSampler/Example/C/gcc_mpi_omp/cavity_flow_vtk) and add the path in Make/options to use the pbvr library as follows. (Already added in the sample code)

This completes the configuration on the OpenFOAM side.

Build the sample code. After compilation is complete, binaries are generated in $FOAM_APPBIN.

wmake -j

As a pre-processing, execute mesh generation scripts and commands.

./Allrun.pre
decomposePar

Set the environment variables as per the in-situ setup.

export VIS_PARAM_DIR=($install_dir_pbvr)/CS-IS-PBVR/IS_DaemonAndSampler/Example/C/gcc_mpi_omp/windAroundBuildings_v2206
export  PARTICLE_DIR=($install_dir_pbvr)/CS-IS-PBVR/IS_DaemonAndSampler/Example/C/gcc_mpi_omp/windAroundBuildings_v2206/particle_out

Create a particle_out as a directory for particle data.

mkdir particle_out

Now that the solver is ready to run, run the OpenFOAM solver. Since icoFoam.C is used as a sample code, the execution command is as follows.

mpirun -n 2 simpleFoam -parallel

The procedure for running the daemon on a remote server is shown here. Note that the solver and daemon are expected to run concurrently, so it is recommended to work in a new tab (don't forget to load the module).

The construction work of the daemon code is omitted because it is performed at the same time as the solver is built. Set the following two environment variables before execution.

export VIS_PARAM_DIR=($install_dir_pbvr)/CS-IS-PBVR/IS_DaemonAndSampler/Example/C/gcc_mpi_omp/windAroundBuildings_v2206
export  PARTICLE_DIR=($install_dir_pbvr)/CS-IS-PBVR/IS_DaemonAndSampler/Example/C/gcc_mpi_omp/windAroundBuildings_v2206/particle_out

Navigate to the Daemon directory and run the pbvr_daemon.

cd ($install_dir_pbvr)/CS-IS-PBVR/IS_DaemonAndSampler/Daemon
./pbvr_daemon

“waiting connection… If the log appears, it is successful. Since it is in a waiting state for connection, move on to connection operation from the client side without stopping.

You have been given a tutorial on how to port forward, run a client, and connect to a remote server.

The visualization result of the sample code is shown in the figure below. The visualization parameters at this time are described below.

• Time step: 400 (steady state)

Transfer function

• Color Function Synthesizer : C1
• Opacity Function Synthesizer : O1
• Color Map Function [C1]: $dq 1x * dq2y + dq2y * dq3z + dq3z * dq1x - dq1y * dq2x - dq2z * dq3y - dq3x * dq1z$
• Opacity Map Function[O1] : $dq 1x * dq2y + dq2y * dq3z + dq3z * dq1x - dq1y * dq2x - dq2z * dq3y - dq3x * dq1z$
• minmax mode: User defined Min Max

Glyph

• scale factor : 15
• Distribution mode: Uniform Distribution
• Size: constant
• Color : Variable Array
  • Number of variables: 3
    • q1
    • q2
    • q3

plot over line

• Resolution:256
• Target: q1
• Start point: (-20, 160, 50)
• End point: (350, 160, 50)

The calculation content of this sample code is a simulation of wind conditions around the city (WindAroundBuilding) in 3D space. Wind is flowing in the x direction (from the boundary on the front side).

The calculation parameters are as follows. The particles are the Q values, the glyphs are the magnitudes of the wind speeds, and the plot over line is the x-component distribution of the wind speeds

• MPI parallel number: 2MPI (If you want to change the number of parallels, see [#MPI並列数の変更方法] below)
• Inflow velocity (m/s): 10
• Number of cells: 185,237 *cell types :hexahedra, pyramid, polyhedron
• Computing field (L_x, L_y, L_z): (350,280,140)
• Reynolds number (kinematic viscosity): 2.33e8 (0.00001)

The above figure is just an example, and the required files vary depending on the solver.

Here, an example of incorporating PBVR function into simpleFoam.C and output interval control is explained.

Like OpenFOAM, the adapter uses a method of inserting include files. As shown in the figure, the PBVR function is called by expanding the specified filter file (red line). (Details will be described later)

There are two types of adapters, the polyhedron type and the mixed element type. However, in this sample code, only the polyhedron type can be used.

The adapter uses a .h file with (initial_step ) on the first use, and an .h file with (later_step) on the second and subsequent uses.

Here, a FoamToPbvr adapter that performs information refilling processing from OpenFOAM format to VTK format will be described. This adapter uses vtk library functions to convert coordinates, connection information, and variable data in OpenFOAM format to the vtk format supported by the particle sampler. Conversion processing is performed for variable data at each output step, but conversion processing is performed only for the first output step for coordinate and connection information data, reducing processing costs. In addition, conversion processing from cell data to point data is also performed during the conversion process.

Here, the procedure for changing the number of MPI parallels in the sample code is explained. If you want to change the number of MPI parallels to N, you need to start over from the preprocessing stage. Change the number of ''numberOfSubdomains'' in ''system/decomposeParDict'' to N, Set the value of simpleCoeffs so that ''N = xyz'' is true as shown below.

After making changes, run Allrun.pre again. (At this time, the pre-processed log file (log....) )

  1 /*--------------------------------*- C++ -*----------------------------------*\
  2 | =========                 |                                                 |
  3 | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
  4 |  \\    /   O peration     | Version:  v2206                                 |
  5 |   \\  /    A nd           | Website:  www.openfoam.com                      |
  6 |    \\/     M anipulation  |                                                 |
  7 \*---------------------------------------------------------------------------*/
  8 FoamFile
  9 {
 10     version     2.0;
 11 ASCII format;
 12     class       dictionary;
 13     object      decomposeParDict;
 14 }
 15 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
 16 
 17 numberOfSubdomains N;   ←this number
 18 
 19 method          simple;
 20 
 21 simpleCoeffs
 22 {
 23   n ( x y z );　← Set the product of the three numbers to be equal to N.
 24   delta 0.001;
 25 }
 26 

After that, the sample code is executed in parallel number N.

mpirun -n N simpleFoam -parallel

If you want to change or add the calculation parameters you want to visualize, you need to edit the adapter.

The adapter is stored in the ''($install_dir_pbvr)/CS-IS-PBVR/IS_DaemonAndSampler /insitulib/unstruct/filter directory.

You can increase the number of variables by copying and pasting the black frame part of the figure below and inserting the variable array to be visualized into InsertNextValue(). Be careful not to duplicate variable names.