GridPACK on Windows - wperkins/GridPACK GitHub Wiki

This describes building GridPACK on a specific system. Perhaps, it will be a useful guide for building on another system.

Table of Contents

Key Things

Building GridPACK on Windows is not for the faint of heart. GridPACK was developed on and for high performance Linux or UNIX-like systems. GridPACK depends on several libraries that were also developed on and for UNIX-like systems. The Windows programming model is very different from Linux or UNIX. If you are familiar with one, the other is usually confusing.

On the other hand, a sucessful build will allow GridPACK to take full advantage of multi-core Windows systems, or perhaps a Windows cluster.

There are a number key things to keep in mind while attempting to build GridPACK on Windows:

  • Make sure all code is compiled with the same compiler. Even if you think only one version of MS Visual C++ is installed, there may be different compiler (CL.EXE) versions. Make certain that that the compiler CL.EXE is the same one that CMake uses. Make sure all the libraries are built with the same compiler.

  • Make sure all libraries are built with the same run time library option. For some good reason, surely, code referring to different run time libraries cannot be linked together. Here, all code is linked to the shared (non-debug) run time library, which means code is compiled with the /MD option.

  • The build description here is that recorded when GridPACK was built on a single system. Your system is different. Adjustments will be required.

  • A Windows user account building and/or debugging programs may need to have special permissions ("Developer"?). Administrator privileges may also be necessary for some tasks.

  • In this build, everything is done from a Command Prompt or shell. Use a 64-bit Visual Studio command prompt, which should be available from the Start Menu. If your version of Visual Studio has more than one type of Command Prompt, use the "Developer" prompt. You may need to open Command Prompts as administrator, depending on your installation and user account. To do this, mouse over the Command Prompt field and right-click. A menu should show up with "Run as Administrator" as one of the options. Type, or copy and paste, the commands below into that command prompt window.

  • Disclaimer: Some of the actions listed here may require Administrator privileges. Mistakes made with Administrator privileges can do serious damage to your system. If you are uncomfortable using Administrator privileges, don't. Just walk away. If you proceed and damage your system, it is your fault.

  • Disclaimer: The writer is not a Windows developer, and not particularly knowledgeable about Windows programming.

System Preparation

There are several requirements that need be installed. These can be installed in the normal Windows way:

  • Visual C++ in some form.

    • A C++ compiler is necessary. Instructions for this are elsewhere. The "free" ones, Visual Studio Express and Community, should work, but were not tested. Visual Studio 2015 was used here.

    • Use 64-bit compilers on a 64-bit system. To get the 64-bit compilers for Visual Studio 2015, you need to use the "x64 Native Tools Command Prompt" (you may need to go into the Visual Studios folder and then into the Visual Studio Tools subfolder and then into the Windows Desktop Command Prompts folder under that to find it).

  • Windows SDK, which is usually installed with Visual Studio

  • CMake is required.

    During the installation process, the dialog will ask if you want CMake added to PATH variable. For some reason, the default is not to include CMake. You should change the selection to add CMake to the PATH for all users.

  • A minimal Cygwin installation is necessary.

    Cygwin is required to build, and test for, PETSc. If the system has a Cygwin installation that is being used, it would probably be best to make another installation for GridPACK installation exclusively. If gcc and cmake are already installed in Cygwin, you can rerun the installer and deselect these packages (they can be found under the developer category).

    • Minimum required packages:

      • Base
      • Python
      • Make

    • Rename /usr/bin/link.exe so it does not interfere with Windows LINK.EXE

    • If you want to use a Cygwin shell to build and/or debug GridPACK applications:

      • Do not install a compiler set.
      • Do not install CMake.

  • Microsoft MPI

    This appears to be the only modern MPI implementation available for Windows. Download and install both the .msi and .exe files.

  • Some software to unpack .zip, .gz, and tar archives. Commands to unpack all of these archives are available with Cygwin or the GIT shell. Windows can handle .zip archives natively. 7-Zip also handles these archive types.

  • GIT

    Git is used to obtaining the GridPACK and other code from the Github repository. It also supplies a Linux-like command prompt, that is easier to use than Cygwin's, with some necessary Linux commands, like patch and tar.

  • (optional) MS-MPI Debugger Extension is useful for debugging problems with parallel programs.

Build System Description

This build was done on a specific system.

Batch files and shells scripts used to build GridPACK and its dependencies on this system are in the source repository.

The writer had Administrator privileges on this system, which may be necessary.

Compiler Identification

The first compiler found by the Command Prompt is the one used by the Boost and PETSc build systems. To verify its location and version,

C:\>where cl
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\cl.exe
C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\cl.exe
C:\>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24210 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

This shows that the first compiler found is from MSVC 14 and its version is 19.00.24210.

C:\>cmake --help
Usage

  cmake [options] <path-to-source>
  cmake [options] <path-to-existing-build>
  cmake [options] -S <path-to-source> -B <path-to-build>

Specify a source directory to (re-)generate a build system for it in the
current working directory.  Specify an existing build directory to
re-generate its build system.

Options
[...]
Generators

The following generators are available on this platform (* marks default):
  Visual Studio 16 2019        = Generates Visual Studio 2019 project files.
                                 Use -A option to specify architecture.
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
* Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
[...]

Build Prerequisite Libraries

  • Choose a single place to install GridPACK and related libraries, C:\GridPACK or C:\Users\TheUser\gridpack. Avoid a path with spaces or special characters in it. It's convenient to set an (user) environment variable to hold this path. GridPACKDir is used here.

  • Make a folder in a convenient location in which source archives can be unpacked and the builds can be performed. %GridPACKDir%\src is used here[^1]. Again, avoid a path with spaces or special characters in it.

  • Open an appropriate "Developer" Command Prompt as described above. It may be necessary to adjust the PATH environment variable to remove anything that might interfere with the build process. These steps are suggested

    • In the Command Prompt, type 
      echo %PATH%
    • Mark and copy the result and paste it into a text editor like NotePAD. Make it into a single line, being careful to preserve any spaces.
    • Examine each path entry, separated by ;, and remove those that are unneeded or might interfere. Some examples:
      • The R statistics package has commands like make that interfere with Cygwin.
      • The only Python available should be that from Cygwin.
    • Save the editted PATH in a text file for future reference.
    • Mark and copy the editted PATH and in the Command Prompt, type 
      set PATH=
      and paste the new path.

  • Make sure that MPI variables are set correctly by typing,

    set msmpi

    The response should be something like

    MSMPI_BIN=C:\Program Files\Microsoft MPI\Bin\
MSMPI_INC=C:\Program Files (x86)\Microsoft SDKs\MPI\Include\
MSMPI_LIB32=C:\Program Files (x86)\Microsoft SDKs\MPI\Lib\x86\
MSMPI_LIB64=C:\Program Files (x86)\Microsoft SDKs\MPI\Lib\x64\

  • Make sure the MPIEXEC command is available.

  • The caret ^ character is the line continuation character for Command Prompt window.

Boost

  • Download source from here and unpack in the source directory. Version 1.68.0 was used here and seemed to work out of the box.

  • Bootstrap boost.build, with

    .\bootstrap.bat

  • Add using mpi ; to project-config.jam

  • Configure, build, and install:

    .\b2 ^
   --prefix="%GridPACKDir%" ^
   --with-mpi ^
   --with-serialization ^
   --with-random ^
   --with-filesystem ^
   --with-system ^
   --build-type=complete ^
   threading=single ^
   address-model=64 ^
   link=static runtime-link=shared ^
   install

Notes:

  • Using runtime-link=shared causes Boost code to be built with /MD.

  • Boost decorates the library names with the compiler version, so explicitly specify the compiler and make sure that's the compiler you use for GridPACK. If multiple compilers are available, it is possible to force the boost configuration to pick a specific one by adding an option like -toolchain=msvc-11.0 to the .\b2 arguments.

  • This version of Boost built out of the box without difficulty.

  • Some references:

Global Arrays

  • Global Arrays is built using CMake. The CMake configuration in the current GA repository does not work without a Fortran compiler (at the time of writing).

    The fixed GA source can be obtained from an alternate repository

    git clone https://github.com/wperkins/ga.git

    This will probably have to be executed in the GIT shell.

  • Make a build directory, configure, build, and install:

    mkdir ga\build
cd ga\build
cmake -Wdev --debug-trycompile ^
    -D BUILD_SHARED_LIBS:BOOL=OFF ^
    -G "Visual Studio 14 2015 Win64" ^
    -D ENABLE_I8:BOOL=No ^
    -D ENABLE_BLAS:BOOL=No ^
    -D ENABLE_FORTRAN:BOOL=No ^
    -D ENABLE_CXX:BOOL=Yes ^
    -D GA_RUNTIME:STRING=MPI_TS ^
    -D CMAKE_INSTALL_PREFIX:PATH="C:\Users\D3G096\gridpack" ^
    ..
cmake --build . --config Release
cmake --build . --config Release --target install

METIS and ParMETIS

  • Get the official distribution from here. Unpack the archive in the source directory. The source distribution needs to modified to
    • have CMake look for MPI in the correct way
    • set the real type size
    • make sure metis.h is installed
    • deal with the lack of rint() on with later versions of MSVC. There is a patch in the GridPACK source repository. This patch can be applied by using the patch command within the GIT shell:
cd parmetis-4.0.3/
patch -p1 < .../parmetis-patch.txt
  • Configure, build, and install 
    cmake ^
 -G "Visual Studio 14 2015 Win64" ^
 -D BUILD_SHARED_LIBS:BOOL=NO ^
 -D METIS_INSTALL:BOOL=YES ^
 -D CMAKE_INSTALL_PREFIX:PATH=%GridPACKDir% ^
 ..
cmake --build . --config Release
cmake --build . --target install --config Release

Required PETSc Dependencies

Ordinarily, several packages used by GridPACK can be built by having PETSc download and build them as part of the PETSc build. Unfortunately, while this works quite well on the Linux OS, it does not work in many cases for Windows. Consequently, these packages must be downloaded and built separately. Note that the builds below all assume that you are calling the CMake command in a separate directory, below the main library directory.

BLAS and LAPACK (CLAPACK)

Some implementation of BLAS/LAPACK is required for PETSc and some other libraries. The implementation described here was chosen because it does not require a Fortran compiler. It is apparently ''really'' slow. It will probably be necessary to install Intel compilers and MKL in order to get improvement in speed. The Windows port of CLAPACK is described here.

  • Get the source here.

  • Configure, build, and install

    cd clapack-3.2.1-CMAKE
mkdir build
cd build
set CFLAGS="/MD"
set CXXFLAGS="/MD"
cmake -Wdev --debug-trycompile ^
    -D BUILD_SHARED_LIBS:BOOL=OFF ^
    -G "Visual Studio 14 2015 Win64" ^
    -D CMAKE_INSTALL_PREFIX:PATH="%GridPACKDir%" ^
    ..
cmake --build . --config Release
cmake --build . --config Release --target install

Suitesparse

The Suitesparse provides the KLU serial direct sparse linear solver, which is a good choice for some GridPACK applications.

  • Obtain the source for a variant suitable for Windows from here.

  • Configure, build, and install

    set GridPACKDir="C:/Users/D3G096/gridpack"
set CFLAGS="/MD"
set CXXFLAGS="/MD"
cmake -Wdev ^
     -G "Visual Studio 14 2015 Win64" ^
     -D BUILD_METIS:BOOL=NO ^
     -D LAPACK_DIR:PATH="%GridPACKDir%" ^
     -D BLAS_DIR:PATH="%GridPACKDir%" ^
     -D LAPACK_LIBRARIES:STRING="%GridPACKDir%/lib/lapack.lib" ^
     -D BLAS_LIBRARIES:STRING="%GridPACKDir%/lib/blas.lib;%GridPACKDir%/lib/libf2c.lib" ^
     -D CMAKE_INSTALL_PREFIX:PATH="%GridPACKDir%" ^
     ..
 cmake --build . --config Release
 cmake --build . --target install --config Release

    Note that forward slashes (/) are required in %GridPACKDir% instead of back slashes (\) for the Suitesparse CMake configuration. This seems to be the only package bothered by that.

Notes:

  • Some building hints are available here

SuperLU_DIST

SuperLU_DIST is needed so a parallel direct linear solver is available in GridPACK via PETSc. A specific SuperLU_DIST revision is required by PETSc depending on the PETSc version. Here, PETSc version 3.10.5 is being used, so SuperLU_DIST revision c0f64b8. This requires a specific Do not simply download the latest version of SuperLU_DIST. This will not work with most versions of PETSc.

  • Obtain the SuperLU_DIST by cloning the tagged revision: 
    git clone -b v6.1.1 https://github.com/xiaoyeli/superlu_dist
  • Edit SRC\util.c and comment out the include of unistd.h. It should look like this 
    #include <math.h>
/* #include <unistd.h> */
#include "superlu_ddefs.h"
  • Configure, build, and install: 
    set CFLAGS=/DWIN32 /D_WINDOWS /W3 /D_CRT_SECURE_NO_WARNINGS
set CXXFLAGS=/DWIN32 /D_WINDOWS /W3 /D_CRT_SECURE_NO_WARNINGS
cmake -Wdev ^
     -G "Visual Studio 14 2015 Win64" ^
     -D CMAKE_INSTALL_PREFIX:PATH="%GridPACKDir%" ^
     -D CMAKE_BUILD_TYPE:STRING=Release ^
     -D enable_double:BOOL=ON ^
     -D enable_openmp:BOOL=OFF ^
     -D enable_tests:BOOL=OFF ^
     -D enable_examples:BOOL=OFF ^
     -D USE_XSDK_DEFAULTS:BOOL=TRUE ^
     -D XSDK_ENABLE_Fortran:BOOL=FALSE ^
     -D BUILD_SHARED_LIBS:BOOL=FALSE ^
     -D TPL_ENABLE_BLASLIB:BOOL=OFF ^
     -D TPL_ENABLE_COMBBLASLIB:BOOL=OFF ^
     -D BLAS_LIBRARIES:STRING="%GridPACKDir%/lib/blas.lib;%GridPACKDir%/lib/libf2c.lib" ^
     -D LAPACK_LIBRARIES:STRING="%GridPACKDir%/lib/lapack.lib" ^
     -D TPL_PARMETIS_INCLUDE_DIRS:STRING="%GridPACKDir%/include" ^
     -D TPL_PARMETIS_LIBRARIES:STRING="%GridPACKDir%/lib/parmetis.lib;%GridPACKDir%/lib/metis.lib" ^
     ..
cmake --build . --config Release
cmake --build . --config Release --target install

PETSc

PETSc version 3.10.5 was used here. PETSc needs to be built within a Cygwin with the appropriate environment. Note that Cygwin file paths are different from Windows paths. This requires some attention.

  • Download the code from here and unpack.
  • Open ...\src\mat\order\amd\amd.c in an editor and change this line 
    tval = (PetscBool)Control[AMD_AGGRESSIVE];
    to 
    if (Control[AMD_AGGRESSIVE]) tval = PETSC_TRUE;
else tval = PETSC_FALSE;
  • Start a developer Command Prompt and set the PATH environment variable as above.
  • Start a minimal Cygwin shell 
    C:\cygwin64\bin\mintty.exe
    This will start a new window running the Cygwin shell.
  • In the Cygwin shell, 
    /usr/bin/bash -l
    will make the needed Cygwin commands available.
  • Configure: 
    prefix="/cygdrive/c/Users/D3G096/gridpack"
export PETSC_DIR="${prefix}/src/petsc-3.10.5"
python ./configure \
    PETSC_ARCH=mswin-c++-real \
    --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe cl" \
    --with-clanguage=c++ \
    --with-cxx-dialect=C++11 \
    --with-c++-support=1 \
    --with-precision=double \
    --with-scalar-type=real \
    --with-shared-libraries=0 \
    --with-debugging=0 \
    --with-mpi-include='[/cygdrive/c/PROGRA~2/MIA713~1/MPI/Include,/cygdrive/c/PROGRA~2/MIA713~1/MPI/Include/x64]' \
    --with-mpi-include=/cygdrive/c/Program\ Files\ \(x86\)/Microsoft\ SDKs/MPI/Include \
    --with-mpi-lib=['/cygdrive/c/Program\ Files\ \(x86\)/Microsoft\ SDKs/MPI/Lib/x64/msmpi.lib'] \
    --with-mpiexec='/cygdrive/c/Program\ Files/Microsoft\ MPI/Bin/mpiexec.exe' \
    --with-blaslapack-lib=[$prefix/lib/lapack.lib,$prefix/lib/blas.lib,$prefix/lib/libf2c.lib] \
    --with-metis=1 \
    --with-metis-include=${prefix}/include \
    --with-metis-lib=[${prefix}/lib/metis.lib] \
    --with-parmetis=1 \
    --with-parmetis-include=${prefix}/include \
    --with-parmetis-lib=[${prefix}/lib/parmetis.lib] \
    --with-suitesparse=1 \
    --with-suitesparse-include="${prefix}/include/suitesparse" \
    --with-suitesparse-lib=[${prefix}/lib/libumfpack.lib,${prefix}/lib/libamd.lib,${prefix}/lib/libbtf.lib,${prefix}/lib/libcamd.lib,${prefix}/lib/libccolamd.lib,${prefix}/lib/libcolamd.lib,${prefix}/lib/libcholmod.lib,${prefix}/lib/libcxsparse.lib,${prefix}/lib/libklu.lib,${prefix}/lib/libspqr.lib,${prefix}/lib/libldl.lib,${prefix}/lib/suitesparseconfig.lib] \
    --with-superlu_dist=1 \
    --with-superlu_dist-include="${prefix}/include" \
    --with-superlu_dist-lib="${prefix}/lib/superlu_dist.lib" \
    CFLAGS="-O2 -MD -wd4996" \
    CXXFLAGS="-O2 -MD -wd4996 -EHsc"
    Some notes on this configuration
    • Set PETSC_DIR to the directory where the PETSc source archive was unpacked.
    • The -EHsc flag is to enable C++ exceptions.
    • The --with-clanguage=c++ option is necessary because PETSc C code requires C99 and MSVC does not support that.
    • Change '--with-scalar-type=real' to '--with-scalar-type=complex' to build PETSc using complex numbers (change the PETSC_ARCH variable accordingly ).
    • Note format of file paths and how some are mangled. Spaces or special (to Cygwin) characters in paths can cause trouble. If problems are encountered, paths can be converted to Linux-type directory names by using the cygpath command inside the Cygwin shell. Type 
      cygpath -d "C:\Program Files (x86)\Microsoft SDKs\MPI\Include"
      to get 
      C:\PROGRA~2\MIA713~1\MPI\Include
      and use this name for any configure options that have a path listing.
  • Build 
    make PETSC_DIR=$PETSC_DIR PETSC_ARCH=mswin-c++-real all
  • Check 
    make PETSC_DIR=$PETSC_DIR PETSC_ARCH=mswin-c++-real check

Notes:

Build GridPACK

  • Clone Windows port branch of GridPACK from Githup

    git clone -b windoze2 https://github.com/wperkins/GridPACK.git

  • Configure

    cmake -Wdev --debug-trycompile ^
    -G "Visual Studio 14 2015 Win64" ^
    -D USE_PROGRESS_RANKS:BOOL=OFF ^
    -D BOOST_ROOT:PATH=%prefix% ^
    -D Boost_USE_STATIC_LIBS:BOOL=ON ^
    -D BOOST_INCLUDEDIR=%prefix%\include\boost-1_68 ^
    -D PETSC_DIR="%prefix%\src\petsc-3.10.5" ^
    -D PETSC_ARCH="mswin-c++-real" ^
    -D GA_DIR:PATH="%prefix%" ^
    -D PARMETIS_DIR:PATH="%prefix%" ^
    -D PARMETIS_INCLUDE_DIR:PATH="%prefix%\include" ^
    -D PARMETIS_LIB_DIR:PATH="%prefix%\lib" ^
    -D MPIEXEC_MAX_NUMPROCS:STRING="2" ^
    -D GRIDPACK_TEST_TIMEOUT:STRING=20 ^
    ..

  • Build

    cmake --build . --config Release -j

  • Run unit tests

    ctest -C Release

Notes

  • CMake refuses to use /MD with try_compile() and instead uses /MDd. This requires the use of CMAKE_USER_MAKE_RULES_OVERRIDE to change the CMAKE_<lang>_DEBUG flag initialization. This will enable FindPETSC and FindGA to work.
  • This build was for static libraries. A shared library build was not attempted.

[^1]: The value of an environment or batch script variable is obtained by placing the name between %.

⚠️ **GitHub.com Fallback** ⚠️