ticket_215_TicketSummary - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

Building UK Met Office ODB version 1.0.0

Source code, Odb-1.0.0-Source-meto

  1. David Davies sent to Jin Lee the UKMO ODB source code, "Odb-1.0.0-Source-meto". The tarball was saved under,

    raijin1:/projects/access/downloads/odb

  2. "Odb-1.0.0-Source-meto" was imported into ODB trunk,

    https://access-svn.nci.org.au/trac/odb/browser/trunk/odb/Odb-1.0.0-Source-meto

  3. Created a branch of the above trunk for this development work,

    https://access-svn.nci.org.au/trac/odb/browser/branches/dev/jtl548/odb/r64_Odb-1.0.0-Source-meto_nci

In this ODB 1.0.0 build the ODB source from UKMO (Odb-1.0.0-Source-meto) was used rather than the source from ECMWF (Odb-1.0.0-Source). This choice was maded to eliminate any possibility of ODB being incompatibility with UKMO OPS. Email exchanges regarding this is here

Compile-time environment

Before doing the build it's important to start in a clean module environment. So following minimalist module environment was used,

module purge
module load subversion

Following module load's are used to set up compile-time environment for building Odb-1.0.0-Source-meto,

module unload intel-cc intel-fc python python cmake odb odbapi netcdf
module load intel-cc/14.2.144
module load intel-fc/14.2.144
module load python/2.7.6
module load cmake/3.0.0
module load odb/30.0 ????
module load odbapi/30.0 ???
module load netcdf/4.3.0

The module load's are put in a standalone script, "compile_time_env.sh".

Note on serial and parallel IO in ODB:

  1. ODB IO is parallel - i.e. IO is carried out using MPI. This means ODB source files make MPI calls such as mpi_init(), etc.
  2. However ODB tools and ODB executabels themselves are non-MPI: i.e. serial and use the library "libmpi_serial.a"
  3. OPS on the other hand use MPI: i.e. the Fortran90 application which interfaces to the ODB library makes use of MPI for parallel data query. The reason why this is possible even though ODB library is built with serial IO is that during the OPS build ODB is built from source files at the same time. This is done using MPI compiler wrappers such as "mpicc" and "mpif90" (or "mpifort")
  4. the ODB build does not require MPI compiler wrappers such as "mpicc" and "mpif90" (or "mpifort") since ODB has its own MPI subroutines which are stubs to MPI calls (????)
  5. ???? need clearer explanation (see Xavi's and David's emails)

Modifying ECMWF-provided configuration script

1. Build basic ODB

This builds the Fortran interface (library and Fortran module files) and various scripts like odbdup, odbless, odbsql for command line usage.

CMake is used to do the build so a new script, "config.nci-intel.sh" was created based on a sample script, "config.intel.sh" which was supplied by ECMWF. Using the instructions in "README" following changes were needed to come up with a config script that builds ODB for Raijin:

Modification Reason for modification Additional remark
module loads were put in to set up compile-time environment on Raijin we use modulefiles
CMAKE_INSTALL_PREFIX our installation directory is different
CMAKE_C_COMPILER our Intel C compiler is icc
CMAKE_C_FLAGS added the compiler option, '-fpic' to produce position-independent .o files during ODB API build at the linking step one or more of ODB API dynamic libraries are linked to "libodb.a" ODB library. This means the object files in "libodb.a" must be position independent
CMAKE_CXX_COMPILER our Intel C++ compiler is icpc
CMAKE_Fortran_COMPILER our Intel Fortran compiler is ifort
CMAKE_Fortran_FLAGS '-dynamic' and '-fpp' are removed as these are not valid Intel Fortran compiler options; added '-fpic'; the others are retained following the recommendation in README file
NETCDF_PATH the path to our NetCDF library is different
ODB_API_PATH the path to our ODB API library is different
CMAKE_VERBOSE_MAKEFILE:BOOL=YES this creates verbose Makefile which in turn will give verbose output during make

The config script was executed from build/ directory (N.B. '..' as a commandline argument to config.nci-intel.sh):

raijin5:/home/548/jtl548/source/odb/r64_Odb-1.0.0-Source-meto_nci/build> ../config.nci-intel.sh .. \
> out.config.nci-intel.sh.txt 2>err.config.nci-intel.sh.txt &

After a successful completion of the config script I ran 'make' (N.B. use of the switch, 'j' causes make to fail). Note that before running 'make' you will need to provide the paths for the C, C++ and Fortran compilers by loading modulefiles of appropriate compiler versions.

'make' was completed successfully so the last step I ran was 'make install'.

2. Building ODB with additional tools

This builds additional tools such as odbsql, odbless, etc. Similar changes as "1. Build basic ODB" are required as well as following additional changes to the config script:

Modification Reason for modification Additional remark
ODB_API_INCLUDE_DIR needed for some reason provided ODB_API_PATH is specified ODB_API_INCLUDE_DIR shouldn't be needed but for some reason this is required (bug?)
ECLIB_INCLUDE_DIR include files for eclib
ODB_API_TOOLS to produce extra ODB tools to build ODB plotting tools such as odbviewer, odbxyplot, etc. you need to build ODB with Magics/Magics++ libraries

Note 1. Config script for producing extra tools doesn't seem quite right:

  • During a linking step the linker doesn't seem to find libOdb.a which is under ODB API library (linker option of '-lOdb')
  • A workaround was that before the 'make' command I manually did 'module load odbapi/30.0' to make LD_RUN_PATH environment variable available to the linker
  • Damian Agius suggested that all compile-time environment be set up using modulefiles rather than hard-wiring paths using the -D options of CMake. The advantage of Damian's suggestion is that correct libraries will be linked
  • He cited an example where a linker incorrectedly linked system NetCDF library instead of \apps NetCDF library because modulefile for NetCDF was not loaded properly
  • Perhaps the config script should be changed to contain minimum set of -D CMake options.

Note 2. There is a bug in this version of the script "odbsql":

  • if the path name of ODB is too long then odbsql query fails. This was fixed by manually applying the change suggested by ECMWF.

Building ODB API libraries

Once ODB is built you need to rebuild ODB API so that the ODB API libraries can properly link to the new ODB. E.g. one of the ODB API dynamic libraries, libOdbmigrator.so contains .o files of ODB.

CMake and its use during ODB build

CMake

Followign is an extract from http://www.cmake.org/cmake/help/v3.0/manual/cmake.1.html:

The “cmake” executable is the CMake command-line interface. It may be used to configure projects in scripts. Project configuration settings may be specified on the command line with the -D option.

CMake is a cross-platform build system generator. Projects specify their build process with platform-independent CMake listfiles included in each directory of a source tree with the name CMakeLists.txt. Users build a project by using CMake to generate a build system for a native tool on their platform.

How CMake is used during ODB build

You won't need a complete understanding of how CMake works but following points might be helpful during ODB build:

  1. Normally you will invoke cmake from a build directory,
raijin5:.../r64_Odb-1.0.0-Source-meto_nci/build> cmake ..

The commandline argument to cmake is '..' which is the source code directory. Note in ODB build we use a config script which sets up compile-time environment and calls cmake.

  1. After "cmake" is finished - i.e. the config script completes successfuly - you can run ccmake to check settings in more detail: e.g whether all the libraries are available
  • to run "ccmake" go down to the build directory and then type ccmake : e.g. ccmake .. or ccmake .
  • type 't' to toggle to advanced mode
  1. If cmake runs successfully Makefiles are created. These Makefiles will enable you to compile ODB source files and then link them to appropriate libraries.

ToDo

  • ran ccmake and it says some worrying things . following libraries are not found,

    NETCDF_CXX_LIBRARY
NETCDF_C_LIBRARY
NETCDF_INCLUDE_DIR
ODB_API_INCLUDE_DIR
ODB_API_LIBRARY

    . ODB_API_TOOLS is OFF

  • rebuild ODB using Magics/Magics++ to build ODB plotting tools such as odbviewer, odbxyplot, etc. . see raijin6:/home/548/ttl548/odb/odb30.0/make.linux for settings . ask ECMWF what part of config script needs changing

  • config script for producing extra tools doesn't seem quite right. During a linking step linker doesn't seem to find libOdb.a which is under ODB API library (linker option of '-lOdb'). I had to do manually 'module load odbapi/3.0' to make LD_RUN_PATH variable available to the linker. Change config script so that it produces correct Makefiles.

  • rearrange installation directory so that the directory tree looks like /home/548/jtl548/apps/odb/linux/1.0.0

    • installation without tools should be 1.0.0.basic

  • Modify ~jtl548/odb/scripts/odb.ksh to enable setting of ODB environment for my ODB build

  • change installation directory so that it conforms with the standard installation of ODB

  • put compile-time environment in a separate file which gets sourced by CMake config file and which can be sourced prior to make

  • config.nci-intel-tools.sh wasn't updated as ODB API library is not quite stable yet: whether to use the build that uses 64-bit integer or 32-bit integer. Once this is decided config.nci-intel-tools.sh should be modified a la config.nci-intel.sh.