Development guide - easybuilders/easybuild GitHub Wiki
So you want to extend EasyBuild? Great idea!
There are three major parts to EasyBuild:
-
Framework with:
- main.py: the main script which sets up the rest of the application and determines the build-order, this is called through the
ebcommand which sets a correct Python environment before starting. - framework: the general EasyBlock, EasyConfig and Extension classes.
- tools: General utilities used trought the framework code.
- asyncprocess: Written by Josiah Carlson (http://code.activestate.com/recipes/440554/ )
- build_log: wrapper around Python's logger (will be replaced by fancylooger soon
- config: proxy for the Configuration
- filetools: file management and running commands
- module_generator: generates module-files
- modules: interface to the module command
- repository: interface to the configured source control system such as pySVN subversion or the filesystem.
- toolchains: Contains implementations of toolchain components with their switches and flags. This inculdes the compiler, math libs, mpi implementations...
- others...
- main.py: the main script which sets up the rest of the application and determines the build-order, this is called through the
-
EasyBlocks: Custom EasyBlock classes are grouped here.
-
EasyConfig: A (huge) list of example EasyConfigs
First check if the available functionality is already available in EasyBuild. Some Generic EasyBlock classes are very flexible and might be sufficient for your needs.
If not, go ahead and create a new class inheriting from EasyBlock or a Generic subclass or even an specific software package class (you can get a quick overview of all available classes using --dump-classes).
Each step in the build-process consists out of one or more methods which you can override. If you want to skip a step, just implement it with pass. See Easyconfig files for options for each step in the default implementation.
- Get the sources:
-
fetch_step: fetches files, either from local filesystem, or from a remote url. (We only support http and ftp for now, git/svn support is on the wishlist, contact us if you feel like adding it)
-
- Generate installation path and create build directory
check_readinesstepgen_installdirmake_builddir-
reset_changes: resets changes in the environment
- Unpack source in the build directory
-
checksum_step: currently empty, see https://github.com/hpcugent/easybuild-framework/issues/214 extract_step
-
- Apply patches
patch_step
- Configure
-
prepare_step: prepare toolkit (using self.tk.prepare) configure_step
-
- Build
build_step
- Test (run make test or variants)
test_step
- Install
stage_install_stepmake_installdirinstall_step
- Extensions: build extensions (if any)
extensions_step
- Upload eb files to repository
post_instal_step
- Check installed files
sanity_check_step
- Cleanup
cleanup_step
- Create module
make_module_step
Some specification options are added in an EasyConfig, to see the available options, run eb --avail-easyconfig-params
On EasyBlock this returns:
MANDATORY
---------
name: Name of software
version: Version of software
toolchain: Name and version of toolchain
description: A short description of the software
homepage: The homepage of the software
EASYBLOCK-SPECIFIC
------------------
tar_config_opts: Override tar settings as determined by configure.
TOOLCHAIN
---------
toolchainopts: Extra options for compilers
onlytcmod: Boolean/string to indicate if the toolchain should only load the environment with module (True) or also set all other variables (False) like compiler CC etc (if string: comma separated list of variables that will be ignored). (default: False)
BUILD
-----
easybuild_version: EasyBuild-version this spec-file was written for
versionsuffix: Additional suffix for software version (placed after toolchain name)
versionprefix: Additional prefix for software version (placed before version and toolchain name)
runtest: Indicates if a test should be run after make; should specify argument after make (for e.g.,"test" for make test) (default: None)
preconfigopts: Extra options pre-passed to configure.
configopts: Extra options passed to configure (default already has --prefix)
premakeopts: Extra options pre-passed to build command.
makeopts: Extra options passed to make (default already has -j X)
preinstallopts: Extra prefix options for installation (default: nothing)
installopts: Extra options for installation (default: nothing)
unpack_options: Extra options for unpacking source (default: None)
stop: Keyword to halt the buildprocess at certain points. Valid are ['cfg', 'source', 'patch', 'prepare', 'configure', 'make', 'install', 'test', 'postproc', 'cleanup', 'extensions']
skip: Skip existing software (default: False)
parallel: Degree of parallelism for e.g. make (default: based on the number of cores and restrictions in ulimit)
maxparallel: Max degree of parallelism (default: None)
sources: List of source files
source_urls: List of URLs for source files
patches: List of patches to apply
tests: List of test-scripts to run after install. A test script should return a non-zero exit status to fail
sanity_check_paths: List of files and directories to check (format: {'files':<list>, 'dirs':<list>}, default: {})
sanity_check_commands: format: [(name, options)] e.g. [('gzip','-h')]. Using a non-tuple is equivalent to (name, '-h')
FILE-MANAGEMENT
---------------
start_dir: Path to start the make in. If the path is absolute, use that path. If not, this is added to the guessed path.
keeppreviousinstall: Boolean to keep the previous installation with identical name. (default: False) Experts only!
cleanupoldbuild: Boolean to remove (True) or backup (False) the previous build directory with identical name or not. (default: True)
cleanupoldinstall: Boolean to remove (True) or backup (False) the previous install directory with identical name or not. (default: True)
dontcreateinstalldir: Boolean to create (False) or not create (True) the install directory (default: False)
keepsymlinks: Boolean to determine whether symlinks are to be kept during copying or if the content of the files pointed to should be copied
DEPENDENCIES
------------
dependencies: List of dependencies (default: [])
builddependencies: List of build dependencies (default: [])
osdependencies: OS dependencies that should be present on the system
LICENSE
-------
license_server: License server for software
license_serverPort: Port for license server
key: Key for installing software
group: Name of the user group for which the software should be available
EXTENSIONS
----------
exts_list: List with extensions added to the base installation (default: [])
exts_defaultclass: List of module for and name of the default extension class (default: None)
exts_filter: Extension filter details: template for cmd and input to cmd (templates for name, version and src). (default: None)
MODULES
-------
modextravars: Extra environment variables to be added to module file (default: {})
moduleclass: Module class to be used for this software (default: base) (valid: ['base', 'compiler', 'lib'])
moduleforceunload: Force unload of all modules when loading the extension (default: False)
moduleloadnoconflict: Don't check for conflicts, unload other versions instead (default: False)
OTHER
-----
buildstats: A list of dicts with build statistics
Sometimes you want to add extra options that should be configurable in the EasyConfig file.
EasyConfig files are read by the EasyConfig-class. Instead of overriding this class, we extend the extra_options function in EasyBlock to return list of your specification options.
To do so, create the an extra_options which returns a list of new options that can be added.
Some software can only be configured by setting some environment variables. Use the easybuild.tools.environment module for this.
This will set environment variables, and keep track of them in between steps, so you have more information what happened when whilst debugging.
For defining what environment variables to set to what path in the produced environment modules file you need to implement the make_module_req_guess method. This method will check if a path exists, and if so create an environment variable pointing to it.
This method should return a dictionary of {'VARIABLE_NAME': 'directory'} Default:
{
'PATH': ['bin'],
'LD_LIBRARY_PATH': ['lib', 'lib64'],
'CPATH':['include'],
'MANPATH': ['man', 'share/man'],
'PKG_CONFIG_PATH' : ['lib/pkgconfig', 'share/pkgconfig'],
}If you want to define environment variables not related to a path you will have to implement the make_module_extra method. This method needs to return a string to add to the module file after the automatic dependencies and paths are set.
from easybuild.framework.easyconfig import CUSTOM
from easybuild.easyblocks.generic import ConfigureMake
from easybuild.tools import environment
class EB_MySoftware(ConfigureMake)
"""Install MySoftware"""
def __init__(self, *args, **kwargs):
"""Constructor"""
ConfigureMake.__init__(self, *args, **kwargs)
self.subdir = ""
def configure_step(self):
"""Configuration step, we set FC to F90,
F77 and F90 are already set by EasyBuild to the right compiler, but this tool uses FF for F90 compiler.
"""
environment.setvar("FC", self.toolchain.get_variable('F90'))
self.cfg.update('makeopts', CC="%(CC)s"' % {'CC': os.getenv('CC')})
ConfigureMake.configure_step(self)
# different subdir if doing a source install
if self.cfg['sourceinstall']:
self.mysubdir = "%s-%s" % (self.name.lower(), self.version)
@staticmethod
def extra_options():
extra_vars = [
('sourceinstall', [False, 'Perform an installation in a different subdirectory', CUSTOM]),
# ('config-key', [<default>, <description>, CUSTOM ]),
]
return ConfigureMake.extra_options(extra_vars)
def make_module_req_guess(self):
"""Specify correct LD_LIBRARY_PATH and CPATH for this installation."""
guesses = ConfigureMake.make_module_req_guess(self)
guesses.update({
'CPATH': [os.path.join(self.subdir, "include")],
'LD_LIBRARY_PATH': [os.path.join(self.subdir, "lib")]
})
return guesses
def make_module_extra(self):
"""Set specific environment variables (mysubdir)."""
txt = ConfigureMake.make_module_extra(self)
txt += self.moduleGenerator.set_environment('MYSOFTWARE_DIR', '$root/%' % self.mysubdir)
return txt
def sanity_check_step(self):
"""Custom sanity check"""
custom_paths = {
'files': [],
'dirs': [os.path.join(self.mysubdir, x) for x in ["conf", "include", "lib"]]
}
ConfigureMake.sanity_check_step(self, custom_paths=custom_paths)