User Tools

Site Tools


easybuild_environment

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
easybuild_environment [2020/10/05 07:41] – [Build a software for which no easyconfig is available] lenocileasybuild_environment [2021/02/08 16:37] (current) jansen
Line 9: Line 9:
  
 ^Where ^OS^Mount Point^Remote Location^Protocol^ ^Where ^OS^Mount Point^Remote Location^Protocol^
-|Workstations \\ & Servers|Fedora 31| /easybuild/easybuild/fc31| Software Server| NFS| +|Workstations \\ & Servers|Fedora 33| /easybuild/easybuild/fc31| Software Server| NFS| 
 | ::: | RHEL7| /easybuild/easybuild/el7| Software Server| NFS|  | ::: | RHEL7| /easybuild/easybuild/el7| Software Server| NFS| 
-| ::: | RHEL7/Lustre| /easybuild/easybuild/el7_lustre| Software Server| NFS|  
 | ::: | RHEL8| /easybuild/easybuild/el8| Software Server| NFS|  | ::: | RHEL8| /easybuild/easybuild/el8| Software Server| NFS| 
 | Xmaris| CentOS 7| /marisdata/easybuild| Marisdata | NFS| | Xmaris| CentOS 7| /marisdata/easybuild| Marisdata | NFS|
Line 30: Line 29:
 First of all you need to setup your EasyBuild development stack. This will be hosted in a location on your server/workstation for which you have writing access. We first make EasyBuild available under our environment, then we define the location of our EasyBuild software stack ''/path/to/your/easybuild/stack'' and ultimately we prepend to the ''MODULEPATH'' the path in which our private-software stack modules will be installed. Using the bash syntax First of all you need to setup your EasyBuild development stack. This will be hosted in a location on your server/workstation for which you have writing access. We first make EasyBuild available under our environment, then we define the location of our EasyBuild software stack ''/path/to/your/easybuild/stack'' and ultimately we prepend to the ''MODULEPATH'' the path in which our private-software stack modules will be installed. Using the bash syntax
  
-<code>+<code bash>
 module load EasyBuild module load EasyBuild
  
Line 40: Line 39:
 If you want to use the softwares installed in your private stack on a variety of hardwares (workstations and servers) you must also instruct EasyBuild to build hardware-independent executables If you want to use the softwares installed in your private stack on a variety of hardwares (workstations and servers) you must also instruct EasyBuild to build hardware-independent executables
  
-<code>+<code bash>
 export EASYBUILD_OPTARCH=GENERIC export EASYBUILD_OPTARCH=GENERIC
 </code> </code>
Line 46: Line 45:
 Failing to do so, can result in the production of __non-portable__ softwares. On the other hand, we advise you build hardware-bound softwares in all cases in which execution performance is paramount. Failing to do so, can result in the production of __non-portable__ softwares. On the other hand, we advise you build hardware-bound softwares in all cases in which execution performance is paramount.
  
-Put particular attention if you are planning  to build  //OpenBLAS// via EasyBuild. In this case defining EASYBUILD_OPTARCH=GENERIC __is not sufficient__ to produce portable software (CPU independent). Use both ''export EASYBUILD_OPTARCH=GENERIC'' in your setup and ''--try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77"' '' as EasyBuild (''eb'') runtime option.+Put particular attention if you are planning  to build  //OpenBLAS// via EasyBuild. In this case defining EASYBUILD_OPTARCH=GENERIC __is not sufficient__ to produce portable software (CPU independent). Use both ''export EASYBUILD_OPTARCH=GENERIC'' in your setup and ''--try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77"' '' as EasyBuild (''eb'') runtime option. If your compilation fails you can try '' --try-amend=buildopts='TARGET=CORE2 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77"' '' instead.
  
  
Line 78: Line 77:
 At this point you can use your newly installed ''Miniconda3'' software by sourcing its module via At this point you can use your newly installed ''Miniconda3'' software by sourcing its module via
  
-<code>+<code bash>
 module load Miniconda3 module load Miniconda3
 which conda which conda
Line 149: Line 148:
 As you might have inferred, in all cases in which EasyBuild does not provide in its repos an easyconfig for the software you would like to install, you will have to pick up the //right// easyblock and write an //ad-hoc// easyconfig file which uses the chosen easyblock with appropriate parameters. This task is not simple. To make things more difficult, there could be cases in which you will have to write your own easyblock from scratch! Here follows an example easyconfig that will install a combo (bundle) of python packages all available in a single module. As you might have inferred, in all cases in which EasyBuild does not provide in its repos an easyconfig for the software you would like to install, you will have to pick up the //right// easyblock and write an //ad-hoc// easyconfig file which uses the chosen easyblock with appropriate parameters. This task is not simple. To make things more difficult, there could be cases in which you will have to write your own easyblock from scratch! Here follows an example easyconfig that will install a combo (bundle) of python packages all available in a single module.
  
-<code>+<code python>
 cat Quantum-TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb cat Quantum-TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb
 # #
Line 221: Line 220:
 Here follows a trivial -- perhaps not very useful -- example in which we create an easyblock that implements the following function: it prints a screen message when its corresponding module is loaded. This example should get you started and give you an idea of how easyblocks work. Here follows a trivial -- perhaps not very useful -- example in which we create an easyblock that implements the following function: it prints a screen message when its corresponding module is loaded. This example should get you started and give you an idea of how easyblocks work.
  
-<code>+<code python>
 # cat anacondaleonardo.py # cat anacondaleonardo.py
 from easybuild.easyblocks.a.anaconda import EB_Anaconda from easybuild.easyblocks.a.anaconda import EB_Anaconda
  
 class AnacondaLeonardo(EB_Anaconda): class AnacondaLeonardo(EB_Anaconda):
- 
     """Support for building/installing Anaconda and Miniconda."""     """Support for building/installing Anaconda and Miniconda."""
     def make_module_extra(self):     def make_module_extra(self):
Line 234: Line 232:
  
 </code> </code>
 +
 +Notice the following
 +
 +  * easyblocks are written in python
 +  * we are modifying the behaviour of the EasyBuild-provided ''EB_Anaconda'' easyblock by subclassing it
 +  * this easyblock can therefore be used to install the anaconda software
 +  * we override the function ''make_module_extra'' to adapt it to our needs
 +
 +Now create en easyconfig file that uses the newly created easyblock
 +
 +<code python>
 +#cat Miniconda2-4.3.21_mod.eb
 +easyblock = 'AnacondaLeonardo'
 +
 +name = 'Miniconda2'
 +version = '4.3.21'
 +
 +homepage = 'https://docs.conda.io/en/latest/miniconda.html'
 +
 +description = """Miniconda is a free minimal installer for conda. It is a small,
 +bootstrap version of Anaconda that includes only conda, Python, the packages they
 +depend on, and a small number of other useful packages.
 +
 +A warning message will be printed on the screen upon module loading.
 +
 +Author: leonardo
 +
 +"""
 +
 +toolchain = SYSTEM
 +
 +source_urls = ['https://repo.anaconda.com/miniconda/']
 +sources = ['%(name)s-%(version)s-Linux-x86_64.sh']
 +checksums = ['5097d5ec484a345c8785655113b19b88bfcd69af5f25a36c832ce498f02ea052']
 +
 +moduleclass = 'lang'
 +
 +</code>
 +
 +And install it via ''eb -r /path/to/easyconfig/Miniconda2-4.3.21_mod.eb --include-easyblocks=/path/to/my/easyblocks/*py''.
 +
 +Finally, do not forget to read the official [[https://easybuild.readthedocs.io/en/latest/Implementing-easyblocks.html|easyblocks documentation]]. 
  
  
Line 242: Line 282:
  
 ==== Heterogeneous environments ==== ==== Heterogeneous environments ====
-If you are planning to use your EasyBuild-built software on a variety of CPUs, do not forget to instruct EasyBuild to do so via ''export EASYBUILD_OPTARCH=GENERIC''  and ''eb ... –try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC=“$CC” FC=“$F77”' '' as EasyBuild (eb) runtime option. +If you are planning to use your EasyBuild-built software on a variety of CPUs, do not forget to instruct EasyBuild to do so via ''export EASYBUILD_OPTARCH=GENERIC''  and ''eb ... –try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC=“$CC” FC=“$F77”' '' as EasyBuild (eb) runtime option if your are building OpenBLAS ((https://easybuild.readthedocs.io/en/latest/Controlling_compiler_optimization_flags.html#build-environment-vs-hardcoding-in-build-scripts)).
  
 Do you want to know  on what hardware you are? '' gcc -march=native -Q --help=target | awk '/march/{print $2}' '' Do you want to know  on what hardware you are? '' gcc -march=native -Q --help=target | awk '/march/{print $2}' ''
Line 248: Line 288:
 Always consult existing EasyBuild recipes and learn from them. ''grep -ri pythonbundle /easybuild/easybuild/fc31/software/EasyBuild/*/easybuild/easyconfigs'' on a workstation  will return a list of easyconfigs from which you can learn all sorts of tricks that concern the ''pythonbundle'' easyblock. Always consult existing EasyBuild recipes and learn from them. ''grep -ri pythonbundle /easybuild/easybuild/fc31/software/EasyBuild/*/easybuild/easyconfigs'' on a workstation  will return a list of easyconfigs from which you can learn all sorts of tricks that concern the ''pythonbundle'' easyblock.
  
 +==== Environment conflicts ====
 +Should any preset environment variables conflict with the correct execution of a program or should you want to modify the environment at all, you can do this directly in your [[https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_parameters.html#modules-parameters|easyconfig file]]. In the example below, we show how to //unset// the environment variable LD_LIBRARY_PATH upon module loading
 +
 +
 +<code python>
 +#...
 +modtclfooter = "unsetenv LD_LIBRARY_PATH" 
 +modluafooter = 'unsetenv("LD_LIBRARY_PATH")'
 +#...
 +</code>
 ==== Python extensions ==== ==== Python extensions ====
 When you install a python package as an extension, EasyBuild checks if the extension is working properly by //python-importing// the extension name. This means that for extensions such as ''PyYAML'', the building process will fail because no module exists named ''PyYAML''. You can overcome the default behaviour by either giving the extension a custom ''modulename'' When you install a python package as an extension, EasyBuild checks if the extension is working properly by //python-importing// the extension name. This means that for extensions such as ''PyYAML'', the building process will fail because no module exists named ''PyYAML''. You can overcome the default behaviour by either giving the extension a custom ''modulename''
  
-<code>+<code python>
  ('PyYAML', '5.3.1', {  ('PyYAML', '5.3.1', {
        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],
Line 260: Line 310:
  
 or by skipping it altogether (dangerous) or by skipping it altogether (dangerous)
-<code>+<code python>
  ('PyYAML', '5.3.1', {  ('PyYAML', '5.3.1', {
        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],
Line 268: Line 318:
 </code> </code>
  
 +
 +If the module you are installing contains a lot of extensions, its rebuild process could last a long time if we were to re-install the software package and all its extensions from scratch. EasyBuild fortunately has an option that let us install one or more additional extensions without having to reinstall the software package and all extensions from scratch. This saves a considerable amount of time
 +
 +<code>
 +eb my_easyconfig.eb --skip --rebuild
 +</code>
 +
 +Read about [[https://easybuild.readthedocs.io/en/latest/Partial_installations.html|partial installations]].
 ==== GPUs ==== ==== GPUs ====
 If you are building software with GPU support, do so on a workstation/server with GPUs and specify the [[https://developer.nvidia.com/cuda-gpus|CUDA compute capability]] of the attached GPU(s), for instance  ''eb ... --cuda-compute-capabilities=6.0''. If you are building software with GPU support, do so on a workstation/server with GPUs and specify the [[https://developer.nvidia.com/cuda-gpus|CUDA compute capability]] of the attached GPU(s), for instance  ''eb ... --cuda-compute-capabilities=6.0''.
Line 275: Line 333:
 If you want to customise the [[https://www.tensorflow.org/TensorFlow]] building process you must know that  TensorFlow installations occur via [[https://bazel.build/|Bazel]]. This means that a whole lot of customisations can take place at the Bazel level. At building time,  Bazel will [[https://docs.bazel.build/versions/master/guide.html#bazelrc-the-bazel-configuration-file|source]] ''$HOME/.bazelrc'' which you could use to manipulate the installation at your convenience, for instance If you want to customise the [[https://www.tensorflow.org/TensorFlow]] building process you must know that  TensorFlow installations occur via [[https://bazel.build/|Bazel]]. This means that a whole lot of customisations can take place at the Bazel level. At building time,  Bazel will [[https://docs.bazel.build/versions/master/guide.html#bazelrc-the-bazel-configuration-file|source]] ''$HOME/.bazelrc'' which you could use to manipulate the installation at your convenience, for instance
  
-<code>+<code bash>
  
 # cat ~/.bazelrc  # cat ~/.bazelrc 
easybuild_environment.1601883664.txt.gz · Last modified: 2020/10/05 07:41 by lenocil