User Tools

Site Tools


easybuild_environment

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
easybuild_environment [2020/10/05 07:49] – [TensorFlow] lenocileasybuild_environment [2021/02/08 16:37] (current) jansen
Line 9: Line 9:
  
 ^Where ^OS^Mount Point^Remote Location^Protocol^ ^Where ^OS^Mount Point^Remote Location^Protocol^
-|Workstations \\ & Servers|Fedora 31| /easybuild/easybuild/fc31| Software Server| NFS| +|Workstations \\ & Servers|Fedora 33| /easybuild/easybuild/fc31| Software Server| NFS| 
 | ::: | RHEL7| /easybuild/easybuild/el7| Software Server| NFS|  | ::: | RHEL7| /easybuild/easybuild/el7| Software Server| NFS| 
-| ::: | RHEL7/Lustre| /easybuild/easybuild/el7_lustre| Software Server| NFS|  
 | ::: | RHEL8| /easybuild/easybuild/el8| Software Server| NFS|  | ::: | RHEL8| /easybuild/easybuild/el8| Software Server| NFS| 
 | Xmaris| CentOS 7| /marisdata/easybuild| Marisdata | NFS| | Xmaris| CentOS 7| /marisdata/easybuild| Marisdata | NFS|
Line 46: Line 45:
 Failing to do so, can result in the production of __non-portable__ softwares. On the other hand, we advise you build hardware-bound softwares in all cases in which execution performance is paramount. Failing to do so, can result in the production of __non-portable__ softwares. On the other hand, we advise you build hardware-bound softwares in all cases in which execution performance is paramount.
  
-Put particular attention if you are planning  to build  //OpenBLAS// via EasyBuild. In this case defining EASYBUILD_OPTARCH=GENERIC __is not sufficient__ to produce portable software (CPU independent). Use both ''export EASYBUILD_OPTARCH=GENERIC'' in your setup and ''--try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77"' '' as EasyBuild (''eb'') runtime option.+Put particular attention if you are planning  to build  //OpenBLAS// via EasyBuild. In this case defining EASYBUILD_OPTARCH=GENERIC __is not sufficient__ to produce portable software (CPU independent). Use both ''export EASYBUILD_OPTARCH=GENERIC'' in your setup and ''--try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77"' '' as EasyBuild (''eb'') runtime option. If your compilation fails you can try '' --try-amend=buildopts='TARGET=CORE2 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77"' '' instead.
  
  
Line 226: Line 225:
  
 class AnacondaLeonardo(EB_Anaconda): class AnacondaLeonardo(EB_Anaconda):
- 
     """Support for building/installing Anaconda and Miniconda."""     """Support for building/installing Anaconda and Miniconda."""
     def make_module_extra(self):     def make_module_extra(self):
Line 245: Line 243:
  
 <code python> <code python>
 +#cat Miniconda2-4.3.21_mod.eb
 +easyblock = 'AnacondaLeonardo'
 +
 +name = 'Miniconda2'
 +version = '4.3.21'
 +
 +homepage = 'https://docs.conda.io/en/latest/miniconda.html'
 +
 +description = """Miniconda is a free minimal installer for conda. It is a small,
 +bootstrap version of Anaconda that includes only conda, Python, the packages they
 +depend on, and a small number of other useful packages.
 +
 +A warning message will be printed on the screen upon module loading.
 +
 +Author: leonardo
 +
 +"""
 +
 +toolchain = SYSTEM
 +
 +source_urls = ['https://repo.anaconda.com/miniconda/']
 +sources = ['%(name)s-%(version)s-Linux-x86_64.sh']
 +checksums = ['5097d5ec484a345c8785655113b19b88bfcd69af5f25a36c832ce498f02ea052']
 +
 +moduleclass = 'lang'
 +
 </code> </code>
 +
 +And install it via ''eb -r /path/to/easyconfig/Miniconda2-4.3.21_mod.eb --include-easyblocks=/path/to/my/easyblocks/*py''.
 +
 +Finally, do not forget to read the official [[https://easybuild.readthedocs.io/en/latest/Implementing-easyblocks.html|easyblocks documentation]]. 
 +
  
 ===== EasyBuild Tips ===== ===== EasyBuild Tips =====
Line 253: Line 282:
  
 ==== Heterogeneous environments ==== ==== Heterogeneous environments ====
-If you are planning to use your EasyBuild-built software on a variety of CPUs, do not forget to instruct EasyBuild to do so via ''export EASYBUILD_OPTARCH=GENERIC''  and ''eb ... –try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC=“$CC” FC=“$F77”' '' as EasyBuild (eb) runtime option. +If you are planning to use your EasyBuild-built software on a variety of CPUs, do not forget to instruct EasyBuild to do so via ''export EASYBUILD_OPTARCH=GENERIC''  and ''eb ... –try-amend=buildopts='TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC=“$CC” FC=“$F77”' '' as EasyBuild (eb) runtime option if your are building OpenBLAS ((https://easybuild.readthedocs.io/en/latest/Controlling_compiler_optimization_flags.html#build-environment-vs-hardcoding-in-build-scripts)).
  
 Do you want to know  on what hardware you are? '' gcc -march=native -Q --help=target | awk '/march/{print $2}' '' Do you want to know  on what hardware you are? '' gcc -march=native -Q --help=target | awk '/march/{print $2}' ''
Line 259: Line 288:
 Always consult existing EasyBuild recipes and learn from them. ''grep -ri pythonbundle /easybuild/easybuild/fc31/software/EasyBuild/*/easybuild/easyconfigs'' on a workstation  will return a list of easyconfigs from which you can learn all sorts of tricks that concern the ''pythonbundle'' easyblock. Always consult existing EasyBuild recipes and learn from them. ''grep -ri pythonbundle /easybuild/easybuild/fc31/software/EasyBuild/*/easybuild/easyconfigs'' on a workstation  will return a list of easyconfigs from which you can learn all sorts of tricks that concern the ''pythonbundle'' easyblock.
  
 +==== Environment conflicts ====
 +Should any preset environment variables conflict with the correct execution of a program or should you want to modify the environment at all, you can do this directly in your [[https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_parameters.html#modules-parameters|easyconfig file]]. In the example below, we show how to //unset// the environment variable LD_LIBRARY_PATH upon module loading
 +
 +
 +<code python>
 +#...
 +modtclfooter = "unsetenv LD_LIBRARY_PATH" 
 +modluafooter = 'unsetenv("LD_LIBRARY_PATH")'
 +#...
 +</code>
 ==== Python extensions ==== ==== Python extensions ====
 When you install a python package as an extension, EasyBuild checks if the extension is working properly by //python-importing// the extension name. This means that for extensions such as ''PyYAML'', the building process will fail because no module exists named ''PyYAML''. You can overcome the default behaviour by either giving the extension a custom ''modulename'' When you install a python package as an extension, EasyBuild checks if the extension is working properly by //python-importing// the extension name. This means that for extensions such as ''PyYAML'', the building process will fail because no module exists named ''PyYAML''. You can overcome the default behaviour by either giving the extension a custom ''modulename''
  
-<code>+<code python>
  ('PyYAML', '5.3.1', {  ('PyYAML', '5.3.1', {
        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],
Line 271: Line 310:
  
 or by skipping it altogether (dangerous) or by skipping it altogether (dangerous)
-<code>+<code python>
  ('PyYAML', '5.3.1', {  ('PyYAML', '5.3.1', {
        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],        'checksums': ['b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d'],
Line 279: Line 318:
 </code> </code>
  
 +
 +If the module you are installing contains a lot of extensions, its rebuild process could last a long time if we were to re-install the software package and all its extensions from scratch. EasyBuild fortunately has an option that let us install one or more additional extensions without having to reinstall the software package and all extensions from scratch. This saves a considerable amount of time
 +
 +<code>
 +eb my_easyconfig.eb --skip --rebuild
 +</code>
 +
 +Read about [[https://easybuild.readthedocs.io/en/latest/Partial_installations.html|partial installations]].
 ==== GPUs ==== ==== GPUs ====
 If you are building software with GPU support, do so on a workstation/server with GPUs and specify the [[https://developer.nvidia.com/cuda-gpus|CUDA compute capability]] of the attached GPU(s), for instance  ''eb ... --cuda-compute-capabilities=6.0''. If you are building software with GPU support, do so on a workstation/server with GPUs and specify the [[https://developer.nvidia.com/cuda-gpus|CUDA compute capability]] of the attached GPU(s), for instance  ''eb ... --cuda-compute-capabilities=6.0''.
easybuild_environment.1601884154.txt.gz · Last modified: 2020/10/05 07:49 by lenocil