This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
easybuild_environment [2020/10/02 14:50] – [Working with EasyBuild] lenocil | easybuild_environment [2021/02/08 16:37] (current) – jansen | ||
---|---|---|---|
Line 2: | Line 2: | ||
{{ https:// | {{ https:// | ||
[[https:// | [[https:// | ||
- | Because EasyBuild is a powerful framework that enables everybody to build and install complex softwares with ease, we give here only a few tips should you want to build and install custom software via EasyBuild in your private software stack. | + | Because EasyBuild is a powerful framework that enables everybody to build and install complex softwares with ease, we encourage users to use it in their development workflows. Here we give a few tips should you decide |
===== Common EasyBuild Environments ===== | ===== Common EasyBuild Environments ===== | ||
Line 9: | Line 9: | ||
^Where ^OS^Mount Point^Remote Location^Protocol^ | ^Where ^OS^Mount Point^Remote Location^Protocol^ | ||
- | |Workstations \\ & Servers|Fedora | + | |Workstations \\ & Servers|Fedora |
| ::: | RHEL7| / | | ::: | RHEL7| / | ||
- | | ::: | RHEL7/ | ||
| ::: | RHEL8| / | | ::: | RHEL8| / | ||
| Xmaris| CentOS 7| / | | Xmaris| CentOS 7| / | ||
Line 30: | Line 29: | ||
First of all you need to setup your EasyBuild development stack. This will be hosted in a location on your server/ | First of all you need to setup your EasyBuild development stack. This will be hosted in a location on your server/ | ||
- | < | + | < |
module load EasyBuild | module load EasyBuild | ||
Line 40: | Line 39: | ||
If you want to use the softwares installed in your private stack on a variety of hardwares (workstations and servers) you must also instruct EasyBuild to build hardware-independent executables | If you want to use the softwares installed in your private stack on a variety of hardwares (workstations and servers) you must also instruct EasyBuild to build hardware-independent executables | ||
- | < | + | < |
export EASYBUILD_OPTARCH=GENERIC | export EASYBUILD_OPTARCH=GENERIC | ||
</ | </ | ||
Line 46: | Line 45: | ||
Failing to do so, can result in the production of __non-portable__ softwares. On the other hand, we advise you build hardware-bound softwares in all cases in which execution performance is paramount. | Failing to do so, can result in the production of __non-portable__ softwares. On the other hand, we advise you build hardware-bound softwares in all cases in which execution performance is paramount. | ||
- | Put particular attention if you are planning | + | Put particular attention if you are planning |
Line 78: | Line 77: | ||
At this point you can use your newly installed '' | At this point you can use your newly installed '' | ||
- | < | + | < |
module load Miniconda3 | module load Miniconda3 | ||
which conda | which conda | ||
Line 149: | Line 148: | ||
As you might have inferred, in all cases in which EasyBuild does not provide in its repos an easyconfig for the software you would like to install, you will have to pick up the //right// easyblock and write an //ad-hoc// easyconfig file which uses the chosen easyblock with appropriate parameters. This task is not simple. To make things more difficult, there could be cases in which you will have to write your own easyblock from scratch! Here follows an example easyconfig that will install a combo (bundle) of python packages all available in a single module. | As you might have inferred, in all cases in which EasyBuild does not provide in its repos an easyconfig for the software you would like to install, you will have to pick up the //right// easyblock and write an //ad-hoc// easyconfig file which uses the chosen easyblock with appropriate parameters. This task is not simple. To make things more difficult, there could be cases in which you will have to write your own easyblock from scratch! Here follows an example easyconfig that will install a combo (bundle) of python packages all available in a single module. | ||
- | < | + | < |
cat Quantum-TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb | cat Quantum-TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb | ||
# | # | ||
Line 217: | Line 216: | ||
=== Write a custom easyblock === | === Write a custom easyblock === | ||
- | Coming soon | + | In the unlikely event that no suitable easyblocks fit your software installation procedure, you will have to implement your own easyblock. |
+ | |||
+ | Here follows a trivial -- perhaps not very useful -- example in which we create an easyblock that implements the following function: it prints a screen message when its corresponding module is loaded. This example should get you started and give you an idea of how easyblocks work. | ||
+ | |||
+ | <code python> | ||
+ | # cat anacondaleonardo.py | ||
+ | from easybuild.easyblocks.a.anaconda import EB_Anaconda | ||
+ | |||
+ | class AnacondaLeonardo(EB_Anaconda): | ||
+ | """ | ||
+ | def make_module_extra(self): | ||
+ | txt = super(AnacondaLeonardo, | ||
+ | txt += self.module_generator.msg_on_load(" | ||
+ | return txt | ||
+ | |||
+ | </ | ||
+ | |||
+ | Notice the following | ||
+ | |||
+ | * easyblocks are written in python | ||
+ | * we are modifying the behaviour of the EasyBuild-provided '' | ||
+ | * this easyblock can therefore be used to install the anaconda software | ||
+ | * we override the function '' | ||
+ | |||
+ | Now create en easyconfig file that uses the newly created easyblock | ||
+ | |||
+ | <code python> | ||
+ | #cat Miniconda2-4.3.21_mod.eb | ||
+ | easyblock = ' | ||
+ | |||
+ | name = ' | ||
+ | version = ' | ||
+ | |||
+ | homepage = ' | ||
+ | |||
+ | description = """ | ||
+ | bootstrap version of Anaconda that includes only conda, Python, the packages they | ||
+ | depend on, and a small number of other useful packages. | ||
+ | |||
+ | A warning message will be printed on the screen upon module loading. | ||
+ | |||
+ | Author: leonardo | ||
+ | |||
+ | """ | ||
+ | |||
+ | toolchain = SYSTEM | ||
+ | |||
+ | source_urls = [' | ||
+ | sources = [' | ||
+ | checksums = [' | ||
+ | |||
+ | moduleclass = ' | ||
+ | |||
+ | </ | ||
+ | |||
+ | And install it via '' | ||
+ | |||
+ | Finally, do not forget to read the official [[https:// | ||
===== EasyBuild Tips ===== | ===== EasyBuild Tips ===== | ||
Line 225: | Line 282: | ||
==== Heterogeneous environments ==== | ==== Heterogeneous environments ==== | ||
- | If you are planning to use your EasyBuild-built software on a variety of CPUs, do not forget to instruct EasyBuild to do so via '' | + | If you are planning to use your EasyBuild-built software on a variety of CPUs, do not forget to instruct EasyBuild to do so via '' |
Do you want to know on what hardware you are? '' | Do you want to know on what hardware you are? '' | ||
Line 231: | Line 288: | ||
Always consult existing EasyBuild recipes and learn from them. '' | Always consult existing EasyBuild recipes and learn from them. '' | ||
+ | ==== Environment conflicts ==== | ||
+ | Should any preset environment variables conflict with the correct execution of a program or should you want to modify the environment at all, you can do this directly in your [[https:// | ||
+ | |||
+ | |||
+ | <code python> | ||
+ | #... | ||
+ | modtclfooter = " | ||
+ | modluafooter = ' | ||
+ | #... | ||
+ | </ | ||
==== Python extensions ==== | ==== Python extensions ==== | ||
When you install a python package as an extension, EasyBuild checks if the extension is working properly by // | When you install a python package as an extension, EasyBuild checks if the extension is working properly by // | ||
- | < | + | < |
| | ||
' | ' | ||
Line 243: | Line 310: | ||
or by skipping it altogether (dangerous) | or by skipping it altogether (dangerous) | ||
- | < | + | < |
| | ||
' | ' | ||
Line 251: | Line 318: | ||
</ | </ | ||
+ | |||
+ | If the module you are installing contains a lot of extensions, its rebuild process could last a long time if we were to re-install the software package and all its extensions from scratch. EasyBuild fortunately has an option that let us install one or more additional extensions without having to reinstall the software package and all extensions from scratch. This saves a considerable amount of time | ||
+ | |||
+ | < | ||
+ | eb my_easyconfig.eb --skip --rebuild | ||
+ | </ | ||
+ | |||
+ | Read about [[https:// | ||
==== GPUs ==== | ==== GPUs ==== | ||
If you are building software with GPU support, do so on a workstation/ | If you are building software with GPU support, do so on a workstation/ | ||
Line 258: | Line 333: | ||
If you want to customise the [[https:// | If you want to customise the [[https:// | ||
- | < | + | < |
# cat ~/ | # cat ~/ |