This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
working_with_python [2020/10/05 10:31] – [Example: how to let python search arbitrary library paths] lenocil | working_with_python [2024/04/22 10:51] (current) – jansen | ||
---|---|---|---|
Line 81: | Line 81: | ||
export PYTHONUSERBASE=$HOME/ | export PYTHONUSERBASE=$HOME/ | ||
fi | fi | ||
- | For users of the '' | + | For users of the '' |
if (! -f / | if (! -f / | ||
setenv PYTHONUSERBASE $HOME/ | setenv PYTHONUSERBASE $HOME/ | ||
Line 88: | Line 88: | ||
And make sure to create that directory ~/ | And make sure to create that directory ~/ | ||
- | ==== METHOD 2: virtualenv ==== | + | ==== METHOD 2: venv ==== |
+ | |||
+ | '' | ||
+ | <code bash> | ||
+ | $ mkdir / | ||
+ | $ python3 -m venv / | ||
+ | </ | ||
+ | to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/ | ||
+ | |||
+ | In the example, we use '' | ||
+ | |||
+ | The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/ | ||
+ | <code bash> | ||
+ | source / | ||
+ | source / | ||
+ | </ | ||
+ | |||
+ | To acknowledge the activation of pymatlab, the terminal prompt will be changed to | ||
+ | <code bash> | ||
+ | (pymatlab)username@hostname: | ||
+ | </ | ||
+ | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment | ||
+ | <code bash> | ||
+ | pip install | ||
+ | </ | ||
+ | Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute | ||
+ | Note that you do NOT use '' | ||
+ | |||
+ | In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages | ||
+ | <code bash> | ||
+ | pip freeze > packages.dat | ||
+ | </ | ||
+ | This can help collaborators and fellow developers to reproduce your environment with | ||
+ | <code bash> | ||
+ | pip install -r packages.dat | ||
+ | </ | ||
+ | When you are done working in a virtual environment deactivate it running | ||
+ | <code bash> | ||
+ | deactivate | ||
+ | </ | ||
+ | At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
+ | |||
+ | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation. | ||
+ | |||
+ | ==== METHOD 2: OBSOLETE: virtualenv (python 2.x) ==== | ||
This guide refers to virtualenv version 12.0.7. | This guide refers to virtualenv version 12.0.7. | ||
Line 138: | Line 182: | ||
virtualenv --help | virtualenv --help | ||
</ | </ | ||
+ | |||
==== METHOD 3: easy_install with the `--user' | ==== METHOD 3: easy_install with the `--user' | ||
Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | ||
Line 155: | Line 200: | ||
python -m easy_install --help | python -m easy_install --help | ||
</ | </ | ||
+ | |||
+ | ==== Migrating packages between python versions ==== | ||
+ | Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). | ||
+ | Notes copied from the [[https:// | ||
+ | - Make a list of installed packages in the old python version: | ||
+ | <code bash> | ||
+ | python3.7 -m pip freeze > installed.txt | ||
+ | </ | ||
+ | - Reinstall for the current python version: | ||
+ | <code bash> | ||
+ | python3.9 -m pip install --user -r installed.txt | ||
+ | </ | ||
+ | - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/ | ||
===== Example: how to let python search arbitrary library paths ===== | ===== Example: how to let python search arbitrary library paths ===== | ||
For instance for python v2.7 installations, | For instance for python v2.7 installations, | ||
Line 168: | Line 226: | ||
All '' | All '' | ||
- | ===== Example: how to create | + | ===== Example: how to create |
- | First enable the module package to search also private module directories | + | Please read [[easybuild_environment|here]]. |
- | <code bash> | + | |
- | module load use.own | + | |
- | </ | + | |
- | the line above will create a $HOME/ | ||
- | |||
- | Let us now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package | ||
- | |||
- | <code bash> | ||
- | export PYTHONUSERBASE=/ | ||
- | pip install --user SomePackage | ||
- | pip install -I --user SomePackageThatWASInstalledSystemwide | ||
- | </ | ||
- | |||
- | Create a file, say `$HOME/ | ||
- | <code bash> | ||
- | #%Module 1.0 | ||
- | # | ||
- | # | ||
- | # | ||
- | prepend-path | ||
- | prepend-path | ||
- | </ | ||
- | |||
- | and type | ||
- | |||
- | <code bash> | ||
- | module load super-module | ||
- | </ | ||
- | and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. | ||
===== Example: numpy with openBLAS ===== | ===== Example: numpy with openBLAS ===== | ||
In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | ||
- | :!: The procedure and paths below will work on any maris node and on the para cluster. | + | :!: The procedure and paths below will work on any maris node. |
<code bash> | <code bash> | ||
Line 280: | Line 309: | ||
====== Anaconda/ | ====== Anaconda/ | ||
Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | ||
+ | |||
+ | |||
+ | ====== Jupyter Notebooks ====== | ||
+ | Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get '' | ||
+ | python3 -m ipykernel install --user | ||
+ | Once this command has run successfully, | ||
+ | |||
+ | After starting '' | ||
+ | |||
+ | If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g: | ||
+ | python3 -m ipykernel install --user --name system-python3 | ||
+ | |||
+ |