This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
working_with_python [2018/08/30 07:47] – [Example: numpy with openBLAS] lenocil | working_with_python [2024/04/22 10:51] (current) – jansen | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Working with Python ====== | ====== Working with Python ====== | ||
- | Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python | + | Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python |
- | Notes from a [[https:// | + | ===== Python |
- | ===== Python | + | |
- | Many common python packages | + | Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the '' |
+ | |||
+ | For local python | ||
<code bash> | <code bash> | ||
- | python | + | python3 |
</ | </ | ||
- | or similarly in python v3 | ||
- | <code bash> | ||
- | python3 -m pip list | ||
- | </ | ||
- | If the package | + | For remote python installations -- installations on our software disk -- you must first load a python module and then list all packages in that module |
- | | + | <code bash> |
- | * Install it locally, that is in one of the directories | + | module load Python/ |
+ | which python | ||
+ | / | ||
+ | python -m pip list | ||
+ | </ | ||
+ | |||
+ | If the package you would like to use is not installed at all you have two options: | ||
+ | |||
+ | | ||
+ | * Install it yourself, to one of the directories | ||
The two options are described in detail in the sections below. | The two options are described in detail in the sections below. | ||
- | ===== Global | + | ===== Request a Python Package Installation |
- | If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request | + | If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request |
- | ===== Local installation | + | ===== Install a Python Package yourself |
- | Sometimes, the requested | + | |
- | Generally speaking, a user or developer will want to install and manage python packages in alternative locations (usually outside the system python environment) for one of following reasons: | + | There are instances in which you would like to install a python package |
- The package is not of interest to the majority of users. | - The package is not of interest to the majority of users. | ||
- | | + | |
- | - They want a custom stash of packages, that is not visible to other users. | + | |
- | | + | |
+ | we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process. | ||
+ | |||
+ | :!: The following methods are valid either you use a local (to your workstation/ | ||
- | In this case you can follow one of the methods below. | ||
==== METHOD 1: pip with the `--user' | ==== METHOD 1: pip with the `--user' | ||
- | Python 2.6 introduced the possibility of package installations via a "user scheme" | + | Python 2.6 introduced the possibility of package installations via a "user scheme" |
+ | <code bash> | ||
+ | python -m site --user-base # value of site.USER_BASE | ||
+ | python -m site --user-site # path to your site-packages directory | ||
+ | </ | ||
+ | |||
+ | If the values returned by the command above satisfy you, you can then proceed to install packages in your user-space | ||
<code bash> | <code bash> | ||
pip install --user SomePackage | pip install --user SomePackage | ||
</ | </ | ||
- | To display | + | In the STRW and IL environments, |
<code bash> | <code bash> | ||
- | python[2,3] -m site --user-base | + | export PYTHONUSERBASE=/ |
+ | pip install | ||
</ | </ | ||
+ | will install `SomePackage' | ||
- | and to show the path to your site-packages directory | + | When using the `user' scheme |
+ | |||
+ | * When globally installed packages are on the python | ||
+ | * When globally installed packages are on the python path, and they satisfy the installation requirements, | ||
+ | * pip will not perform a '' | ||
+ | |||
+ | === Method 1 - subsection Incompatible versions === | ||
+ | |||
+ | Unfortunately, | ||
+ | |||
+ | In cases like this, it might be necessary to create a separate python user directory structure for those machines: | ||
+ | |||
+ | Add to your .bashrc something like this: | ||
+ | if [ ! -f / | ||
+ | export PYTHONUSERBASE=$HOME/ | ||
+ | fi | ||
+ | For users of the '' | ||
+ | if (! -f / | ||
+ | setenv PYTHONUSERBASE $HOME/ | ||
+ | endif | ||
+ | |||
+ | And make sure to create that directory ~/ | ||
+ | |||
+ | ==== METHOD 2: venv ==== | ||
+ | '' | ||
<code bash> | <code bash> | ||
- | python[2, | + | $ mkdir / |
+ | $ python3 | ||
</ | </ | ||
+ | to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/ | ||
- | In the STRW and IL environments | + | In the example, we use '' |
+ | |||
+ | The last step before starting to use the newly generated environment is to activate | ||
<code bash> | <code bash> | ||
- | # in bash | + | source / |
- | export PYTHONUSERBASE=/somewhere/I/can/write/to | + | source |
- | pip install --user SomePackage | + | |
</ | </ | ||
- | will install `SomePackage' | ||
- | When using the `user' scheme | + | To acknowledge |
+ | <code bash> | ||
+ | (pymatlab)username@hostname: | ||
+ | </ | ||
+ | to emphasize that you are operating in a virtual environment. To install | ||
+ | <code bash> | ||
+ | pip install | ||
+ | </ | ||
+ | Your virtual environment now should have the same core python | ||
+ | Note that you do NOT use '' | ||
- | * When globally | + | In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages |
- | | + | <code bash> |
- | * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying '' | + | pip freeze > packages.dat |
+ | </ | ||
+ | This can help collaborators | ||
+ | <code bash> | ||
+ | pip install -r packages.dat | ||
+ | </ | ||
+ | When you are done working in a virtual environment deactivate it running | ||
+ | <code bash> | ||
+ | deactivate | ||
+ | </ | ||
+ | At any time, any virtual environment can be destroyed by removing | ||
+ | |||
+ | Note: __System administrators | ||
- | ==== METHOD 2: virtualenv ==== | + | ==== METHOD 2: OBSOLETE: virtualenv |
This guide refers to virtualenv version 12.0.7. | This guide refers to virtualenv version 12.0.7. | ||
Line 78: | Line 142: | ||
to create a virtual environment (folder) called pymatlab. | to create a virtual environment (folder) called pymatlab. | ||
- | Please note that your newly created | + | Python |
- | | + | |
- | The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to your $PATH environment variable. This is done by issuing | + | The last step before starting to use the newly generated environment is to activate it, that is to prepend its '' |
<code bash> | <code bash> | ||
- | source pymatlab/ | + | source pymatlab/ |
- | </ | + | source pymatlab/ |
- | or | + | |
- | < | + | |
- | source pymatlab/ | + | |
</ | </ | ||
- | if you are using csh! | + | |
- | To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt | + | To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt |
<code bash> | <code bash> | ||
(pymatlab)username@hostname: | (pymatlab)username@hostname: | ||
</ | </ | ||
- | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run | + | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment |
<code bash> | <code bash> | ||
pip install | pip install | ||
</ | </ | ||
- | Your virtual environment | + | Your virtual environment now should |
- | In any cases, it is advisable | + | In any cases, it is advisable |
<code bash> | <code bash> | ||
pip freeze > packages.dat | pip freeze > packages.dat | ||
Line 118: | Line 178: | ||
so do not panic if things do not work, just delete your virtual environment and start all over again. | so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
- | Finally, it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option | + | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised |
- | <code bash> | + | |
- | virtualenv -p / | + | |
- | </ | + | |
- | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised | + | |
<code bash> | <code bash> | ||
virtualenv --help | virtualenv --help | ||
</ | </ | ||
- | ==== METHOD 3: easy_install ==== | + | |
+ | ==== METHOD 3: easy_install | ||
Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | ||
- | By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so | + | By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the '' |
- | to install | + | |
<code bash> | <code bash> | ||
easy_install -N --user pymatlab | easy_install -N --user pymatlab | ||
</ | </ | ||
- | This will install pymatlab in ~/.local/lib ready to be imported in your next python session. | + | This will install pymatlab in '' |
- | Note: If you want to install your package in a different location than ~/.local, then set the environment variable | + | If you want to install your package in a different location than '' |
<code bash> | <code bash> | ||
- | export PYTHONUSERBASE=/ | + | export PYTHONUSERBASE=/ |
</ | </ | ||
Line 144: | Line 200: | ||
python -m easy_install --help | python -m easy_install --help | ||
</ | </ | ||
- | ===== Example: how to let python search arbitrary library paths ===== | ||
- | Create/edit | ||
- | '' | + | ==== Migrating packages between python versions ==== |
- | + | Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). | |
- | by appending | + | Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/ |
- | + | - Make a list of installed packages in the old python version: | |
- | < | + | < |
- | echo "/ | + | |
</ | </ | ||
- | + | - Reinstall for the current | |
- | All '' | + | |
- | + | ||
- | ===== Example: how to create a python environment module | + | |
- | + | ||
- | First enable the module package to search also private module directories | + | |
<code bash> | <code bash> | ||
- | module load use.own | + | python3.9 -m pip install --user -r installed.txt |
</ | </ | ||
+ | - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/ | ||
+ | ===== Example: how to let python search arbitrary library paths ===== | ||
+ | For instance for python v2.7 installations, | ||
- | the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files. | + | '' |
- | Let us now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package | + | by appending the path of your choice |
- | < | + | < |
- | export PYTHONUSERBASE=/somewhere/I/can/write/to | + | echo "/my/home/sweet/home/library" |
- | pip install | + | |
- | pip install -I --user SomePackageThatWASInstalledSystemwide | + | |
</ | </ | ||
- | Create a file, say `$HOME/ | + | All ''.pth'' |
- | <code bash> | + | |
- | #%Module 1.0 | + | |
- | # | + | |
- | # | + | |
- | # | + | |
- | prepend-path | + | |
- | prepend-path | + | |
- | </ | + | |
- | and type | + | ===== Example: how to create your own python environment module |
+ | |||
+ | Please read [[easybuild_environment|here]]. | ||
- | <code bash> | ||
- | module load super-module | ||
- | </ | ||
- | and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. | ||
===== Example: numpy with openBLAS ===== | ===== Example: numpy with openBLAS ===== | ||
In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | ||
- | :!: The procedure and paths below will work on any maris node and on the para cluster. | + | :!: The procedure and paths below will work on any maris node. |
<code bash> | <code bash> | ||
Line 252: | Line 292: | ||
====== Bypassing the existing python environment ====== | ====== Bypassing the existing python environment ====== | ||
Occasionally, | Occasionally, | ||
+ | <code bash> | ||
mkdir / | mkdir / | ||
setenv PYTHONPATH / | setenv PYTHONPATH / | ||
+ | </ | ||
The '' | The '' | ||
+ | <code python> | ||
import sys | import sys | ||
import site | import site | ||
Line 262: | Line 305: | ||
# In order to find also eggs and subdirectories, | # In order to find also eggs and subdirectories, | ||
site.addsitedir(mypath, | site.addsitedir(mypath, | ||
+ | </ | ||
+ | |||
+ | ====== Anaconda/ | ||
+ | Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | ||
+ | |||
+ | |||
+ | ====== Jupyter Notebooks ====== | ||
+ | Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get '' | ||
+ | python3 -m ipykernel install --user | ||
+ | Once this command has run successfully, | ||
+ | |||
+ | After starting '' | ||
+ | |||
+ | If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g: | ||
+ | python3 -m ipykernel install --user --name system-python3 | ||