This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
working_with_python [2018/02/22 12:54] – [easy_install] lenocil | working_with_python [2024/04/22 10:51] (current) – jansen | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Working with Python ====== | ====== Working with Python ====== | ||
- | Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python | + | Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python |
- | Notes from a [[https:// | + | ===== Python |
- | ===== Python | + | |
- | Many common python packages | + | Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the '' |
- | < | + | For local python installations, |
+ | |||
+ | < | ||
+ | python3 -m pip list # or python2 -m pip list | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | For remote python installations -- installations on our software disk -- you must first load a python module and then list all packages in that module | ||
+ | |||
+ | <code bash> | ||
+ | module load Python/ | ||
+ | which python | ||
+ | / | ||
python -m pip list | python -m pip list | ||
</ | </ | ||
- | or similarly in python v3 | ||
- | < | ||
- | python3 -m pip list | ||
- | </ | ||
+ | If the package you would like to use is not installed at all you have two options: | ||
- | If the package you would like to use is not installed you have two options: | + | * Request |
- | + | * Install it yourself, to one of the directories | |
- | * Contact | + | |
- | * Install it locally, that is in one of the directories | + | |
The two options are described in detail in the sections below. | The two options are described in detail in the sections below. | ||
- | ===== Global | + | ===== Request a Python Package Installation |
- | If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request | + | If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request |
- | ===== Local installation | + | ===== Install a Python Package yourself |
- | Sometimes, the requested | + | |
- | Generally speaking, a user or developer will want to install and manage python packages in alternative locations (usually outside the system python environment) for one of following reasons: | + | There are instances in which you would like to install a python package |
- The package is not of interest to the majority of users. | - The package is not of interest to the majority of users. | ||
- | | + | |
- | - They want a custom stash of packages, that is not visible to other users. | + | |
- | | + | |
- | In this case you can follow one of the methods below. | + | we advise |
- | ==== METHOD 1: pip with the `--user' | + | |
- | Python 2.6 introduced the possibility of package installations via a "user scheme" | + | :!: The following methods are valid either you use a local (to your workstation/ |
- | < | + | ==== METHOD 1: pip with the `--user' option ==== |
- | pip install | + | |
- | </ | + | |
- | To display | + | Python 2.6 introduced |
- | < | + | < |
- | python[2,3] -m site --user-base | + | python -m site --user-base |
+ | python -m site --user-site # path to your site-packages directory | ||
</ | </ | ||
- | and to show the path to your site-packages directory | + | If the values returned by the command above satisfy you, you can then proceed |
- | + | < | |
- | < | + | pip install |
- | python[2,3] -m site --user-site | + | |
</ | </ | ||
- | In the STRW and IL environments | + | In the STRW and IL environments, site.USER_BASE |
- | < | + | |
- | # in bash | + | < |
- | export PYTHONUSERBASE=/ | + | export PYTHONUSERBASE=/ |
pip install --user SomePackage | pip install --user SomePackage | ||
</ | </ | ||
- | will install `SomePackage' | + | will install `SomePackage' |
When using the `user' scheme to install packages, it is important to note | When using the `user' scheme to install packages, it is important to note | ||
Line 66: | Line 69: | ||
* When globally installed packages are on the python path, and they conflict with the installation requirements, | * When globally installed packages are on the python path, and they conflict with the installation requirements, | ||
* When globally installed packages are on the python path, and they satisfy the installation requirements, | * When globally installed packages are on the python path, and they satisfy the installation requirements, | ||
- | * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying '' | + | * pip will not perform a '' |
- | ==== METHOD 2: virtualenv ==== | + | === Method 1 - subsection Incompatible versions === |
+ | |||
+ | Unfortunately, | ||
+ | |||
+ | In cases like this, it might be necessary to create a separate python user directory structure for those machines: | ||
+ | |||
+ | Add to your .bashrc something like this: | ||
+ | if [ ! -f / | ||
+ | export PYTHONUSERBASE=$HOME/ | ||
+ | fi | ||
+ | For users of the '' | ||
+ | if (! -f / | ||
+ | setenv PYTHONUSERBASE $HOME/ | ||
+ | endif | ||
+ | |||
+ | And make sure to create that directory ~/ | ||
+ | |||
+ | ==== METHOD 2: venv ==== | ||
+ | |||
+ | '' | ||
+ | <code bash> | ||
+ | $ mkdir / | ||
+ | $ python3 -m venv / | ||
+ | </ | ||
+ | to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/ | ||
+ | |||
+ | In the example, we use '' | ||
+ | |||
+ | The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/ | ||
+ | <code bash> | ||
+ | source / | ||
+ | source / | ||
+ | </ | ||
+ | |||
+ | To acknowledge the activation of pymatlab, the terminal prompt will be changed to | ||
+ | <code bash> | ||
+ | (pymatlab)username@hostname: | ||
+ | </ | ||
+ | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment | ||
+ | <code bash> | ||
+ | pip install | ||
+ | </ | ||
+ | Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute | ||
+ | Note that you do NOT use '' | ||
+ | |||
+ | In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages | ||
+ | <code bash> | ||
+ | pip freeze > packages.dat | ||
+ | </ | ||
+ | This can help collaborators and fellow developers to reproduce your environment with | ||
+ | <code bash> | ||
+ | pip install -r packages.dat | ||
+ | </ | ||
+ | When you are done working in a virtual environment deactivate it running | ||
+ | <code bash> | ||
+ | deactivate | ||
+ | </ | ||
+ | At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
+ | |||
+ | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation. | ||
+ | |||
+ | ==== METHOD 2: OBSOLETE: virtualenv (python 2.x) ==== | ||
This guide refers to virtualenv version 12.0.7. | This guide refers to virtualenv version 12.0.7. | ||
virtualenv is a tool that creates isolated Python environments. A python environment is essentially | virtualenv is a tool that creates isolated Python environments. A python environment is essentially | ||
- | < | + | < |
$ mkdir python_virt_envs && cd python_virt_envs | $ mkdir python_virt_envs && cd python_virt_envs | ||
$ virtualenv --system-site-packages pymatlab | $ virtualenv --system-site-packages pymatlab | ||
Line 78: | Line 142: | ||
to create a virtual environment (folder) called pymatlab. | to create a virtual environment (folder) called pymatlab. | ||
- | Please note that your newly created | + | Python |
- | | + | |
- | The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to your $PATH environment variable. This is done by issuing | + | The last step before starting to use the newly generated environment is to activate it, that is to prepend its '' |
- | < | + | < |
- | source pymatlab/ | + | source pymatlab/ |
- | </ | + | source pymatlab/ |
- | or | + | |
- | < | + | |
- | source pymatlab/ | + | |
</ | </ | ||
- | if you are using csh! | + | |
- | To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt | + | To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt |
- | < | + | < |
(pymatlab)username@hostname: | (pymatlab)username@hostname: | ||
</ | </ | ||
- | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run | + | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment |
- | < | + | < |
pip install | pip install | ||
</ | </ | ||
- | Your virtual environment | + | Your virtual environment now should |
- | In any cases, it is advisable | + | In any cases, it is advisable |
- | < | + | < |
pip freeze > packages.dat | pip freeze > packages.dat | ||
</ | </ | ||
This can help collaborators and fellow developers to reproduce your environment with | This can help collaborators and fellow developers to reproduce your environment with | ||
- | < | + | < |
pip install -r packages.dat | pip install -r packages.dat | ||
</ | </ | ||
When you are done working in a virtual environment deactivate it running | When you are done working in a virtual environment deactivate it running | ||
- | < | + | < |
deactivate | deactivate | ||
</ | </ | ||
At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system by executing | At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system by executing | ||
- | < | + | < |
rm -rf ~/ | rm -rf ~/ | ||
</ | </ | ||
so do not panic if things do not work, just delete your virtual environment and start all over again. | so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
- | Finally, it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option | + | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised |
- | < | + | < |
- | virtualenv -p / | + | |
- | </ | + | |
- | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised | + | |
- | < | + | |
virtualenv --help | virtualenv --help | ||
</ | </ | ||
- | ==== METHOD 3: easy_install ==== | + | |
+ | ==== METHOD 3: easy_install | ||
Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | ||
- | By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so | + | By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the '' |
- | to install | + | < |
- | < | + | |
easy_install -N --user pymatlab | easy_install -N --user pymatlab | ||
</ | </ | ||
- | This will install pymatlab in ~/.local/lib ready to be imported in your next python session. | + | This will install pymatlab in '' |
- | Note: If you want to install your package in a different location than ~/.local, then set the environment variable | + | If you want to install your package in a different location than '' |
- | < | + | < |
- | export PYTHONUSERBASE=/ | + | export PYTHONUSERBASE=/ |
</ | </ | ||
- | ===== Example: how to let python search arbitrary library paths ===== | ||
- | Create/edit | ||
- | '' | + | Please consult the docs to know more: |
- | by appending the path of your choice, for instance | + | < |
- | + | python | |
- | < | + | |
- | echo "/ | + | |
</ | </ | ||
- | All '' | + | ==== Migrating packages between |
- | + | Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). | |
- | ===== Example: how to create a python | + | Notes copied from the [[https:// |
- | + | - Make a list of installed packages in the old python version: | |
- | First enable | + | < |
- | < | + | |
- | module load use.own | + | </ |
+ | - Reinstall for the current python version: | ||
+ | <code bash> | ||
+ | python3.9 -m pip install --user -r installed.txt | ||
</ | </ | ||
+ | - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/ | ||
+ | ===== Example: how to let python search arbitrary library paths ===== | ||
+ | For instance for python v2.7 installations, | ||
- | the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files. | + | '' |
- | Let us now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package | + | by appending the path of your choice |
< | < | ||
- | export PYTHONUSERBASE=/somewhere/I/can/write/to | + | echo "/my/home/sweet/home/library" |
- | pip install | + | |
- | pip install -I --user SomePackageThatWASInstalledSystemwide | + | |
</ | </ | ||
- | Create a file, say `$HOME/ | + | All ''.pth'' |
- | < | + | |
- | #%Module 1.0 | + | |
- | # | + | |
- | # | + | |
- | # | + | |
- | prepend-path | + | |
- | prepend-path | + | |
- | </ | + | |
- | and type | + | ===== Example: how to create your own python environment module |
+ | |||
+ | Please read [[easybuild_environment|here]]. | ||
- | < | ||
- | module load super-module | ||
- | </ | ||
- | and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. | ||
===== Example: numpy with openBLAS ===== | ===== Example: numpy with openBLAS ===== | ||
In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | ||
- | :!: The procedure and paths below will work on any maris node and on the para cluster. | + | :!: The procedure and paths below will work on any maris node. |
- | < | + | < |
virtualenv py2_numpy_openBLAS | virtualenv py2_numpy_openBLAS | ||
source py2_numpy_openBLAS/ | source py2_numpy_openBLAS/ | ||
Line 200: | Line 246: | ||
</ | </ | ||
Edit '' | Edit '' | ||
- | < | + | < |
[openblas] | [openblas] | ||
libraries = openblas | libraries = openblas | ||
Line 208: | Line 254: | ||
</ | </ | ||
then install numpy | then install numpy | ||
- | < | + | < |
python setup.py install | python setup.py install | ||
</ | </ | ||
If the installation is going smoothly you should see | If the installation is going smoothly you should see | ||
- | < | + | < |
.... | .... | ||
Line 228: | Line 274: | ||
</ | </ | ||
Now that numpy is installed you could also install scipy, for instance | Now that numpy is installed you could also install scipy, for instance | ||
- | < | + | < |
pip install scipy | pip install scipy | ||
</ | </ | ||
Line 246: | Line 292: | ||
====== Bypassing the existing python environment ====== | ====== Bypassing the existing python environment ====== | ||
Occasionally, | Occasionally, | ||
+ | <code bash> | ||
mkdir / | mkdir / | ||
setenv PYTHONPATH / | setenv PYTHONPATH / | ||
+ | </ | ||
The '' | The '' | ||
+ | <code python> | ||
import sys | import sys | ||
import site | import site | ||
Line 256: | Line 305: | ||
# In order to find also eggs and subdirectories, | # In order to find also eggs and subdirectories, | ||
site.addsitedir(mypath, | site.addsitedir(mypath, | ||
+ | </ | ||
+ | |||
+ | ====== Anaconda/ | ||
+ | Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | ||
+ | |||
+ | |||
+ | ====== Jupyter Notebooks ====== | ||
+ | Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get '' | ||
+ | python3 -m ipykernel install --user | ||
+ | Once this command has run successfully, | ||
+ | |||
+ | After starting '' | ||
+ | |||
+ | If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g: | ||
+ | python3 -m ipykernel install --user --name system-python3 | ||