This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
working_with_python [2018/02/22 12:40] – [pip using the `user' scheme] lenocil | working_with_python [2023/11/06 13:12] (current) – [Jupyter Notebooks] jansen | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Working with Python ====== | ====== Working with Python ====== | ||
- | Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python | + | Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python |
- | Notes from a [[https:// | + | ===== Python |
- | ===== Python | + | |
- | Many common python packages | + | Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the '' |
+ | |||
+ | For local python | ||
+ | |||
+ | <code bash> | ||
+ | python3 -m pip list # or python2 -m pip list | ||
- | < | ||
- | python -m pip list | ||
</ | </ | ||
- | or similarly in python v3 | ||
- | < | ||
- | python3 -m pip list | ||
- | </ | ||
- | If the package | + | For remote python installations -- installations on our software disk -- you must first load a python module and then list all packages in that module |
- | | + | <code bash> |
- | * Install it locally, that is in one of the directories | + | module load Python/ |
+ | which python | ||
+ | / | ||
+ | python -m pip list | ||
+ | </ | ||
+ | |||
+ | If the package you would like to use is not installed at all you have two options: | ||
+ | |||
+ | | ||
+ | * Install it yourself, to one of the directories | ||
The two options are described in detail in the sections below. | The two options are described in detail in the sections below. | ||
- | ===== Global | + | ===== Request a Python Package Installation |
- | If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request | + | If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request |
- | ===== Local installation | + | ===== Install a Python Package yourself |
- | Sometimes, the requested | + | |
- | Generally speaking, a user or developer will want to install and manage python packages in alternative locations (usually outside the system python environment) for one of following reasons: | + | There are instances in which you would like to install a python package |
- The package is not of interest to the majority of users. | - The package is not of interest to the majority of users. | ||
- | | + | |
- | - They want a custom stash of packages, that is not visible to other users. | + | |
- | | + | |
+ | we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process. | ||
+ | |||
+ | :!: The following methods are valid either you use a local (to your workstation/ | ||
- | In this case you can follow one of the methods below. | ||
==== METHOD 1: pip with the `--user' | ==== METHOD 1: pip with the `--user' | ||
- | Python 2.6 introduced the possibility of package installations via a "user scheme" | + | Python 2.6 introduced the possibility of package installations via a "user scheme" |
- | < | + | < |
- | pip install | + | python -m site --user-base # value of site.USER_BASE |
+ | python -m site --user-site # path to your site-packages directory | ||
</ | </ | ||
- | To display | + | If the values returned by the command above satisfy you, you can then proceed to install packages in your user-space |
- | + | < | |
- | < | + | pip install |
- | python[2,3] -m site --user-base | + | |
</ | </ | ||
- | and to show the path to your site-packages directory | + | In the STRW and IL environments, |
- | < | + | < |
- | python -m site --user-site | + | export PYTHONUSERBASE=/ |
- | </ | + | |
- | + | ||
- | In the STRW and IL environments the site.USER_BASE variable defaults to a user's '' | + | |
- | < | + | |
- | # in bash | + | |
- | export PYTHONUSERBASE=/ | + | |
pip install --user SomePackage | pip install --user SomePackage | ||
</ | </ | ||
- | will install `SomePackage' | + | will install `SomePackage' |
When using the `user' scheme to install packages, it is important to note | When using the `user' scheme to install packages, it is important to note | ||
Line 66: | Line 69: | ||
* When globally installed packages are on the python path, and they conflict with the installation requirements, | * When globally installed packages are on the python path, and they conflict with the installation requirements, | ||
* When globally installed packages are on the python path, and they satisfy the installation requirements, | * When globally installed packages are on the python path, and they satisfy the installation requirements, | ||
- | * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying '' | + | * pip will not perform a '' |
+ | |||
+ | === Method 1 - subsection Incompatible versions === | ||
+ | |||
+ | Unfortunately, | ||
+ | |||
+ | In cases like this, it might be necessary to create a separate python user directory structure for those machines: | ||
+ | |||
+ | Add to your .bashrc something like this: | ||
+ | if [ ! -f / | ||
+ | export PYTHONUSERBASE=$HOME/ | ||
+ | fi | ||
+ | For users of the '' | ||
+ | if (! -f / | ||
+ | setenv PYTHONUSERBASE $HOME/ | ||
+ | endif | ||
+ | |||
+ | And make sure to create that directory ~/ | ||
- | ==== virtualenv ==== | + | ==== METHOD 2: virtualenv ==== |
This guide refers to virtualenv version 12.0.7. | This guide refers to virtualenv version 12.0.7. | ||
- | virtualenv is a tool that creates isolated Python environments. A python environment is essentially | + | virtualenv is a tool that creates isolated Python environments. A python environment is essentially |
- | < | + | < |
- | $ mkdir PYTHON_VIRT_ENVS | + | $ mkdir python_virt_envs |
- | $ virtualenv --system-site-packages | + | $ virtualenv --system-site-packages |
</ | </ | ||
- | will create a virtual environment called | + | to create a virtual environment |
- | < | + | |
- | . | + | |
- | ├── bin | + | |
- | ├── include | + | |
- | │ └── python2.7 -> / | + | |
- | ├── lib | + | |
- | │ └── python2.7 | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
- | │ | + | |
+ | Python virtual environments containing specific versions of python can be created using the '' | ||
+ | |||
+ | The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/ | ||
+ | <code bash> | ||
+ | source pymatlab/ | ||
+ | source pymatlab/ | ||
</ | </ | ||
- | NOTE: Despite they are written using capital letters which are usually used for constants or environment variables, PYTHON_VIRT_ENVS and PYMATLAB are both directories. | ||
- | The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to the $PATH environment variable. This is done by issuing | + | To acknowledge |
- | < | + | < |
- | source PYMATLAB/bin/activate | + | (pymatlab)username@hostname: |
- | </ | + | |
- | or | + | |
- | < | + | |
- | source PYMATLAB/ | + | |
</ | </ | ||
- | if you are using csh! | + | to emphasize |
- | To acknowledge the activation of PYMATLAB, virtualenv will change the terminal prompt ($PS1) | + | < |
- | < | + | |
- | (PYMATLAB)username@hostname: | + | |
- | </ | + | |
- | so that you know that you are operating in a virtual environment. To install pymatlab locally (in your virtual environment) run | + | |
- | < | + | |
pip install | pip install | ||
</ | </ | ||
- | Your virtual environment | + | Your virtual environment now should |
- | In any cases, it is advisable | + | In any cases, it is advisable |
- | < | + | < |
pip freeze > packages.dat | pip freeze > packages.dat | ||
</ | </ | ||
This can help collaborators and fellow developers to reproduce your environment with | This can help collaborators and fellow developers to reproduce your environment with | ||
- | < | + | < |
pip install -r packages.dat | pip install -r packages.dat | ||
</ | </ | ||
When you are done working in a virtual environment deactivate it running | When you are done working in a virtual environment deactivate it running | ||
- | < | + | < |
deactivate | deactivate | ||
</ | </ | ||
At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system by executing | At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system by executing | ||
- | < | + | < |
- | rm -rf ~/PYTHON_VIRT_ENVS/PYMATLAB | + | rm -rf ~/python_virt_envs/pymatlab |
</ | </ | ||
so do not panic if things do not work, just delete your virtual environment and start all over again. | so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
- | Finally , it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option | + | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised |
- | < | + | < |
- | virtualenv -p / | + | |
- | </ | + | |
- | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised | + | |
- | < | + | |
virtualenv --help | virtualenv --help | ||
</ | </ | ||
- | ==== easy_install ==== | + | |
+ | ==== METHOD 3: easy_install | ||
Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | ||
- | By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so | + | By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the '' |
- | to install | + | < |
- | < | + | |
easy_install -N --user pymatlab | easy_install -N --user pymatlab | ||
</ | </ | ||
- | This will install pymatlab in ~/.local/lib ready to be imported in your next python session. | + | This will install pymatlab in '' |
- | Note: If you want to install your package in a different location than ~/.local, then set the environment variable | + | If you want to install your package in a different location than '' |
- | < | + | < |
- | export PYTHONUSERBASE=/ | + | export PYTHONUSERBASE=/ |
</ | </ | ||
- | ===== Example: how to let python search arbitrary library paths ===== | ||
- | Create/edit | ||
- | '' | + | Please consult the docs to know more: |
- | by appending the path of your choice, for instance | + | <code bash> |
+ | python -m easy_install --help | ||
+ | </ | ||
- | < | + | ==== Migrating packages between python versions ==== |
- | echo "/my/home/sweet/home/library" | + | Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). |
+ | Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/# | ||
+ | | ||
+ | <code bash> | ||
+ | python3.7 -m pip freeze > installed.txt | ||
+ | </code> | ||
+ | - Reinstall for the current python version: | ||
+ | <code bash> | ||
+ | python3.9 -m pip install --user -r installed.txt | ||
</ | </ | ||
+ | - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/ | ||
+ | ===== Example: how to let python search arbitrary library paths ===== | ||
+ | For instance for python v2.7 installations, | ||
- | All '' | + | '' |
- | ===== Example: how to create a python environment module | + | by appending the path of your choice |
- | First enable the module package to search also private module directories | ||
< | < | ||
- | module load use.own | + | echo "/ |
</ | </ | ||
- | the line above will create a $HOME/ | + | All '' |
- | Let us now install some packages | + | ===== Example: how to create your own python environment module |
- | < | + | Please read [[easybuild_environment|here]]. |
- | export PYTHONUSERBASE=/ | + | |
- | pip install --user SomePackage | + | |
- | pip install -I --user SomePackageThatWASInstalledSystemwide | + | |
- | </ | + | |
- | Create a file, say `$HOME/ | ||
- | < | ||
- | #%Module 1.0 | ||
- | # | ||
- | # | ||
- | # | ||
- | prepend-path | ||
- | prepend-path | ||
- | </ | ||
- | |||
- | and type | ||
- | |||
- | < | ||
- | module load super-module | ||
- | </ | ||
- | and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. | ||
===== Example: numpy with openBLAS ===== | ===== Example: numpy with openBLAS ===== | ||
In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | ||
- | :!: The procedure and paths below will work on any maris node and on the para cluster. | + | :!: The procedure and paths below will work on any maris node. |
- | < | + | < |
virtualenv py2_numpy_openBLAS | virtualenv py2_numpy_openBLAS | ||
source py2_numpy_openBLAS/ | source py2_numpy_openBLAS/ | ||
Line 253: | Line 202: | ||
</ | </ | ||
Edit '' | Edit '' | ||
- | < | + | < |
[openblas] | [openblas] | ||
libraries = openblas | libraries = openblas | ||
Line 261: | Line 210: | ||
</ | </ | ||
then install numpy | then install numpy | ||
- | < | + | < |
python setup.py install | python setup.py install | ||
</ | </ | ||
If the installation is going smoothly you should see | If the installation is going smoothly you should see | ||
- | < | + | < |
.... | .... | ||
Line 281: | Line 230: | ||
</ | </ | ||
Now that numpy is installed you could also install scipy, for instance | Now that numpy is installed you could also install scipy, for instance | ||
- | < | + | < |
pip install scipy | pip install scipy | ||
</ | </ | ||
Line 299: | Line 248: | ||
====== Bypassing the existing python environment ====== | ====== Bypassing the existing python environment ====== | ||
Occasionally, | Occasionally, | ||
+ | <code bash> | ||
mkdir / | mkdir / | ||
setenv PYTHONPATH / | setenv PYTHONPATH / | ||
+ | </ | ||
The '' | The '' | ||
+ | <code python> | ||
import sys | import sys | ||
import site | import site | ||
Line 309: | Line 261: | ||
# In order to find also eggs and subdirectories, | # In order to find also eggs and subdirectories, | ||
site.addsitedir(mypath, | site.addsitedir(mypath, | ||
+ | </ | ||
+ | |||
+ | ====== Anaconda/ | ||
+ | Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | ||
+ | |||
+ | |||
+ | ====== Jupyter Notebooks ====== | ||
+ | Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get '' | ||
+ | python3 -m ipykernel install --user | ||
+ | Once this command has run successfully, | ||
+ | |||
+ | After starting '' | ||
+ | |||
+ | If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g: | ||
+ | python3 -m ipykernel install --user --name system-python3 | ||