This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
working_with_python [2020/10/05 09:25] – [METHOD 1: pip with the `--user' option] lenocil | working_with_python [2024/04/22 10:51] (current) – jansen | ||
---|---|---|---|
Line 81: | Line 81: | ||
export PYTHONUSERBASE=$HOME/ | export PYTHONUSERBASE=$HOME/ | ||
fi | fi | ||
- | For users of the '' | + | For users of the '' |
if (! -f / | if (! -f / | ||
setenv PYTHONUSERBASE $HOME/ | setenv PYTHONUSERBASE $HOME/ | ||
Line 88: | Line 88: | ||
And make sure to create that directory ~/ | And make sure to create that directory ~/ | ||
- | ==== METHOD 2: virtualenv ==== | + | ==== METHOD 2: venv ==== |
+ | |||
+ | '' | ||
+ | <code bash> | ||
+ | $ mkdir / | ||
+ | $ python3 -m venv / | ||
+ | </ | ||
+ | to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/ | ||
+ | |||
+ | In the example, we use '' | ||
+ | |||
+ | The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/ | ||
+ | <code bash> | ||
+ | source / | ||
+ | source / | ||
+ | </ | ||
+ | |||
+ | To acknowledge the activation of pymatlab, the terminal prompt will be changed to | ||
+ | <code bash> | ||
+ | (pymatlab)username@hostname: | ||
+ | </ | ||
+ | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment | ||
+ | <code bash> | ||
+ | pip install | ||
+ | </ | ||
+ | Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute | ||
+ | Note that you do NOT use '' | ||
+ | |||
+ | In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages | ||
+ | <code bash> | ||
+ | pip freeze > packages.dat | ||
+ | </ | ||
+ | This can help collaborators and fellow developers to reproduce your environment with | ||
+ | <code bash> | ||
+ | pip install -r packages.dat | ||
+ | </ | ||
+ | When you are done working in a virtual environment deactivate it running | ||
+ | <code bash> | ||
+ | deactivate | ||
+ | </ | ||
+ | At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
+ | |||
+ | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation. | ||
+ | |||
+ | ==== METHOD 2: OBSOLETE: virtualenv (python 2.x) ==== | ||
This guide refers to virtualenv version 12.0.7. | This guide refers to virtualenv version 12.0.7. | ||
Line 98: | Line 142: | ||
to create a virtual environment (folder) called pymatlab. | to create a virtual environment (folder) called pymatlab. | ||
- | Please note that your newly created | + | Python |
- | | + | |
- | The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to your $PATH environment variable. This is done by issuing | + | The last step before starting to use the newly generated environment is to activate it, that is to prepend its '' |
<code bash> | <code bash> | ||
- | source pymatlab/ | + | source pymatlab/ |
- | </ | + | source pymatlab/ |
- | or | + | |
- | < | + | |
- | source pymatlab/ | + | |
</ | </ | ||
- | if you are using csh! | + | |
- | To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt | + | To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt |
<code bash> | <code bash> | ||
(pymatlab)username@hostname: | (pymatlab)username@hostname: | ||
</ | </ | ||
- | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run | + | to emphasize that you are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment |
<code bash> | <code bash> | ||
pip install | pip install | ||
</ | </ | ||
- | Your virtual environment | + | Your virtual environment now should |
- | In any cases, it is advisable | + | In any cases, it is advisable |
<code bash> | <code bash> | ||
pip freeze > packages.dat | pip freeze > packages.dat | ||
Line 138: | Line 178: | ||
so do not panic if things do not work, just delete your virtual environment and start all over again. | so do not panic if things do not work, just delete your virtual environment and start all over again. | ||
- | Finally, it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option | + | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised |
- | <code bash> | + | |
- | virtualenv -p / | + | |
- | </ | + | |
- | Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised | + | |
<code bash> | <code bash> | ||
virtualenv --help | virtualenv --help | ||
</ | </ | ||
- | ==== METHOD 3: easy_install ==== | + | |
+ | ==== METHOD 3: easy_install | ||
Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. | ||
- | By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so | + | By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the '' |
- | to install | + | |
<code bash> | <code bash> | ||
easy_install -N --user pymatlab | easy_install -N --user pymatlab | ||
</ | </ | ||
- | This will install pymatlab in ~/.local/lib ready to be imported in your next python session. | + | This will install pymatlab in '' |
- | Note: If you want to install your package in a different location than ~/.local, then set the environment variable | + | If you want to install your package in a different location than '' |
<code bash> | <code bash> | ||
- | export PYTHONUSERBASE=/ | + | export PYTHONUSERBASE=/ |
</ | </ | ||
Line 164: | Line 200: | ||
python -m easy_install --help | python -m easy_install --help | ||
</ | </ | ||
- | ===== Example: how to let python search arbitrary library paths ===== | ||
- | Create/edit | ||
- | |||
- | '' | ||
- | |||
- | by appending the path of your choice, for instance | ||
- | < | + | ==== Migrating packages between python versions ==== |
- | echo "/my/home/sweet/home/library" | + | Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). |
+ | Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/# | ||
+ | | ||
+ | <code bash> | ||
+ | python3.7 | ||
</ | </ | ||
- | + | - Reinstall for the current | |
- | All '' | + | |
- | + | ||
- | ===== Example: how to create a python environment module | + | |
- | + | ||
- | First enable the module package to search also private module directories | + | |
<code bash> | <code bash> | ||
- | module load use.own | + | python3.9 -m pip install --user -r installed.txt |
</ | </ | ||
+ | - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/ | ||
+ | ===== Example: how to let python search arbitrary library paths ===== | ||
+ | For instance for python v2.7 installations, | ||
- | the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files. | + | '' |
- | Let us now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package | + | by appending the path of your choice |
- | < | + | < |
- | export PYTHONUSERBASE=/somewhere/I/can/write/to | + | echo "/my/home/sweet/home/library" |
- | pip install | + | |
- | pip install -I --user SomePackageThatWASInstalledSystemwide | + | |
</ | </ | ||
- | Create a file, say `$HOME/ | + | All ''.pth'' |
- | <code bash> | + | |
- | #%Module 1.0 | + | |
- | # | + | |
- | # | + | |
- | # | + | |
- | prepend-path | + | |
- | prepend-path | + | |
- | </ | + | |
- | and type | + | ===== Example: how to create your own python environment module |
+ | |||
+ | Please read [[easybuild_environment|here]]. | ||
- | <code bash> | ||
- | module load super-module | ||
- | </ | ||
- | and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. | ||
===== Example: numpy with openBLAS ===== | ===== Example: numpy with openBLAS ===== | ||
In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. | ||
- | :!: The procedure and paths below will work on any maris node and on the para cluster. | + | :!: The procedure and paths below will work on any maris node. |
<code bash> | <code bash> | ||
Line 289: | Line 309: | ||
====== Anaconda/ | ====== Anaconda/ | ||
Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/ | ||
+ | |||
+ | |||
+ | ====== Jupyter Notebooks ====== | ||
+ | Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get '' | ||
+ | python3 -m ipykernel install --user | ||
+ | Once this command has run successfully, | ||
+ | |||
+ | After starting '' | ||
+ | |||
+ | If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g: | ||
+ | python3 -m ipykernel install --user --name system-python3 | ||
+ | |||
+ |