User Tools

Site Tools


working_with_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
working_with_python [2020/10/05 08:54]
lenocil [Python modules]
working_with_python [2020/12/22 14:06] (current)
jansen
Line 2: Line 2:
 Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python v2 and python v3 are available. Please note that python v2 has reached its  //end-of-life// on 01-01-2020 and therefore is no longer actively supported. Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python v2 and python v3 are available. Please note that python v2 has reached its  //end-of-life// on 01-01-2020 and therefore is no longer actively supported.
  
-Notes from a [[https://www.strw.leidenuniv.nl/python/| python introduction course]] are available on the observatory web site.  +===== Python packages =====
-===== Python modules =====+
  
 Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the ''module'' command.  Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the ''module'' command. 
Line 10: Line 9:
  
 <code bash> <code bash>
-python2 -m pip list+python3 -m pip list # or python2 -m pip list
  
-# or 
- 
-python3 -m pip list 
 </code> </code>
  
Line 29: Line 25:
 If the package you would like to use is not installed at all you have two options: If the package you would like to use is not installed at all you have two options:
  
-  * Contact the system administrators to install it globally (system wide)+  * Request the installation to the system administrators
-  * Install it locallythat is in one of the directories writeable by you.+  * Install it yourselfto one of the directories to which you have writing access.
  
 The two options are described in detail in the sections below. The two options are described in detail in the sections below.
-===== Global  installation ===== +===== Request a Python Package Installation ===== 
-If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request the installation on our helpdesk https://helpdesk.strw.leidenuniv.nl/ (STRW) or https://helpdesk.lorentz.leidenuniv.nl/ (Lorentz) giving motivations and detailed instructions on where to find the requested package and its license information. We will notify you when the installation is complete. +If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request its  installation via our helpdesk https://helpdesk.strw.leidenuniv.nl/ (STRW) or https://helpdesk.lorentz.leidenuniv.nl/ (Lorentz) giving motivations and detailed instructions on where to find the requested package and its license information. We will notify you when the installation is complete. 
-===== Local installation ===== +===== Install a Python Package yourself ===== 
-Sometimes, the requested python package is not useful to other researchers in your department and/or you are a developer who wants to try and modify development versions of installed packages or new packages.  + 
-Generally speakinga user or developer will want to install and manage python packages in  alternative locations (usually outside the system python environment) for one of following reasons:+There are instances in which you would like to install a python package that is not useful to other researchers in your department and/or you are a developer who wants to try and modify development versions of installed packages or new packages. In other wordsif
  
    - The package is not of interest to the majority of users.    - The package is not of interest to the majority of users.
-   They don't have write access to the main Python site-packages directories. +   You want a custom stash of packages, that is not visible to other users. 
-   - They want a custom stash of packages, that is not visible to other users. +   You want to isolate a set of packages to a specific python application, usually to minimize the possibility of version conflicts
-   They want to isolate a set of packages to a specific python application, usually to minimize the possibility of version conflicts.+ 
 +we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process. 
 + 
 +:!: The following methods are valid either you use a local (to your workstation/server) python installation or a python installation provided via the ''module'' command. If you choose the latter, remember to load first an appropriate python module.
  
-In this case you can follow one of the methods below. 
 ==== METHOD 1: pip with the `--user' option ==== ==== METHOD 1: pip with the `--user' option ====
  
-Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. A user's default install location is defined through the `site' module with the variable ''site.USER_BASE''. In the GNU/Linux OS  ''site.USER_BASE'' defaults to ''~/.local''. This mode of installation can be turned on by specifying the `--user' option to pip installfor instance+Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. Python provides this functionality via the ''site'' module which commands where you, as a user, will be installing python packages
  
 <code bash> <code bash>
-pip install --user SomePackage+python -m site --user-base # value of site.USER_BASE 
 +python -m site --user-site # path to your site-packages directory 
 </code> </code>
  
-To display the value of `site.USER_BASE'type +If the values returned by the command above satisfy youyou can then proceed to install packages in your user-space
 <code bash> <code bash>
-python[2,3] -m site --user-base+pip install --user SomePackage
 </code> </code>
  
-and to show the path to your site-packages directory  +In the STRW and IL environments, site.USER_BASE  defaults to  ''$HOME/.local''. This path can be customised/updated by modifying the environment variable ''PYTHONUSERBASE''
- +
-<code bash> +
-python[2,3] -m site --user-site +
-</code>+
  
-In the STRW and IL environments the site.USER_BASE variable defaults to a user's ''~/.local'' directory nonetheless it can be customised/updated by modifying the environment variable PYTHONUSERBASE: 
 <code bash> <code bash>
-# in bash +export PYTHONUSERBASE=/somewhere/I/can/write/to # alternative location
-export PYTHONUSERBASE=/somewhere/I/can/write/to+
 pip install --user SomePackage pip install --user SomePackage
 </code> </code>
-will install `SomePackage' in ''/somewhere/I/can/write/to''.+will install `SomePackage' in ''/somewhere/I/can/write/to/lib/python*/site-packages''.
  
 When using the `user' scheme to install packages, it is important to note When using the `user' scheme to install packages, it is important to note
Line 77: Line 69:
   * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.   * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
   * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied.   * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied.
-  * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages.+  * pip will not perform a ''--user'' install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages.
  
 === Method 1 - subsection Incompatible versions === === Method 1 - subsection Incompatible versions ===
Line 89: Line 81:
     export PYTHONUSERBASE=$HOME/.local-rhel7     export PYTHONUSERBASE=$HOME/.local-rhel7
   fi   fi
-For users of the ''tcsh'' shell, add this to your .cshrc in stead:+For users of the ''tcsh'' shell, add this to your .tcshrc in stead:
   if (! -f /etc/fedora-release) then   if (! -f /etc/fedora-release) then
     setenv PYTHONUSERBASE $HOME/.local-rhel7     setenv PYTHONUSERBASE $HOME/.local-rhel7
Line 106: Line 98:
 to create a virtual environment (folder) called pymatlab. to create a virtual environment (folder) called pymatlab.
  
-Please note that your newly created virtual environment will be a `python2one if you used `virtualenvor a +Python virtual environments containing specific versions of python can be created using the ''-p'' option as in  ''virtualenv -p /usr/bin/python3.6''
- `python3' one if using `virtualenv-3'. Note that if the command `virtualenv-3is missing you could use ''virtualenv -p /usr/bin/python3''+
  
-The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to your $PATH environment variable. This is done by issuing+The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing
 <code bash> <code bash>
-source pymatlab/bin/activate +source pymatlab/bin/activate bash shells 
-</code>  +source pymatlab/bin/activate.csh # c shells
-or  +
-<code bash> +
-source pymatlab/bin/activate.csh+
 </code> </code>
-if you are using csh! + 
-To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ($PS1to +To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ''$PS1'' to 
 <code bash> <code bash>
 (pymatlab)username@hostname:~/python_virt_envs/pymatlab$ (pymatlab)username@hostname:~/python_virt_envs/pymatlab$
 </code> </code>
-to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run+to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment
 <code bash> <code bash>
 pip install  pymatlab pip install  pymatlab
 </code> </code>
-Your virtual environment should now have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. +Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. 
  
-In any cases, it is advisable to keep a backup of your virtual environment configuration by creating a list of installed packages+In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages
 <code bash> <code bash>
 pip freeze > packages.dat pip freeze > packages.dat
Line 146: Line 134:
 so do not panic if things do not work, just delete your virtual environment and start all over again. so do not panic if things do not work, just delete your virtual environment and start all over again.
  
-Finally, it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option +Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation
-<code bash> +
-virtualenv -p /usr/bin/python3.4 pymatlab +
-</code> +
-Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised to consult the documentation:+
 <code bash> <code bash>
 virtualenv --help virtualenv --help
 </code> </code>
-==== METHOD 3: easy_install ====+ 
 +==== METHOD 3: easy_install with the `--user' option ====
 Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages.  Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. 
-By default, easy_install installs python packages into Pythons main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so +By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the ''--user'' option in similar fashion to pip's
-to install package locally type+
 <code bash> <code bash>
 easy_install -N --user pymatlab easy_install -N --user pymatlab
 </code> </code>
-This will install pymatlab in ~/.local/lib ready to be imported in your next python session. +This will install pymatlab in ''${HOME}/.local/'' ready to be imported in your next python session. 
-Note: If you want to install your package in a different location than ~/.local, then set the environment variable  $PYTHONUSERBASE to a custom location, e.g, +If you want to install your package in a different location than ''$HOME/.local'', then set the environment variable  $PYTHONUSERBASE to a custom location, e.g, 
 <code bash> <code bash>
-export PYTHONUSERBASE=/home/user/some/where/I/can/write+export PYTHONUSERBASE=/home/user/some/where/I/can/write # alternative location
 </code> </code>
  
Line 172: Line 156:
 python -m easy_install --help python -m easy_install --help
 </code> </code>
 +
 +==== Migrating packages between python versions ====
 +Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9).
 +Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/#_notes_on_migrating_user_installed_pip_packages|Fedora release notes]]:
 +  - Make a list of installed packages in the old python version:
 +<code bash>  
 +  python3.7 -m pip freeze > installed.txt 
 +</code>
 +  - Reinstall for the current python version:
 +<code bash>
 +  python3.9 -m pip install --user -r installed.txt
 +</code>
 +  - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/.local/lib/python3.7
 ===== Example: how to let python search arbitrary library paths  ===== ===== Example: how to let python search arbitrary library paths  =====
-Create/edit+For instance for python v2.7 installations, create or edit
  
-''~/.local/lib/python2.7/site-packages/my-super-library.pth''+''$HOME/.local/lib/python2.7/site-packages/my-super-library.pth''
  
-by appending the path of your choice, for instance+by appending the path of your choice
  
 <code> <code>
-echo "/my/home/sweet/home/library" >> ~/.local/lib/python2.7/site-packages/my-super-library.pth+echo "/my/home/sweet/home/library" >> $HOME/.local/lib/python2.7/site-packages/my-super-library.pth
 </code> </code>
  
 All ''.pth'' files will be sourced by python provided they are in the right location. All ''.pth'' files will be sourced by python provided they are in the right location.
  
-===== Example: how to create python environment module  =====+===== Example: how to create your own python environment module  =====
  
-First enable the module package to search also private module directories +Please read [[easybuild_environment|here]].
-<code bash> +
-module load use.own +
-</code>+
  
-the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files. 
- 
-Let us  now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package  
- 
-<code bash> 
-export PYTHONUSERBASE=/somewhere/I/can/write/to 
-pip install --user SomePackage 
-pip install -I --user SomePackageThatWASInstalledSystemwide 
-</code> 
- 
-Create a file, say `$HOME/privatemodules/super-module' with the following contents 
-<code bash> 
-#%Module 1.0 
-# 
-#   
- 
-prepend-path  PATH          /somewhere/I/can/write/to/bin # if executables were installed 
-prepend-path  PYTHONPATH    /somewhere/I/can/write/to/lib/python2.7/site-packages 
-</code> 
- 
-and type  
- 
-<code bash> 
-module load super-module 
-</code> 
-and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. 
 ===== Example: numpy with openBLAS ===== ===== Example: numpy with openBLAS =====
 In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library.  In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. 
Line 297: Line 265:
 ====== Anaconda/Miniconda ====== ====== Anaconda/Miniconda ======
 Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls). Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls).
 +
 +
 +====== Jupyter Notebooks ======
 +Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing:
 +    python3 -m ipykernel install –user
 +Once this command has run successfully, it will have installed python3 as a jupyter kernel.
 +
 +After starting ''jupyter notebook'' you can select ''python3'' as kernel.
working_with_python.1601888053.txt.gz · Last modified: 2020/10/05 08:54 by lenocil