Computer Documentation Wiki

User Tools

Site Tools

Trace:
working_with_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
working_with_python [2018/08/30 07:45] – [METHOD 2: virtualenv] lenocilworking_with_python [2024/04/22 10:51] (current) – jansen
Line 1: Line 1:
 ====== Working with Python ====== ====== Working with Python ======
-Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python 2 (currently 2.7) and python 3 (currently 3.6) are available.+Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python v2 and python v3 are available. Please note that python v2 has reached its  //end-of-life// on 01-01-2020 and therefore is no longer actively supported.
  
-Notes from a [[https://www.strw.leidenuniv.nl/python/| python introduction course]] are available on the observatory web site.  +===== Python packages =====
-===== Python modules =====+
  
-Many common python packages (or modules), such as numpy, scipy, and astropy are available to any users regardless of the workstation. To display a list of available modules in python v2 type+Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the ''module'' command.  
 + 
 +For local python installations, you can list all installed packages via 
  
 <code bash> <code bash>
-python -m pip list+python3 -m pip list # or python2 -m pip list 
 </code> </code>
-or similarly in python v3 
-<code bash> 
-python3 -m pip list 
-</code>  
  
  
-If the package you would like to use is not installed you have two options:+For remote python installations -- installations on our software disk -- you must first load a python module and then list all packages in that module
  
-  * Contact the system administrators to install it globally (system wide). +<code bash> 
-  * Install it locally, that is in one of the directories writeable by you.+module load Python/3.6.6-foss-2018b 
 +which python 
 +/easybuild/easybuild/fc31/software/Python/3.6.6-foss-2018b/bin/python 
 +python -m pip list 
 +</code> 
 + 
 +If the package you would like to use is not installed at all you have two options: 
 + 
 +  * Request the installation to the system administrators. 
 +  * Install it yourself, to one of the directories to which you have writing access.
  
 The two options are described in detail in the sections below. The two options are described in detail in the sections below.
-===== Global  installation ===== +===== Request a Python Package Installation ===== 
-If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request the installation on our helpdesk https://helpdesk.strw.leidenuniv.nl/ (STRW) or https://helpdesk.lorentz.leidenuniv.nl/ (Lorentz) giving motivations and detailed instructions on where to find the requested package and its license information. We will notify you when the installation is complete. +If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request its  installation via our helpdesk https://helpdesk.strw.leidenuniv.nl/ (STRW) or https://helpdesk.lorentz.leidenuniv.nl/ (Lorentz) giving motivations and detailed instructions on where to find the requested package and its license information. We will notify you when the installation is complete. 
-===== Local installation ===== +===== Install a Python Package yourself ===== 
-Sometimes, the requested python package is not useful to other researchers in your department and/or you are a developer who wants to try and modify development versions of installed packages or new packages.  + 
-Generally speaking, a user or developer will want to install and manage python packages in  alternative locations (usually outside the system python environment) for one of following reasons:+There are instances in which you would like to install a python package that is not useful to other researchers in your department and/or you are a developer who wants to try and modify development versions of installed packages or new packages. In other words, if
  
    - The package is not of interest to the majority of users.    - The package is not of interest to the majority of users.
-   - They don't have write access to the main Python site-packages directories. +   - You want a custom stash of packages, that is not visible to other users. 
-   - They want a custom stash of packages, that is not visible to other users. +   - You want to isolate a set of packages to a specific python application, usually to minimize the possibility of version conflicts. 
-   - They want to isolate a set of packages to a specific python application, usually to minimize the possibility of version conflicts.+ 
 +we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process. 
 + 
 +:!: The following methods are valid either you use a local (to your workstation/server) python installation or a python installation provided via the ''module'' command. If you choose the latter, remember to load first an appropriate python module.
  
-In this case you can follow one of the methods below. 
 ==== METHOD 1: pip with the `--user' option ==== ==== METHOD 1: pip with the `--user' option ====
  
-Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. A user's default install location is defined through the `site' module with the variable ''site.USER_BASE''. In the GNU/Linux OS  ''site.USER_BASE'' defaults to ''~/.local''. This mode of installation can be turned on by specifying the `--user' option to pip install, for instance+Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. Python provides this functionality via the ''site'' module which commands where you, as a user, will be installing python packages
  
 +<code bash>
 +python -m site --user-base # value of site.USER_BASE
 +python -m site --user-site # path to your site-packages directory 
 +</code>
 +
 +If the values returned by the command above satisfy you, you can then proceed to install packages in your user-space
 <code bash> <code bash>
 pip install --user SomePackage pip install --user SomePackage
 </code> </code>
  
-To display the value of `site.USER_BASE', type+In the STRW and IL environments, site.USER_BASE  defaults to  ''$HOME/.local''. This path can be customised/updated by modifying the environment variable ''PYTHONUSERBASE''
  
 <code bash> <code bash>
-python[2,3] -m site --user-base+export PYTHONUSERBASE=/somewhere/I/can/write/to # alternative location 
 +pip install --user SomePackage
 </code> </code>
 +will install `SomePackage' in ''/somewhere/I/can/write/to/lib/python*/site-packages''.
  
-and to show the path to your site-packages directory +When using the `user' scheme to install packages, it is important to note 
 + 
 +  * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled. 
 +  * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied. 
 +  * pip will not perform a ''--user'' install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages. 
 + 
 +=== Method 1 - subsection Incompatible versions === 
 + 
 +Unfortunately, python's 'user' directory is independent of the operating system version, but most of the compute nodes including the LOFAR cluster, run RedHat Enterprise Linux, which is sufficiently different to cause packages installed on the desktop not to work all the time. 
 + 
 +In cases like this, it might be necessary to create a separate python user directory structure for those machines: 
 + 
 +Add to your .bashrc something like this: 
 +  if [ ! -f /etc/fedora-release ]; then 
 +    export PYTHONUSERBASE=$HOME/.local-rhel7 
 +  fi 
 +For users of the ''tcsh'' shell, add this to your .tcshrc in stead: 
 +  if (! -f /etc/fedora-release) then 
 +    setenv PYTHONUSERBASE $HOME/.local-rhel7 
 +  endif 
 + 
 +And make sure to create that directory ~/.local-rhel7. Now the pip --user commands on RHEL7 machines will install into that newly created directory in stead of the default one used by the desktop systems. 
 + 
 +==== METHOD 2: venv ====
  
 +''venv'' is a tool that creates isolated Python environments; it replaces the obsolete ''virtualenv'' that provided similar functionality for python 2.x. A python environment is essentially  a folder which contains copies of all necessary files needed for a Python project to run. In addition each virtual environment will contain a copy of the utility pip to manage packages. For example, let us suppose you would like to install ''pymatlab'' which is not installed on the departmental workstations, then you could do
 <code bash> <code bash>
-python[2,3] -m site --user-site+$ mkdir /data2/username/venvs  
 +$ python3 -m venv /data2/username/venvs/pymatlab
 </code> </code>
 +to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/data2'' disk, in order to avoid running out of disk quota in your home directory, which can easily happen since venvs can become rather big).
  
-In the STRW and IL environments the site.USER_BASE variable defaults to a user's ''~/.local'' directory nonetheless it can be customised/updated by modifying the environment variable PYTHONUSERBASE:+In the example, we use ''python3'' as the python for this environment; if there are multiple python versions on the system, and you want to base your venv on a specific version, use that version to create the venv, e.g. ''python3.12 -m venv /data2/username/venvs/pymatlab''. 
 + 
 +The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing
 <code bash> <code bash>
-# in bash +source /data2/username/pymatlab/bin/activate # bash shells 
-export PYTHONUSERBASE=/somewhere/I/can/write/to +source /data2/username/pymatlab/bin/activate.csh # c shells
-pip install --user SomePackage+
 </code> </code>
-will install `SomePackage' in ''/somewhere/I/can/write/to''. 
  
-When using the `user' scheme to install packages, it is important to note+To acknowledge the activation of pymatlab, the terminal prompt will be changed to 
 +<code bash> 
 +(pymatlab)username@hostname:~/python_virt_envs/pymatlab$ 
 +</code> 
 +to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment 
 +<code bash> 
 +pip install  pymatlab 
 +</code> 
 +Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment.  
 +Note that you do NOT use ''--user'' on the pip command in this case, since that would install in your ''$PYTHONUSERBASE'' directory (see above) instead of the venv!!
  
-  * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled. +In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages 
-  * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied. +<code bash> 
-  * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages.+pip freeze > packages.dat 
 +</code> 
 +This can help collaborators and fellow developers to reproduce your environment with  
 +<code bash> 
 +pip install -r packages.dat 
 +</code> 
 +When you are done working in a virtual environment deactivate it running  
 +<code bash> 
 +deactivate  
 +</code> 
 +At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system so do not panic if things do not work, just delete your virtual environment and start all over again. 
 + 
 +Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation.
  
-==== METHOD 2: virtualenv ====+==== METHOD 2: OBSOLETE: virtualenv (python 2.x) ====
 This guide refers to virtualenv version 12.0.7.  This guide refers to virtualenv version 12.0.7. 
  
Line 78: Line 142:
 to create a virtual environment (folder) called pymatlab. to create a virtual environment (folder) called pymatlab.
  
-Please note that your newly created virtual environment will be a `python2' one if you used `virtualenv' or a +Python virtual environments containing specific versions of python can be created using the ''-p'' option as in  ''virtualenv -p /usr/bin/python3.6''. 
- `python3' one if using `virtualenv-3'.+
  
-The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to your $PATH environment variable. This is done by issuing+The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing
 <code bash> <code bash>
-source pymatlab/bin/activate +source pymatlab/bin/activate # bash shells 
-</code>  +source pymatlab/bin/activate.csh # c shells
-or  +
-<code bash> +
-source pymatlab/bin/activate.csh+
 </code> </code>
-if you are using csh! + 
-To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ($PS1) to +To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ''$PS1'' to 
 <code bash> <code bash>
 (pymatlab)username@hostname:~/python_virt_envs/pymatlab$ (pymatlab)username@hostname:~/python_virt_envs/pymatlab$
 </code> </code>
-to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run+to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment
 <code bash> <code bash>
 pip install  pymatlab pip install  pymatlab
 </code> </code>
-Your virtual environment should now have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. +Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. 
  
-In any cases, it is advisable to keep a backup of your virtual environment configuration by creating a list of installed packages+In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages
 <code bash> <code bash>
 pip freeze > packages.dat pip freeze > packages.dat
Line 118: Line 178:
 so do not panic if things do not work, just delete your virtual environment and start all over again. so do not panic if things do not work, just delete your virtual environment and start all over again.
  
-Finally, it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option +Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation
-<code bash> +
-virtualenv -p /usr/bin/python3.4 pymatlab +
-</code> +
-Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised to consult the documentation:+
 <code bash> <code bash>
 virtualenv --help virtualenv --help
 </code> </code>
-==== METHOD 3: easy_install ====+ 
 +==== METHOD 3: easy_install with the `--user' option ====
 Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages.  Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. 
-By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so +By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the ''--user'' option in a similar fashion to pip's 
-to install a package locally type +<code bash>
-<code>+
 easy_install -N --user pymatlab easy_install -N --user pymatlab
 </code> </code>
-This will install pymatlab in ~/.local/lib ready to be imported in your next python session. +This will install pymatlab in ''${HOME}/.local/'' ready to be imported in your next python session. 
-Note: If you want to install your package in a different location than ~/.local, then set the environment variable  $PYTHONUSERBASE to a custom location, e.g,  +If you want to install your package in a different location than ''$HOME/.local'', then set the environment variable  $PYTHONUSERBASE to a custom location, e.g,  
-<code> +<code bash> 
-export PYTHONUSERBASE=/home/user/some/where/I/can/write+export PYTHONUSERBASE=/home/user/some/where/I/can/write # alternative location
 </code> </code>
  
 Please consult the docs to know more: Please consult the docs to know more:
  
-<code>+<code bash>
 python -m easy_install --help python -m easy_install --help
 </code> </code>
-===== Example: how to let python search arbitrary library paths  ===== 
-Create/edit 
  
-''~/.local/lib/python2.7/site-packages/my-super-library.pth'' +==== Migrating packages between python versions ==== 
- +Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). 
-by appending the path of your choice, for instance +Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/#_notes_on_migrating_user_installed_pip_packages|Fedora release notes]]: 
- +  - Make a list of installed packages in the old python version: 
-<code> +<code bash>   
-echo "/my/home/sweet/home/library" >> ~/.local/lib/python2.7/site-packages/my-super-library.pth+  python3.7 -m pip freeze > installed.txt 
 </code> </code>
 +  - Reinstall for the current python version:
 +<code bash>
 +  python3.9 -m pip install --user -r installed.txt
 +</code>
 +  - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/.local/lib/python3.7
 +===== Example: how to let python search arbitrary library paths  =====
 +For instance for python v2.7 installations, create or edit
  
-All ''.pth'' files will be sourced by python provided they are in the right location.+''$HOME/.local/lib/python2.7/site-packages/my-super-library.pth''
  
-===== Example: how to create a python environment module  =====+by appending the path of your choice
  
-First enable the module package to search also private module directories 
 <code> <code>
-module load use.own+echo "/my/home/sweet/home/library" >> $HOME/.local/lib/python2.7/site-packages/my-super-library.pth
 </code> </code>
  
-the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files.+All ''.pth'' files will be sourced by python provided they are in the right location.
  
-Let us  now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package +===== Example: how to create your own python environment module  =====
  
-<code> +Please read [[easybuild_environment|here]].
-export PYTHONUSERBASE=/somewhere/I/can/write/to +
-pip install --user SomePackage +
-pip install -I --user SomePackageThatWASInstalledSystemwide +
-</code>+
  
-Create a file, say `$HOME/privatemodules/super-module' with the following contents 
-<code> 
-#%Module 1.0 
-# 
-#   
-#  
-prepend-path  PATH          /somewhere/I/can/write/to/bin # if executables were installed 
-prepend-path  PYTHONPATH    /somewhere/I/can/write/to/lib/python2.7/site-packages 
-</code> 
- 
-and type  
- 
-<code> 
-module load super-module 
-</code> 
-and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. 
 ===== Example: numpy with openBLAS ===== ===== Example: numpy with openBLAS =====
 In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library.  In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. 
  
-:!: The procedure and paths below will work on any maris node and on the para cluster. +:!: The procedure and paths below will work on any maris node. 
  
-<code>+<code bash>
 virtualenv py2_numpy_openBLAS virtualenv py2_numpy_openBLAS
 source py2_numpy_openBLAS/bin/activate source py2_numpy_openBLAS/bin/activate
Line 206: Line 246:
 </code>  </code> 
 Edit ''site.cfg'' with your favorite editor such that  Edit ''site.cfg'' with your favorite editor such that 
-<code>+<code bash>
 [openblas] [openblas]
 libraries = openblas libraries = openblas
Line 214: Line 254:
 </code> </code>
 then install numpy  then install numpy 
-<code>+<code bash>
 python setup.py install python setup.py install
 </code> </code>
 If the installation is going smoothly you should see  If the installation is going smoothly you should see 
-<code>+<code bash>
 .... ....
  
Line 234: Line 274:
 </code> </code>
 Now that numpy is installed you could also install scipy, for instance Now that numpy is installed you could also install scipy, for instance
-<code>+<code bash>
 pip install scipy pip install scipy
 </code> </code>
Line 252: Line 292:
 ====== Bypassing the existing python environment ====== ====== Bypassing the existing python environment ======
 Occasionally, something in the systemwide directories (e.g ''/software/local/lib64/python2.7/site-packages'') interferes with your python application. Perhaps you have a code that requires a specific, older, version of numpy or matplotlib. Just installing that version is not always sufficient. The trick is, to set the PYTHONPATH to point first to a directory where you place a private ''sitecustomize.py'' which then overrides the one we have placed in /usr/lib64/python2.7/site-packages (which is where we add the /software directories to the path for everyone). Here is how: Occasionally, something in the systemwide directories (e.g ''/software/local/lib64/python2.7/site-packages'') interferes with your python application. Perhaps you have a code that requires a specific, older, version of numpy or matplotlib. Just installing that version is not always sufficient. The trick is, to set the PYTHONPATH to point first to a directory where you place a private ''sitecustomize.py'' which then overrides the one we have placed in /usr/lib64/python2.7/site-packages (which is where we add the /software directories to the path for everyone). Here is how:
 +<code bash>
   mkdir /some/location/python_custom_dir   mkdir /some/location/python_custom_dir
   setenv PYTHONPATH /some/location/python_custom_dir:/usr/lib64/python2.7/site-packages   setenv PYTHONPATH /some/location/python_custom_dir:/usr/lib64/python2.7/site-packages
 +</code>
 The ''sitecustomize.py'' could be something like this: The ''sitecustomize.py'' could be something like this:
 +<code python>
   import sys   import sys
   import site   import site
Line 262: Line 305:
   # In order to find also eggs and subdirectories, addsitedir seems necessary:   # In order to find also eggs and subdirectories, addsitedir seems necessary:
   site.addsitedir(mypath, known_paths=None)   site.addsitedir(mypath, known_paths=None)
 +</code>
 +
 +====== Anaconda/Miniconda ======
 +Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls).
 +
 +
 +====== Jupyter Notebooks ======
 +Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing:
 +    python3 -m ipykernel install --user
 +Once this command has run successfully, it will have installed python3 as a jupyter kernel.
 +
 +After starting ''jupyter notebook'' you can select ''python3'' as kernel.
 +
 +If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g:
 +    python3 -m ipykernel install --user --name system-python3
  
  
working_with_python.1535615132.txt.gz · Last modified: 2018/08/30 07:45 by lenocil

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki