Computer Documentation Wiki

User Tools

Site Tools

Trace:
working_with_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
working_with_python [2018/02/22 12:18] – [Global installation] lenocilworking_with_python [2023/11/06 13:12] (current) – [Jupyter Notebooks] jansen
Line 1: Line 1:
 ====== Working with Python ====== ====== Working with Python ======
-Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python 2 (currently 2.7) and python 3 (currently 3.6) are available.+Python is available on all Sterrewacht and Lorentz Institute GNU/Linux workstations. In most cases, both python v2 and python v3 are available. Please note that python v2 has reached its  //end-of-life// on 01-01-2020 and therefore is no longer actively supported.
  
-Notes from a [[https://www.strw.leidenuniv.nl/python/| python introduction course]] are available on the observatory web site.  +===== Python packages =====
-===== Python modules =====+
  
-Many common python packages (or modules), such as numpy, scipy, and astropy are available to any users regardless of the workstation. To display a list of available modules in python v2 type+Many common python packages, such as numpy, scipy, and astropy are available to any users regardless of the workstation/server. These are installed either locally to the workstation -- usually via the OS package manager -- or remotely and exposed to the users by means of the ''module'' command.  
 + 
 +For local python installations, you can list all installed packages via  
 + 
 +<code bash> 
 +python3 -m pip list # or python2 -m pip list
  
-<code> 
-python -m pip list 
 </code> </code>
-or similarly in python v3 
-<code> 
-python3 -m pip list 
-</code>  
  
  
-If the package you would like to use is not installed you have two options:+For remote python installations -- installations on our software disk -- you must first load a python module and then list all packages in that module
  
-  * Contact the system administrators to install it globally (system wide). +<code bash> 
-  * Install it locally, that is in one of the directories writeable by you.+module load Python/3.6.6-foss-2018b 
 +which python 
 +/easybuild/easybuild/fc31/software/Python/3.6.6-foss-2018b/bin/python 
 +python -m pip list 
 +</code> 
 + 
 +If the package you would like to use is not installed at all you have two options: 
 + 
 +  * Request the installation to the system administrators. 
 +  * Install it yourself, to one of the directories to which you have writing access.
  
 The two options are described in detail in the sections below. The two options are described in detail in the sections below.
-===== Global  installation ===== +===== Request a Python Package Installation ===== 
-If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request the installation on our helpdesk https://helpdesk.strw.leidenuniv.nl/ (STRW) or https://helpdesk.lorentz.leidenuniv.nl/ (Lorentz) giving motivations and detailed instructions on where to find the requested package and its license information. We will notify you when the installation is complete. +If you believe that the required package could be useful to other researchers in the Observatory or Lorentz Institute, then you can request its  installation via our helpdesk https://helpdesk.strw.leidenuniv.nl/ (STRW) or https://helpdesk.lorentz.leidenuniv.nl/ (Lorentz) giving motivations and detailed instructions on where to find the requested package and its license information. We will notify you when the installation is complete. 
-===== Local installation ===== +===== Install a Python Package yourself ===== 
-Sometimes, the requested python package is not useful to other researchers in your department and/or you are a developer who wants to try and modify development versions of installed packages or new packages.  + 
-Generally speaking, a user or developer will want to install and manage python packages in an alternative location (usually outside the system python environment) for one of following reasons:+There are instances in which you would like to install a python package that is not useful to other researchers in your department and/or you are a developer who wants to try and modify development versions of installed packages or new packages. In other words, if
  
    - The package is not of interest to the majority of users.    - The package is not of interest to the majority of users.
-   - They don't have access to write to the main Python site-packages directory. +   - You want a custom stash of packages, that is not visible to other users. 
-   - They want a user-specific stash of packages, that is not visible to other users. +   - You want to isolate a set of packages to a specific python application, usually to minimize the possibility of version conflicts. 
-   - They want to isolate a set of packages to a specific python application, usually to minimize the possibility of version conflicts.+ 
 +we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process. 
 + 
 +:!: The following methods are valid either you use a local (to your workstation/server) python installation or a python installation provided via the ''module'' command. If you choose the latter, remember to load first an appropriate python module.
  
-In this case you can follow one of the methods below. +==== METHOD 1: pip with the `--user' option ====
-==== pip using the `user' scheme ====+
  
-With Python 2.6 came the "user scheme" for installation, which means that all Python distributions support an alternative install location that is specific to a user. The default location for each OS is explained in the python documentation for the site.USER_BASE variable. This mode of installation can be turned on by specifying the --user option to pip install+Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. Python provides this functionality via the ''site'' module which commands where you, as a user, will be installing python packages
  
-<code>+<code bash> 
 +python -m site --user-base # value of site.USER_BASE 
 +python -m site --user-site # path to your site-packages directory  
 +</code> 
 + 
 +If the values returned by the command above satisfy you, you can then proceed to install packages in your user-space 
 +<code bash>
 pip install --user SomePackage pip install --user SomePackage
 </code> </code>
  
-In the STRW and IL environments the site.USER_BASE variable defaults to the user's ''.local'' directory and can be customised/updated by modifying the environment variable PYTHONUSERBASE, for instance in bash +In the STRW and IL environments, site.USER_BASE  defaults to  ''$HOME/.local''. This path can be customised/updated by modifying the environment variable ''PYTHONUSERBASE'' 
-<code> + 
-export PYTHONUSERBASE=/somewhere/I/can/write/to+<code bash> 
 +export PYTHONUSERBASE=/somewhere/I/can/write/to # alternative location
 pip install --user SomePackage pip install --user SomePackage
 </code> </code>
-will install `SomePackage' in ''/somewhere/I/can/write/to''.+will install `SomePackage' in ''/somewhere/I/can/write/to/lib/python*/site-packages''.
  
-When using the `user' scheme to install packages, it is important to consider the following+When using the `user' scheme to install packages, it is important to note
  
   * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.   * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
   * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied.   * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied.
-  * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages.+  * pip will not perform a ''--user'' install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages. 
 + 
 +=== Method 1 - subsection Incompatible versions === 
 + 
 +Unfortunately, python's 'user' directory is independent of the operating system version, but most of the compute nodes including the LOFAR cluster, run RedHat Enterprise Linux, which is sufficiently different to cause packages installed on the desktop not to work all the time. 
 + 
 +In cases like this, it might be necessary to create a separate python user directory structure for those machines: 
 + 
 +Add to your .bashrc something like this: 
 +  if [ ! -f /etc/fedora-release ]; then 
 +    export PYTHONUSERBASE=$HOME/.local-rhel7 
 +  fi 
 +For users of the ''tcsh'' shell, add this to your .tcshrc in stead: 
 +  if (! -f /etc/fedora-release) then 
 +    setenv PYTHONUSERBASE $HOME/.local-rhel7 
 +  endif 
 + 
 +And make sure to create that directory ~/.local-rhel7. Now the pip --user commands on RHEL7 machines will install into that newly created directory in stead of the default one used by the desktop systems.
  
-==== virtualenv ====+==== METHOD 2: virtualenv ====
 This guide refers to virtualenv version 12.0.7.  This guide refers to virtualenv version 12.0.7. 
  
-virtualenv is a tool that creates isolated Python environments. A python environment is essentially  a folder which contains copies of all necessary files needed for a Python project to run. In addition each virtual environment will contain a copy of the utility pip to manage packages. For example, let us suppose you would like to install ''pymatlab'' which is not installed on the departmental workstations, then do +virtualenv is a tool that creates isolated Python environments. A python environment is essentially  a folder which contains copies of all necessary files needed for a Python project to run. In addition each virtual environment will contain a copy of the utility pip to manage packages. For example, let us suppose you would like to install ''pymatlab'' which is not installed on the departmental workstations, then you could do 
-<code> +<code bash> 
-$ mkdir PYTHON_VIRT_ENVS && cd PYTHON_VIRT_ENVS +$ mkdir python_virt_envs && cd python_virt_envs 
-$ virtualenv --system-site-packages PYMATLAB+$ virtualenv --system-site-packages pymatlab
 </code> </code>
-will create a virtual environment called PYMATLAB whose contents will be +to create a virtual environment (folder) called pymatlab. 
-<code> + 
-. +Python virtual environments containing specific versions of python can be created using the ''-p'' option as in  ''virtualenv -p /usr/bin/python3.6''. 
-├── bin +
-├── include +
-│   └── python2.7 -> /usr/include/python2.7 +
-├── lib +
-│   └── python2.7 +
-│       ├── config -> /usr/lib64/python2.7/config +
-│       ├── distutils +
-│       ├── encodings -> /usr/lib64/python2.7/encodings +
-│       ├── lib-dynload -> /usr/lib64/python2.7/lib-dynload +
-│       └── site-packages +
-│           ├── _markerlib +
-│           ├── pip +
-│           │   ├── commands +
-│           │   ├── compat +
-│           │   ├── models +
-│           │   ├── operations +
-│           │   ├── req +
-│           │   ├── utils +
-│           │   ├── vcs +
-│           │   └── _vendor +
-│           │       ├── cachecontrol +
-│           │       │   └── caches +
-│           │       ├── certifi +
-│           │       ├── colorama +
-│           │       ├── distlib +
-│           │       │   └── _backport +
-│           │       ├── html5lib +
-│           │       │   ├── filters +
-│           │       │   ├── serializer +
-│           │       │   ├── treeadapters +
-│           │       │   ├── treebuilders +
-│           │       │   ├── treewalkers +
-│           │       │   └── trie +
-│           │       ├── lockfile +
-│           │       ├── _markerlib +
-│           │       ├── packaging +
-│           │       ├── pkg_resources +
-│           │       ├── progress +
-│           │       └── requests +
-│           │           └── packages +
-│           │               ├── chardet +
-│           │               └── urllib3 +
-│           │                   ├── contrib +
-│           │                   ├── packages +
-│           │                   │   └── ssl_match_hostname +
-│           │                   └── util +
-│           ├── pip-6.0.8.dist-info +
-│           ├── pkg_resources +
-│           │   ├── tests +
-│           │   └── _vendor +
-│           │       └── packaging+
  
 +The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing
 +<code bash>
 +source pymatlab/bin/activate # bash shells
 +source pymatlab/bin/activate.csh # c shells
 </code> </code>
-NOTE: Despite they are written using capital letters which are usually used for constants or environment variables, PYTHON_VIRT_ENVS and PYMATLAB are both directories. 
  
-The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to the $PATH environment variable. This is done by issuing +To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ''$PS1'' to  
-<code> +<code bash> 
-source PYMATLAB/bin/activate +(pymatlab)username@hostname:~/python_virt_envs/pymatlab$
-</code>  +
-or  +
-<code> +
-source PYMATLAB/bin/activate.csh+
 </code> </code>
-if you are using csh! +to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment 
-To acknowledge the activation of PYMATLAB, virtualenv will change the terminal prompt ($PS1) to  +<code bash>
-<code> +
-(PYMATLAB)username@hostname:~/PYTHON_VIRT_ENVS/PYMATLAB$ +
-</code> +
-so that you know that you are operating in a virtual environment. To install pymatlab locally (in your virtual environment) run +
-<code>+
 pip install  pymatlab pip install  pymatlab
 </code> </code>
-Your virtual environment should now have the same core python packages defined globally for all the Observatory or Lorentz Institute  users. +Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. 
  
-In any cases, it is advisable to backup your virtual environment configuration by creating a list of installed packages +In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages 
-<code>+<code bash>
 pip freeze > packages.dat pip freeze > packages.dat
 </code> </code>
 This can help collaborators and fellow developers to reproduce your environment with  This can help collaborators and fellow developers to reproduce your environment with 
-<code>+<code bash>
 pip install -r packages.dat pip install -r packages.dat
 </code> </code>
 When you are done working in a virtual environment deactivate it running  When you are done working in a virtual environment deactivate it running 
-<code>+<code bash>
 deactivate  deactivate 
 </code> </code>
 At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system by executing  At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system by executing 
-<code> +<code bash> 
-rm -rf ~/PYTHON_VIRT_ENVS/PYMATLAB+rm -rf ~/python_virt_envs/pymatlab
 </code> </code>
 so do not panic if things do not work, just delete your virtual environment and start all over again. so do not panic if things do not work, just delete your virtual environment and start all over again.
  
-Finally , it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option +Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation 
-<code> +<code bash>
-virtualenv -p /usr/bin/python3.4 PYMATLAB +
-</code> +
-Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised to consult the documentation: +
-<code>+
 virtualenv --help virtualenv --help
 </code> </code>
-==== easy_install ====+ 
 +==== METHOD 3: easy_install with the `--user' option ====
 Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages.  Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. 
-By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so +By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the ''--user'' option in a similar fashion to pip's 
-to install a package locally type +<code bash>
-<code>+
 easy_install -N --user pymatlab easy_install -N --user pymatlab
 </code> </code>
-This will install pymatlab in ~/.local/lib ready to be imported in your next python session. +This will install pymatlab in ''${HOME}/.local/'' ready to be imported in your next python session. 
-Note: If you want to install your package in a different location than ~/.local, then set the environment variable  $PYTHONUSERBASE to a custom location, e.g,  +If you want to install your package in a different location than ''$HOME/.local'', then set the environment variable  $PYTHONUSERBASE to a custom location, e.g,  
-<code> +<code bash> 
-export PYTHONUSERBASE=/home/user/some/where/I/can/write+export PYTHONUSERBASE=/home/user/some/where/I/can/write # alternative location
 </code> </code>
-===== Example: how to let python search arbitrary library paths  ===== 
-Create/edit 
  
-''~/.local/lib/python2.7/site-packages/my-super-library.pth''+Please consult the docs to know more:
  
-by appending the path of your choice, for instance+<code bash> 
 +python -m easy_install --help 
 +</code>
  
-<code> +==== Migrating packages between python versions ==== 
-echo "/my/home/sweet/home/library" >> ~/.local/lib/python2.7/site-packages/my-super-library.pth+Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9). 
 +Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/#_notes_on_migrating_user_installed_pip_packages|Fedora release notes]]: 
 +  - Make a list of installed packages in the old python version: 
 +<code bash>   
 +  python3.7 -m pip freeze > installed.txt 
 </code> </code>
 +  - Reinstall for the current python version:
 +<code bash>
 +  python3.9 -m pip install --user -r installed.txt
 +</code>
 +  - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/.local/lib/python3.7
 +===== Example: how to let python search arbitrary library paths  =====
 +For instance for python v2.7 installations, create or edit
  
-All ''.pth'' files will be sourced by python provided they are in the right location.+''$HOME/.local/lib/python2.7/site-packages/my-super-library.pth''
  
-===== Example: how to create a python environment module  =====+by appending the path of your choice
  
-First enable the module package to search also private module directories 
 <code> <code>
-module load use.own+echo "/my/home/sweet/home/library" >> $HOME/.local/lib/python2.7/site-packages/my-super-library.pth
 </code> </code>
  
-the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files.+All ''.pth'' files will be sourced by python provided they are in the right location.
  
-Let us  now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package +===== Example: how to create your own python environment module  =====
  
-<code> +Please read [[easybuild_environment|here]].
-export PYTHONUSERBASE=/somewhere/I/can/write/to +
-pip install --user SomePackage +
-pip install -I --user SomePackageThatWASInstalledSystemwide +
-</code>+
  
-Create a file, say `$HOME/privatemodules/super-module' with the following contents 
-<code> 
-#%Module 1.0 
-# 
-#   
-#  
-prepend-path  PATH          /somewhere/I/can/write/to/bin # if executables were installed 
-prepend-path  PYTHONPATH    /somewhere/I/can/write/to/lib/python2.7/site-packages 
-</code> 
- 
-and type  
- 
-<code> 
-module load super-module 
-</code> 
-and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. 
 ===== Example: numpy with openBLAS ===== ===== Example: numpy with openBLAS =====
 In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library.  In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. 
  
-:!: The procedure and paths below will work on any maris node and on the para cluster. +:!: The procedure and paths below will work on any maris node. 
  
-<code>+<code bash>
 virtualenv py2_numpy_openBLAS virtualenv py2_numpy_openBLAS
 source py2_numpy_openBLAS/bin/activate source py2_numpy_openBLAS/bin/activate
Line 240: Line 202:
 </code>  </code> 
 Edit ''site.cfg'' with your favorite editor such that  Edit ''site.cfg'' with your favorite editor such that 
-<code>+<code bash>
 [openblas] [openblas]
 libraries = openblas libraries = openblas
Line 248: Line 210:
 </code> </code>
 then install numpy  then install numpy 
-<code>+<code bash>
 python setup.py install python setup.py install
 </code> </code>
 If the installation is going smoothly you should see  If the installation is going smoothly you should see 
-<code>+<code bash>
 .... ....
  
Line 268: Line 230:
 </code> </code>
 Now that numpy is installed you could also install scipy, for instance Now that numpy is installed you could also install scipy, for instance
-<code>+<code bash>
 pip install scipy pip install scipy
 </code> </code>
Line 286: Line 248:
 ====== Bypassing the existing python environment ====== ====== Bypassing the existing python environment ======
 Occasionally, something in the systemwide directories (e.g ''/software/local/lib64/python2.7/site-packages'') interferes with your python application. Perhaps you have a code that requires a specific, older, version of numpy or matplotlib. Just installing that version is not always sufficient. The trick is, to set the PYTHONPATH to point first to a directory where you place a private ''sitecustomize.py'' which then overrides the one we have placed in /usr/lib64/python2.7/site-packages (which is where we add the /software directories to the path for everyone). Here is how: Occasionally, something in the systemwide directories (e.g ''/software/local/lib64/python2.7/site-packages'') interferes with your python application. Perhaps you have a code that requires a specific, older, version of numpy or matplotlib. Just installing that version is not always sufficient. The trick is, to set the PYTHONPATH to point first to a directory where you place a private ''sitecustomize.py'' which then overrides the one we have placed in /usr/lib64/python2.7/site-packages (which is where we add the /software directories to the path for everyone). Here is how:
 +<code bash>
   mkdir /some/location/python_custom_dir   mkdir /some/location/python_custom_dir
   setenv PYTHONPATH /some/location/python_custom_dir:/usr/lib64/python2.7/site-packages   setenv PYTHONPATH /some/location/python_custom_dir:/usr/lib64/python2.7/site-packages
 +</code>
 The ''sitecustomize.py'' could be something like this: The ''sitecustomize.py'' could be something like this:
 +<code python>
   import sys   import sys
   import site   import site
Line 296: Line 261:
   # In order to find also eggs and subdirectories, addsitedir seems necessary:   # In order to find also eggs and subdirectories, addsitedir seems necessary:
   site.addsitedir(mypath, known_paths=None)   site.addsitedir(mypath, known_paths=None)
 +</code>
 +
 +====== Anaconda/Miniconda ======
 +Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls).
 +
 +
 +====== Jupyter Notebooks ======
 +Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing:
 +    python3 -m ipykernel install --user
 +Once this command has run successfully, it will have installed python3 as a jupyter kernel.
 +
 +After starting ''jupyter notebook'' you can select ''python3'' as kernel.
 +
 +If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g:
 +    python3 -m ipykernel install --user --name system-python3
  
  
working_with_python.1519301887.txt.gz · Last modified: 2018/02/22 12:18 by lenocil

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki