Computer Documentation Wiki

User Tools

Site Tools

Trace:
working_with_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
working_with_python [2020/10/05 09:14] – [METHOD 1: pip with the `--user' option] lenocilworking_with_python [2023/11/06 13:12] (current) – [Jupyter Notebooks] jansen
Line 40: Line 40:
  
 we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process. we advise you follow one of the methods below to install the package yourself. Rest assured though that we can always assist you during the process.
-==== METHOD 1: pip with the `--user' option ==== 
  
-Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. A user's default install location is defined through the `site' module with the variable ''site.USER_BASE''. In the GNU/Linux OS  ''site.USER_BASE'' defaults to ''~/.local''. +:!: The following methods are valid either you use a local (to your workstation/server) python installation or a python installation provided via the ''module'' command. If you choose the latter, remember to load first an appropriate python module.
  
 +==== METHOD 1: pip with the `--user' option ====
  
-This mode of installation can be turned on by specifying the `--user' option to pip install, for instance+Python 2.6 introduced the possibility of package installations via a "user scheme". According to this scheme, Python distributions support an alternative install location that is specific to a user. Python provides this functionality via the ''site'' module which commands where you, as a user, will be installing python packages
  
 <code bash> <code bash>
-pip install --user SomePackage+python -m site --user-base # value of site.USER_BASE 
 +python -m site --user-site # path to your site-packages directory 
 </code> </code>
  
-To display the value of `site.USER_BASE', type +If the values returned by the command above satisfy you, you can then proceed to install packages in your user-space
 <code bash> <code bash>
-python[2,3] -m site --user-base+pip install --user SomePackage
 </code> </code>
  
-and to show the path to your site-packages directory  +In the STRW and IL environments, site.USER_BASE  defaults to  ''$HOME/.local''. This path can be customised/updated by modifying the environment variable ''PYTHONUSERBASE''
- +
-<code bash> +
-python[2,3] -m site --user-site +
-</code>+
  
-In the STRW and IL environments the site.USER_BASE variable defaults to a user's ''~/.local'' directory nonetheless it can be customised/updated by modifying the environment variable PYTHONUSERBASE: 
 <code bash> <code bash>
-# in bash +export PYTHONUSERBASE=/somewhere/I/can/write/to # alternative location
-export PYTHONUSERBASE=/somewhere/I/can/write/to+
 pip install --user SomePackage pip install --user SomePackage
 </code> </code>
-will install `SomePackage' in ''/somewhere/I/can/write/to''.+will install `SomePackage' in ''/somewhere/I/can/write/to/lib/python*/site-packages''.
  
 When using the `user' scheme to install packages, it is important to note When using the `user' scheme to install packages, it is important to note
Line 75: Line 69:
   * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.   * When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
   * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied.   * When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied.
-  * pip will not perform a --user install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages.+  * pip will not perform a ''--user'' install in a virtualenv unless the virtualenv was created specifying ''--system-site-packages''. Nonetheless, pip will never install a package that conflicts with a package in the virtualenv site-packages.
  
 === Method 1 - subsection Incompatible versions === === Method 1 - subsection Incompatible versions ===
Line 87: Line 81:
     export PYTHONUSERBASE=$HOME/.local-rhel7     export PYTHONUSERBASE=$HOME/.local-rhel7
   fi   fi
-For users of the ''tcsh'' shell, add this to your .cshrc in stead:+For users of the ''tcsh'' shell, add this to your .tcshrc in stead:
   if (! -f /etc/fedora-release) then   if (! -f /etc/fedora-release) then
     setenv PYTHONUSERBASE $HOME/.local-rhel7     setenv PYTHONUSERBASE $HOME/.local-rhel7
Line 104: Line 98:
 to create a virtual environment (folder) called pymatlab. to create a virtual environment (folder) called pymatlab.
  
-Please note that your newly created virtual environment will be a `python2' one if you used `virtualenv' or a +Python virtual environments containing specific versions of python can be created using the ''-p'' option as in  ''virtualenv -p /usr/bin/python3.6''. 
- `python3' one if using `virtualenv-3'. Note that if the command `virtualenv-3' is missing you could use ''virtualenv -p /usr/bin/python3''. +
  
-The last step before starting to use the newly generated environment is to activate it, that is to prepend its /bin folder to your $PATH environment variable. This is done by issuing+The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing
 <code bash> <code bash>
-source pymatlab/bin/activate +source pymatlab/bin/activate # bash shells 
-</code>  +source pymatlab/bin/activate.csh # c shells
-or  +
-<code bash> +
-source pymatlab/bin/activate.csh+
 </code> </code>
-if you are using csh! + 
-To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ($PS1) to +To acknowledge the activation of pymatlab, virtualenv will change the terminal prompt ''$PS1'' to 
 <code bash> <code bash>
 (pymatlab)username@hostname:~/python_virt_envs/pymatlab$ (pymatlab)username@hostname:~/python_virt_envs/pymatlab$
 </code> </code>
-to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run+to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment
 <code bash> <code bash>
 pip install  pymatlab pip install  pymatlab
 </code> </code>
-Your virtual environment should now have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. +Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment. 
  
-In any cases, it is advisable to keep a backup of your virtual environment configuration by creating a list of installed packages+In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages
 <code bash> <code bash>
 pip freeze > packages.dat pip freeze > packages.dat
Line 144: Line 134:
 so do not panic if things do not work, just delete your virtual environment and start all over again. so do not panic if things do not work, just delete your virtual environment and start all over again.
  
-Finally, it possible to choose which python interpreter to use in your virtual environment and that is done by running virtualenv with the `p' option +Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation
-<code bash> +
-virtualenv -p /usr/bin/python3.4 pymatlab +
-</code> +
-Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised to consult the documentation:+
 <code bash> <code bash>
 virtualenv --help virtualenv --help
 </code> </code>
-==== METHOD 3: easy_install ====+ 
 +==== METHOD 3: easy_install with the `--user' option ====
 Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages.  Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages. 
-By default, easy_install installs python packages into Python’s main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location, so +By default, easy_install installs python packages into Python's main site-packages directory, and manages them using a custom .pth file in that same directory. Very often though, a user or developer wants easy_install to install and manage python packages in an alternative location. This possible via the ''--user'' option in a similar fashion to pip's
-to install a package locally type+
 <code bash> <code bash>
 easy_install -N --user pymatlab easy_install -N --user pymatlab
 </code> </code>
-This will install pymatlab in ~/.local/lib ready to be imported in your next python session. +This will install pymatlab in ''${HOME}/.local/'' ready to be imported in your next python session. 
-Note: If you want to install your package in a different location than ~/.local, then set the environment variable  $PYTHONUSERBASE to a custom location, e.g, +If you want to install your package in a different location than ''$HOME/.local'', then set the environment variable  $PYTHONUSERBASE to a custom location, e.g, 
 <code bash> <code bash>
-export PYTHONUSERBASE=/home/user/some/where/I/can/write+export PYTHONUSERBASE=/home/user/some/where/I/can/write # alternative location
 </code> </code>
  
Line 170: Line 156:
 python -m easy_install --help python -m easy_install --help
 </code> </code>
 +
 +==== Migrating packages between python versions ====
 +Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9).
 +Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/#_notes_on_migrating_user_installed_pip_packages|Fedora release notes]]:
 +  - Make a list of installed packages in the old python version:
 +<code bash>  
 +  python3.7 -m pip freeze > installed.txt 
 +</code>
 +  - Reinstall for the current python version:
 +<code bash>
 +  python3.9 -m pip install --user -r installed.txt
 +</code>
 +  - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/.local/lib/python3.7
 ===== Example: how to let python search arbitrary library paths  ===== ===== Example: how to let python search arbitrary library paths  =====
-Create/edit+For instance for python v2.7 installations, create or edit
  
-''~/.local/lib/python2.7/site-packages/my-super-library.pth''+''$HOME/.local/lib/python2.7/site-packages/my-super-library.pth''
  
-by appending the path of your choice, for instance+by appending the path of your choice
  
 <code> <code>
-echo "/my/home/sweet/home/library" >> ~/.local/lib/python2.7/site-packages/my-super-library.pth+echo "/my/home/sweet/home/library" >> $HOME/.local/lib/python2.7/site-packages/my-super-library.pth
 </code> </code>
  
 All ''.pth'' files will be sourced by python provided they are in the right location. All ''.pth'' files will be sourced by python provided they are in the right location.
  
-===== Example: how to create a python environment module  =====+===== Example: how to create your own python environment module  =====
  
-First enable the module package to search also private module directories +Please read [[easybuild_environment|here]].
-<code bash> +
-module load use.own +
-</code>+
  
-the line above will create a $HOME/privatemodules if it does not exist and its path will be searched for the presence of environment modules files. 
- 
-Let us  now install some packages to an arbitrary location and upgrade (__only in $PYTHONUSERBASE__) an already system-wide installed package  
- 
-<code bash> 
-export PYTHONUSERBASE=/somewhere/I/can/write/to 
-pip install --user SomePackage 
-pip install -I --user SomePackageThatWASInstalledSystemwide 
-</code> 
- 
-Create a file, say `$HOME/privatemodules/super-module' with the following contents 
-<code bash> 
-#%Module 1.0 
-# 
-#   
-#  
-prepend-path  PATH          /somewhere/I/can/write/to/bin # if executables were installed 
-prepend-path  PYTHONPATH    /somewhere/I/can/write/to/lib/python2.7/site-packages 
-</code> 
- 
-and type  
- 
-<code bash> 
-module load super-module 
-</code> 
-and you are ready to use your newly created python environment. Note that is similar procedure can be repeated using python3. 
 ===== Example: numpy with openBLAS ===== ===== Example: numpy with openBLAS =====
 In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library.  In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. 
  
-:!: The procedure and paths below will work on any maris node and on the para cluster. +:!: The procedure and paths below will work on any maris node. 
  
 <code bash> <code bash>
Line 295: Line 265:
 ====== Anaconda/Miniconda ====== ====== Anaconda/Miniconda ======
 Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls). Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls).
 +
 +
 +====== Jupyter Notebooks ======
 +Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing:
 +    python3 -m ipykernel install --user
 +Once this command has run successfully, it will have installed python3 as a jupyter kernel.
 +
 +After starting ''jupyter notebook'' you can select ''python3'' as kernel.
 +
 +If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g:
 +    python3 -m ipykernel install --user --name system-python3
 +
 +
working_with_python.1601889279.txt.gz · Last modified: 2020/10/05 09:14 by lenocil

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki