Differences

This shows you the differences between two versions of the page.

--- working_with_python [2020/10/05 10:32] – [Example: how to create a python environment module] lenocil
+++ working_with_python [2024/04/22 10:51] (current) – jansen
@@ Line 81: / Line 81: @@
     export PYTHONUSERBASE=$HOME/.local-rhel7
   fi
-For users of the ''tcsh'' shell, add this to your .cshrc in stead:
+For users of the ''tcsh'' shell, add this to your .tcshrc in stead:
   if (! -f /etc/fedora-release) then
     setenv PYTHONUSERBASE $HOME/.local-rhel7
@@ Line 88: / Line 88: @@
 And make sure to create that directory ~/.local-rhel7. Now the pip --user commands on RHEL7 machines will install into that newly created directory in stead of the default one used by the desktop systems.
-==== METHOD 2: virtualenv ====
+==== METHOD 2: venv ====
+''venv'' is a tool that creates isolated Python environments; it replaces the obsolete ''virtualenv'' that provided similar functionality for python 2.x. A python environment is essentially  a folder which contains copies of all necessary files needed for a Python project to run. In addition each virtual environment will contain a copy of the utility pip to manage packages. For example, let us suppose you would like to install ''pymatlab'' which is not installed on the departmental workstations, then you could do
+<code bash>
+$ mkdir /data2/username/venvs
+$ python3 -m venv /data2/username/venvs/pymatlab
+</code>
+to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/data2'' disk, in order to avoid running out of disk quota in your home directory, which can easily happen since venvs can become rather big).
+In the example, we use ''python3'' as the python for this environment; if there are multiple python versions on the system, and you want to base your venv on a specific version, use that version to create the venv, e.g. ''python3.12 -m venv /data2/username/venvs/pymatlab''.
+The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing
+<code bash>
+source /data2/username/pymatlab/bin/activate # bash shells
+source /data2/username/pymatlab/bin/activate.csh # c shells
+</code>
+To acknowledge the activation of pymatlab, the terminal prompt will be changed to
+<code bash>
+(pymatlab)username@hostname:~/python_virt_envs/pymatlab$
+</code>
+to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment
+<code bash>
+pip install  pymatlab
+</code>
+Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment.
+Note that you do NOT use ''--user'' on the pip command in this case, since that would install in your ''$PYTHONUSERBASE'' directory (see above) instead of the venv!!
+In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages
+<code bash>
+pip freeze > packages.dat
+</code>
+This can help collaborators and fellow developers to reproduce your environment with
+<code bash>
+pip install -r packages.dat
+</code>
+When you are done working in a virtual environment deactivate it running
+<code bash>
+deactivate
+</code>
+At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system so do not panic if things do not work, just delete your virtual environment and start all over again.
+Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation.
+==== METHOD 2: OBSOLETE: virtualenv (python 2.x) ====
 This guide refers to virtualenv version 12.0.7.
@@ Line 138: / Line 182: @@
 virtualenv --help
 </code>
 ==== METHOD 3: easy_install with the `--user' option ====
 Easy Install is a python module (easy_install) that lets you automatically download, build, install, and manage Python packages.
@@ Line 155: / Line 200: @@
 python -m easy_install --help
 </code>
+==== Migrating packages between python versions ====
+Another issue when using personal installs may arrise on operating system upgrades, when a newer version of python is made the default (eg, moving from python 3.7 to python 3.9).
+Notes copied from the [[https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/developers/Development_Python/#_notes_on_migrating_user_installed_pip_packages|Fedora release notes]]:
+  - Make a list of installed packages in the old python version:
+<code bash>
+  python3.7 -m pip freeze > installed.txt
+</code>
+  - Reinstall for the current python version:
+<code bash>
+  python3.9 -m pip install --user -r installed.txt
+</code>
+  - Optionally, uninstall the packages from the old python version and/or remove the obsolete directory under $HOME/.local/lib/python3.7
 ===== Example: how to let python search arbitrary library paths  =====
 For instance for python v2.7 installations, create or edit
@@ Line 175: / Line 233: @@
 In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library.
-:!: The procedure and paths below will work on any maris node and on the para cluster.
+:!: The procedure and paths below will work on any maris node.
 <code bash>
@@ Line 251: / Line 309: @@
 ====== Anaconda/Miniconda ======
 Another way of using a private python install (separate versions etc), is to install and use [[conda|Anaconda/Miniconda]]. Since these environments can encompass much more than just python, they deserve their own page (especially since they come with their own share of pitfalls).
+====== Jupyter Notebooks ======
+Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing:
+    python3 -m ipykernel install --user
+Once this command has run successfully, it will have installed python3 as a jupyter kernel.
+After starting ''jupyter notebook'' you can select ''python3'' as kernel.
+If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g:
+    python3 -m ipykernel install --user --name system-python3

Computer Documentation Wiki

User Tools

Site Tools

Differences

Page Tools