Differences

This shows you the differences between two versions of the page.

--- institute_lorentz:institutelorentz_maris_slurm_jupyterhub [2017/09/21 08:19] – [`IPython Clusters' Tab] lenocil
+++ institute_lorentz:institutelorentz_maris_slurm_jupyterhub [2020/01/15 13:34] (current) – removed lenocil
@@ Line 1: / Line 1: @@
-====== JupyterHub with Slurm on Maris ======
-JupyterHub is a multi-user server for Jupyter (AKA IPython) notebooks. Though JupyterHub you can create and share documents that contain live code, equations, visualizations and text notes (comments).
-Maris cluster provides a JupyterHub service which uses a dedicated ''slurm'' partition accessible only by
-authorized users. Users should submit access requests via email or preferably in person to ''support''.
-JupyterHub spawns single-user notebooks by requesting consumable resources to ''slurm'' using precoded parameters in order to prevent erroneous usages. Resources are allocated upon each successful login via the JupyterHub web interface. No notebooks will be spawned in the case of insufficient consumable resources (Memory and CPU).
-===== Access =====
-Maris' JupyterHub can be reached via a web browser at
-https://jupyterhub.lorentz.leidenuniv.nl/
-Please note that access to the page above is restricted to within the IL network.
-{{ :institute_lorentz:jupyterhub.png?800 |}}
-:!: Please note that maris' jupyterhub has **admin_access enabled**. Admin users  have the ability to take actions on users' behalf, such as stopping and restarting their servers.
-===== Launching Notebooks =====
-All users that have access to maris will also be able to use JupyterHub. Launching a notebook is as easy as selecting the resources needed from the dropdown menu that appears upon a successful login. Currently, the choices are
-  * NOTEBOOK1: NumNodes=1 NumCPUs=4 NumTasks=1 CPUs/Task=4 mem=16G Permanent
-  * NOTEBOOK2: NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 mem=8G Permanent
-  * NOTEBOOK3: NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 mem=4G Permanent
-  * NOTEBOOK4: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G Permanent
-Additions to the list above are possible and should be discussed with ''support''.
-If the resources requested are available a single-user notebook instance will be launched and you will be ready to run your code interactively.
-Jupyterhub will write output and error files relative to the spawning of each notebook instance using the filename scheme ''${HOME}/jupyterhub_%u_%j.[log|err]'', where `HOME' refers to your home directory and %u and %j to your username and slurm job number respectively.
-:!: If launching a notebook fails, please **read these output files before contacting ''support''**.
-:!: It has been observed that jupyterhub `forgets' its open sessions when the slurm controller is unreachable for a long period of time. You are highly advised to save the results of your running notebooks to the disk periodically. Failure to do so might make your running notebooks inaccessible via the jupyterhub interface.
-===== Jupyterhub Enabled Extensions =====
-==== `Files' Tab ====
-The //Files// tab  allows users to create, modify and remove files. Furthermore, it allows to initiate a terminal application and a notebook. Because maris' jupyterhub environment contains only a python3 kernel only python3 notebooks can be instantiated unless you install your own kernels. An example session to create a custom python kernel is given below
-<code>
-# use system's ipython to create a python2 kernel
-ipython kernel install --name=my-python-kernel --user
-ls ${HOME}/.local/share/jupyter/kernels/
-my-python-kernel
-# or more appropriately
-jupyter kernelspec list
-</code>
-Set the JUPYTER_PATH environment variable to provide extra directories for the kernels search path. JUPYTER_PATH should contain a series of directories, separated by `:'. Directories given in JUPYTER_PATH are searched before other locations.
-:!: Users can install their custom python environments (see [[:working_with_python|here]]) in order to create custom kernels.
-==== `Running' Tab ====
-This tab shows and controls  any running jupyter processes, such as notebooks and terminals.
-==== `IPython Clusters' Tab ====
-JupyterHub users can distribute the load of their calculations to different maris' nodes using [[http://ipyparallel.readthedocs.io/en/latest/intro.html|ipyparallel]]. This is done by selecting a predefined profile in this tab. Please follow the instructions in [[|this]] guide to get started after reading our
-[[:slurm_tutorial|quick slurm tutorial]], [[institute_lorentz:institutelorentz_maris_slurm|slurm on maris]] and http://slurm.schedmd.com/ .
-==== JupyterHub `Conda' Tab ====
-This tab allows users to list and manage conda environments using jupyter's web interface. In maris' setup, conda environments are displayed if they are in the following paths:
-<code>
-export CONDA_ENVS_PATH=/marisdata/$USER/.conda/envs:/home/$USER/.cond/envs${CONDA_ENVS_PATH:+:${CONDA_ENVS_PATH}}
-</code>
-:!: If the creation of a new conda environment through the web interface produces an `xsrf' error, then edit your jupyter configuration file usually located at ''${HOME}/.jupyter/jupyter_notebook_config.py'' adding the line
-<code>
-c.NotebookApp.disable_check_xsrf = True
-</code>

Computer Documentation Wiki

User Tools

Site Tools

Differences

Page Tools