User Tools

Site Tools


institute_lorentz:institutelorentz_maris_slurm_jupyterhub

This is an old revision of the document!


JupyterHub with Slurm on Maris

JupyterHub is a multi-user server for Jupyter (AKA IPython) notebooks. Though JupyterHub you can create and share documents that contain live code, equations, visualizations and text notes (comments).

Maris cluster provides a JupyterHub service which uses a dedicated slurm partition accessible only by authorized users. Users should submit access requests via email or preferably in person to support.

JupyterHub spawns single-user notebooks by requesting consumable resources to slurm using precoded parameters in order to prevent erroneous usages. Resources are allocated upon each successful login via the JupyterHub web interface. No notebooks will be spawned in the case of insufficient consumable resources (Memory and CPU).

Access

Maris' JupyterHub can be reached via a web browser at

https://jupyterhub.lorentz.leidenuniv.nl/

Please note that access to the page above is restricted to within the IL network.

:!: Please note that maris' jupyterhub has admin_access enabled. Admin users have the ability to take actions on users' behalf, such as stopping and restarting their servers.

Launching Notebooks

All users that have access to maris will also be able to use JupyterHub. Launching a notebook is as easy as selecting the resources needed from the dropdown menu that appears upon a successful login. Currently, the choices are

  • NOTEBOOK1: NumNodes=1 NumCPUs=4 NumTasks=1 CPUs/Task=4 mem=16G Permanent
  • NOTEBOOK2: NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 mem=8G Permanent
  • NOTEBOOK3: NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 mem=4G Permanent
  • NOTEBOOK4: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G Permanent

Additions to the list above are possible and should be discussed with support.

If the resources requested are available a single-user notebook instance will be launched and you will be ready to run your code interactively.

Jupyterhub will write output and error files relative to the spawning of each notebook instance using the filename scheme ${HOME}/jupyterhub_%u_%j.[log|err], where `HOME' refers to your home directory and %u and %j to your username and slurm job number respectively.

:!: If launching a notebook fails, please read these output files before contacting support.

:!: It has been observed that jupyterhub `forgets' its open sessions when the slurm controller is unreachable for a long period of time. You are highly advised to save the results of your running notebooks to the disk periodically. Failure to do so might make your running notebooks inaccessible via the jupyterhub interface.

Jupyterhub Enabled Extensions

`Files' Tab

The Files tab allows users to create, modify and remove files. Furthermore, it allows to initiate a terminal application and a notebook. Because maris' jupyterhub environment contains only a python3 kernel only python3 notebooks can be instantiated unless you install your own kernels. An example session to create a custom python kernel is given below

# use system's ipython to create a python2 kernel
ipython kernel install --name=my-python-kernel --user
ls ${HOME}/.local/share/jupyter/kernels/
my-python-kernel
# or more appropriately
jupyter kernelspec list

Set the JUPYTER_PATH environment variable to provide extra directories for the kernels search path. JUPYTER_PATH should contain a series of directories, separated by `:'. Directories given in JUPYTER_PATH are searched before other locations.

:!: Users can install their custom python environments (see here) in order to create custom kernels.

`Running' Tab

This tab shows and controls any running jupyter processes, such as notebooks and terminals.

`IPython Clusters' Tab

JupyterHub users can distribute the load of their calculations to different maris' nodes using ipyparallel. This is done by selecting a predefined profile in this tab. Please follow the instructions in this guide to get started after reading our quick slurm tutorial, slurm's setup on maris and http://slurm.schedmd.com/ .

JupyterHub `Conda' Tab

This tab allows users to list and manage conda environments using jupyter's web interface. In maris' setup, conda environments are displayed if they are in the following paths:

export CONDA_ENVS_PATH=/marisdata/$USER/.conda/envs:/home/$USER/.cond/envs${CONDA_ENVS_PATH:+:${CONDA_ENVS_PATH}}

:!: If the creation of a new conda environment through the web interface produces an `xsrf' error, then edit your jupyter configuration file usually located at ${HOME}/.jupyter/jupyter_notebook_config.py adding the line

c.NotebookApp.disable_check_xsrf = True
institute_lorentz/institutelorentz_maris_slurm_jupyterhub.1505981995.txt.gz ยท Last modified: 2017/09/21 08:19 by lenocil