Computer Documentation Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== JupyterHub with Slurm on Maris ======
JupyterHub is a multi-user server for Jupyter (AKA IPython) notebooks. Though JupyterHub you can create and share documents that contain live code, equations, visualizations and text notes (comments). 

Maris cluster provides a JupyterHub service which uses a dedicated ''slurm'' partition accessible only by 
authorized users. Users should request access to the maris cluster as described  [[institute_lorentz:institutelorentz_maris|here]].

JupyterHub spawns single-user notebooks by requesting consumable resources to ''slurm'' using hard-coded parameters that define the different types of notebooks available on the system (see below). Once a user logs in the JupyterHub web interface, the notebook  resources will only be allocated upon initialising the notebook server. No notebooks will be spawned in the case of insufficient consumable resources (Memory and CPU). Users are advised to check the output of the command ''squeue -u <username>'' in case no spawn occurs in a reasonable amount of time.

===== Access =====
Maris' JupyterHub can be reached via a web browser at 

https://jupyterhub.lorentz.leidenuniv.nl/

Please note that access to the page above is restricted to within the IL network.

{{ :institute_lorentz:jupyterhub.png?800 |}}

:!: Maris' jupyterhub has the feature **admin_access** enabled. This means admin users  have the ability to take actions on users' behalf, such as stopping and restarting their servers. 

===== Launching Notebooks =====

All users that have access to maris will also be able to use JupyterHub. Launching a notebook is as easy as selecting the resources needed from the dropdown menu that appears upon a successful login. Currently, the choices are

  * NOTEBOOK1: NumNodes=1 NumCPUs=4 NumTasks=1 CPUs/Task=4 mem=16G 30 days
  * NOTEBOOK2: NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 mem=8G 30 days
  * NOTEBOOK3: NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 mem=4G 30 days
  * NOTEBOOK4: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G 10 days
  * NOTEBOOK5: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G 3Days 2 GPUs

Additions to the list above are possible and should be discussed with ''support''.


If the resources requested are available a single-user notebook instance will be launched and you will be ready to run your code interactively.

Jupyterhub will write output and error files relative to the spawning of each notebook instance using the filename scheme ''${HOME}/jupyterhub_%u_%j.[log|err]'', where `HOME' refers to your home directory and %u and %j to your username and slurm job number respectively. Because maris' home disk has an enforced quota of 10GB per user, **please remove old log files to save space**.

:!: If launching a notebook fails, please **read these output files before contacting ''support''**.

:!: It has been observed that jupyterhub `forgets' its open sessions when the slurm controller is unreachable for a long period of time. You are highly advised to save the results of your running notebooks to the disk periodically. Failure to do so might make your running notebooks inaccessible via the jupyterhub interface. 

===== Jupyterhub Enabled Extensions =====

==== `Files' Tab ====
The //Files// tab  allows users to create, modify and remove files. Furthermore, it allows to initiate a terminal application and a notebook. Because maris' jupyterhub environment contains only a python3 kernel only python3 notebooks can be instantiated unless you install your own kernels. An example session to create a custom python kernel is given below

<code>
# use system's ipython to create a python2 kernel
ipython kernel install --name=my-python-kernel --user
ls ${HOME}/.local/share/jupyter/kernels/
my-python-kernel
# or more appropriately
jupyter kernelspec list
</code>


Set the JUPYTER_PATH environment variable to provide extra directories for the kernels search path. JUPYTER_PATH should contain a series of directories, separated by `:'. Directories given in JUPYTER_PATH are searched before other locations.

:!: Users can install their custom python environments (see [[:working_with_python|here]]) in order to create custom kernels.

==== `Running' Tab ====

This tab shows and controls  any running jupyter processes, such as notebooks and terminals.

==== `IPython Clusters' Tab ====

JupyterHub users can distribute the load of their calculations to different maris' nodes using [[http://ipyparallel.readthedocs.io/en/latest/intro.html|ipyparallel]]. This is done by selecting a predefined profile in this tab. Please follow the instructions in [[institute_lorentz:slurm_ipyparallel|this]] guide to get started after reading our
[[:slurm_tutorial|quick slurm tutorial]], [[institute_lorentz:institutelorentz_maris_slurm|slurm's setup  on maris]] and http://slurm.schedmd.com/ .


==== JupyterHub `Conda' Tab ====
This tab allows users to list and manage conda environments using jupyter's web interface. In maris' setup, conda environments are displayed if they are in the following paths:

<code>
/marisdata/${USER}/.conda/envs
${HOME}/.conda/envs
</code>


:!: If the creation of a new conda environment through the web interface produces an `xsrf' error, then edit your jupyter configuration file  located at ''${HOME}/.jupyter/jupyter_notebook_config.py'' adding the line
<code>
c.NotebookApp.disable_check_xsrf = True
</code>

===== Matlab on maris jupyterhub =====

The Leiden Institute of Physics (LION) provides a certain number of matlab licenses available to its users. Here follows a simple session that will enable your notebooks to use matlab's engine

<code>
cd /software/matlab/matlab-R2017a/extern/engines/python
python3 setup.py build --build-base="/marisdata/<username>" install --user
pip3 install imatlab --user
python3 -m imatlab install --user
# check that the matlab kernel was installed
jupyter kernelspec list
Available kernels:
  python3    
  imatlab    
</code>

Then as usual log on to https://jupyterhub.lorentz.leidenuniv.nl select the appropriate notebook profile and then the matlab kernel upon creating a new notebook.

:!: Be aware that if there are NOT any matlab licenses available you will not be able to initiate the matlab kernel.

:!: license availability can be checked using

<code>
license matlab
</code>

See also: https://github.com/imatlab/imatlab
Computer Documentation Wiki

User Tools

Site Tools

Page Tools
