Computer Documentation Wiki

User Tools

Site Tools

Trace:
institute_lorentz:institutelorentz_maris_slurm_jupyterhub

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
institute_lorentz:institutelorentz_maris_slurm_jupyterhub [2017/09/21 08:19] – [`IPython Clusters' Tab] lenocilinstitute_lorentz:institutelorentz_maris_slurm_jupyterhub [2018/07/19 15:19] – [Launching Notebooks] lenocil
Line 3: Line 3:
  
 Maris cluster provides a JupyterHub service which uses a dedicated ''slurm'' partition accessible only by  Maris cluster provides a JupyterHub service which uses a dedicated ''slurm'' partition accessible only by 
-authorized users. Users should submit access requests via email or preferably in person to ''support''.+authorized users. Users should request access to the maris cluster as described  [[institute_lorentz:institutelorentz_maris|here]].
  
-JupyterHub spawns single-user notebooks by requesting consumable resources to ''slurm'' using precoded parameters in order to prevent erroneous usages. Resources are allocated upon each successful login via the JupyterHub web interface. No notebooks will be spawned in the case of insufficient consumable resources (Memory and CPU).+JupyterHub spawns single-user notebooks by requesting consumable resources to ''slurm'' using hard-coded parameters that define the different types of notebooks available on the system (see below). Once a user logs in the JupyterHub web interface, the notebook  resources will only be allocated upon initialising the notebook server. No notebooks will be spawned in the case of insufficient consumable resources (Memory and CPU). Users are advised to check the output of the command ''squeue -u <username>'' in case no spawn occurs in a reasonable amount of time.
  
 ===== Access ===== ===== Access =====
Line 16: Line 16:
 {{ :institute_lorentz:jupyterhub.png?800 |}} {{ :institute_lorentz:jupyterhub.png?800 |}}
  
-:!: Please note that maris' jupyterhub has **admin_access enabled**. Admin users  have the ability to take actions on users' behalf, such as stopping and restarting their servers. +:!: Maris' jupyterhub has the feature **admin_access** enabled. This means admin users  have the ability to take actions on users' behalf, such as stopping and restarting their servers. 
  
 ===== Launching Notebooks ===== ===== Launching Notebooks =====
Line 22: Line 22:
 All users that have access to maris will also be able to use JupyterHub. Launching a notebook is as easy as selecting the resources needed from the dropdown menu that appears upon a successful login. Currently, the choices are All users that have access to maris will also be able to use JupyterHub. Launching a notebook is as easy as selecting the resources needed from the dropdown menu that appears upon a successful login. Currently, the choices are
  
-  * NOTEBOOK1: NumNodes=1 NumCPUs=4 NumTasks=1 CPUs/Task=4 mem=16G Permanent +  * NOTEBOOK1: NumNodes=1 NumCPUs=4 NumTasks=1 CPUs/Task=4 mem=16G 30 days 
-  * NOTEBOOK2: NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 mem=8G Permanent +  * NOTEBOOK2: NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 mem=8G 30 days 
-  * NOTEBOOK3: NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 mem=4G Permanent +  * NOTEBOOK3: NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 mem=4G 30 days 
-  * NOTEBOOK4: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G Permanent+  * NOTEBOOK4: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G 10 days 
 +  * NOTEBOOK5: NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 mem=64G 3Days 2 GPUs
  
 Additions to the list above are possible and should be discussed with ''support''. Additions to the list above are possible and should be discussed with ''support''.
Line 32: Line 33:
 If the resources requested are available a single-user notebook instance will be launched and you will be ready to run your code interactively. If the resources requested are available a single-user notebook instance will be launched and you will be ready to run your code interactively.
  
-Jupyterhub will write output and error files relative to the spawning of each notebook instance using the filename scheme ''${HOME}/jupyterhub_%u_%j.[log|err]'', where `HOME' refers to your home directory and %u and %j to your username and slurm job number respectively. +Jupyterhub will write output and error files relative to the spawning of each notebook instance using the filename scheme ''${HOME}/jupyterhub_%u_%j.[log|err]'', where `HOME' refers to your home directory and %u and %j to your username and slurm job number respectively. Because maris' home disk has an enforced quota of 10GB per user, **please remove old log files to save space**.
  
 :!: If launching a notebook fails, please **read these output files before contacting ''support''**. :!: If launching a notebook fails, please **read these output files before contacting ''support''**.
Line 63: Line 64:
 ==== `IPython Clusters' Tab ==== ==== `IPython Clusters' Tab ====
  
-JupyterHub users can distribute the load of their calculations to different maris' nodes using [[http://ipyparallel.readthedocs.io/en/latest/intro.html|ipyparallel]]. This is done by selecting a predefined profile in this tab. Please follow the instructions in [[|this]] guide to get started after reading our +JupyterHub users can distribute the load of their calculations to different maris' nodes using [[http://ipyparallel.readthedocs.io/en/latest/intro.html|ipyparallel]]. This is done by selecting a predefined profile in this tab. Please follow the instructions in [[institute_lorentz:slurm_ipyparallel|this]] guide to get started after reading our 
-[[:slurm_tutorial|quick slurm tutorial]], [[institute_lorentz:institutelorentz_maris_slurm|slurm on maris]] and http://slurm.schedmd.com/ .+[[:slurm_tutorial|quick slurm tutorial]], [[institute_lorentz:institutelorentz_maris_slurm|slurm's setup  on maris]] and http://slurm.schedmd.com/ .
  
  
Line 71: Line 72:
  
 <code> <code>
-export CONDA_ENVS_PATH=/marisdata/$USER/.conda/envs:/home/$USER/.cond/envs${CONDA_ENVS_PATH:+:${CONDA_ENVS_PATH}}+/marisdata/${USER}/.conda/envs 
 +${HOME}/.conda/envs
 </code> </code>
  
  
-:!: If the creation of a new conda environment through the web interface produces an `xsrf' error, then edit your jupyter configuration file usually located at ''${HOME}/.jupyter/jupyter_notebook_config.py'' adding the line+:!: If the creation of a new conda environment through the web interface produces an `xsrf' error, then edit your jupyter configuration file  located at ''${HOME}/.jupyter/jupyter_notebook_config.py'' adding the line
 <code> <code>
 c.NotebookApp.disable_check_xsrf = True c.NotebookApp.disable_check_xsrf = True
 </code> </code>
  
 +===== Matlab on maris jupyterhub =====
  
 +The Leiden Institute of Physics (LION) provides a certain number of matlab licenses available to its users. Here follows a simple session that will enable your notebooks to use matlab's engine
 +
 +<code>
 +cd /software/matlab/matlab-R2017a/extern/engines/python
 +python3 setup.py build --build-base="/marisdata/<username>" install --user
 +pip3 install imatlab --user
 +python3 -m imatlab install --user
 +# check that the matlab kernel was installed
 +jupyter kernelspec list
 +Available kernels:
 +  python3    
 +  imatlab    
 +</code>
 +
 +Then as usual log on to https://jupyterhub.lorentz.leidenuniv.nl select the appropriate notebook profile and then the matlab kernel upon creating a new notebook.
 +
 +:!: Be aware that if there are NOT any matlab licenses available you will not be able to initiate the matlab kernel.
 +
 +:!: license availability can be checked using
 +
 +<code>
 +license matlab
 +</code>
 +
 +See also: https://github.com/imatlab/imatlab

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki