Computer Documentation Wiki

User Tools

Site Tools

Trace:
institute_lorentz:irods_fair_storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
institute_lorentz:irods_fair_storage [2022/10/20 07:29] – [FAIR Storage Access] lenocilinstitute_lorentz:irods_fair_storage [2023/06/13 07:36] (current) – lenocil
Line 1: Line 1:
 ====== FAIR Storage Service ====== ====== FAIR Storage Service ======
- 
-|:!: DRAFT: Doc in preparation | 
-|Comments, suggestions and corrections are really appreciated | 
  
 Lorentz Institute offers its members a data storage option that complies with the //**FAIR** Guiding Principles for scientific data management and stewardship.//((https://www.go-fair.org/fair-principles/)) The goal is to offer a standard platform that enables, facilitates, and promotes the implementation of robust data-management plans complying with the most stringent professional requirements (scientific publications, research grants applications, etc..).   Lorentz Institute offers its members a data storage option that complies with the //**FAIR** Guiding Principles for scientific data management and stewardship.//((https://www.go-fair.org/fair-principles/)) The goal is to offer a standard platform that enables, facilitates, and promotes the implementation of robust data-management plans complying with the most stringent professional requirements (scientific publications, research grants applications, etc..).  
Line 18: Line 15:
 ===== FAIR Storage Access ===== ===== FAIR Storage Access =====
  
-Access to this server must be requested by sending an email to <support@lorentz.leidenuniv.nl>.+Writing access to this server must be requested by sending an email to <support@lorentz.leidenuniv.nl>.
  
 Once access has been granted, you can interact with the server via any of the [[https://irods.org/clients/|supported iRODS clients]]. For the sake of simplicity, we have set up all the IL GNU/Linux workstations and the xmaris cluster with the following iRODS clients so that you can start using our storage system immediately Once access has been granted, you can interact with the server via any of the [[https://irods.org/clients/|supported iRODS clients]]. For the sake of simplicity, we have set up all the IL GNU/Linux workstations and the xmaris cluster with the following iRODS clients so that you can start using our storage system immediately
Line 24: Line 21:
   * [[https://github.com/irods/irods_client_icommands|iCommands]]   * [[https://github.com/irods/irods_client_icommands|iCommands]]
   * [[https://github.com/UtrechtUniversity/davrods|DavRODS]]   * [[https://github.com/UtrechtUniversity/davrods|DavRODS]]
 +
 +Additionally our [[https://remote.lorentz.leidenuniv.nl|remote desktop system]] is also able to access our iRODS server directly from its OpenOnDemand files app using [[https://rclone.org/]] as described [[institute_lorentz:irods_fair_storage#rclone_access|here]]. 
  
 iCommands is the most flexible and powerful client, but it might require a basic prior knowledge of the GNU/Linux command line.  iCommands is the most flexible and powerful client, but it might require a basic prior knowledge of the GNU/Linux command line. 
Line 34: Line 33:
   * custom ingestion rules for your data? Contact <support@lorentz.leidenuniv.nl>   * custom ingestion rules for your data? Contact <support@lorentz.leidenuniv.nl>
  
-:!: Please note that the automatic post-processing functionality has currently been tested only with iCommands and DavRODS.+:!: Please note that the automatic post-processing functionality has currently been tested __only__ with iCommands and DavRODS.
 ==== iCommands Setup ==== ==== iCommands Setup ====
  
Line 75: Line 74:
 Please not that even if you specify a default resource name in your iRODS configuration file, it is always possible to overwrite the resource destination during iCommands operations by specifying the option ''-R''. Please not that even if you specify a default resource name in your iRODS configuration file, it is always possible to overwrite the resource destination during iCommands operations by specifying the option ''-R''.
  
-In a terminal window, load the iCommands module +In a terminal window of your GNU/Linux workstation or the xmaris cluster, load the iCommands module 
  
 <code bash> <code bash>
Line 164: Line 163:
  
 == Metadata == == Metadata ==
 +Metadata are one of the most important features of iRODS that makes this software compliant  with the //FAIR// principles. We advice you spend some time familiarize yourself with the concept of metadata and you get used to the idea that data without metadata is in practice incomplete data.
 +
 iRODS metadata are defined by Attribute-Value-Unit (AVU) triplets, for instance  iRODS metadata are defined by Attribute-Value-Unit (AVU) triplets, for instance 
 Length 10 meters Length 10 meters
Line 184: Line 185:
 |Grant Other IL User Recursive READ Access to Collection | ichmod -r read IL_USERNAME collection_path|  |Grant Other IL User Recursive READ Access to Collection | ichmod -r read IL_USERNAME collection_path| 
 |List Existing Groups |iquest "select USER_GROUP_NAME"| |List Existing Groups |iquest "select USER_GROUP_NAME"|
 +
 +== Integrity: Checksums ==
 +^More Info:^ **iput -h, irsync -h, ichksum -h**^
 +|Check Object Integrity during Transfer| iput -[r]K object|
 +|Check Object Integrity during Transfer| irsync -[r]K source i:dest_object|
 +
  
 == External Collaborators == == External Collaborators ==
Line 189: Line 196:
 iRODS lets you easily share your data with external collaborators (users unknown to the IL systems). The only requirement is that they have access to iCommands or have a web browser. See the examples below iRODS lets you easily share your data with external collaborators (users unknown to the IL systems). The only requirement is that they have access to iCommands or have a web browser. See the examples below
  
-iCommands: Tickets +__iCommands: Tickets__
  
 iRODS tickets constitute a powerful and flexible way to share your data with external collaborators who have access to iCommands at their institutions. In the example session below an IL user creates a //read-only// access ticket for a data object called  //results.dat// and shares this unique alphanumeric code with his collaborator who will use it to gain access to the data. iRODS tickets constitute a powerful and flexible way to share your data with external collaborators who have access to iCommands at their institutions. In the example session below an IL user creates a //read-only// access ticket for a data object called  //results.dat// and shares this unique alphanumeric code with his collaborator who will use it to gain access to the data.
Line 229: Line 236:
 </code> </code>
  
-If your collaborator has no access to icommands, then place the object you would like to share in the folder ''/ilZone/home/anonymous'' and //ichmod// to give the user anonymous reading access to your object which will be shared using the web browser at https://access.lorentz.leidenuniv.nl/anon .+If your collaborator has no access to icommands, then place the object you would like to share in the folder ''/ilZone/home/anonymous'' and //ichmod// to give the user anonymous reading access to your object which will be shared using the web browser at https://access.lorentz.leidenuniv.nl/anon/anonymous .
  
-== Integrity: Checksums == 
-^More Info:^ **iput -h, irsync -h, ichksum -h**^ 
-|Check Object Integrity during Transfer| iput -[r]K object| 
-|Check Object Integrity during Transfer| irsync -[r]K source i:dest_object| 
  
 +__Web Browsers: webdav__
  
-= Web Browsers: webdav = +Without creating a ticket for a data object, you could login to our //FAIR// storage system place any object you would like to share in a special collection called ''/ilZone/home/anonymous''. Any external collaborator can then access it (READ ONLY) by browsing to https://access.lorentz.leidenuniv.nl/anon/anonymous in a similar fashion to the example [[#webdav_access|here]]. 
 + 
 +Access via the ticket system using a web browser will follow the scheme  
 +<code> 
 +https://access.lorentz.leidenuniv.nl/anon/<YOUR_HOME_PATH_TO_DATA_TO_BE_SHARED>?ticket=<TICKET_ID> 
 +# example  
 +https://access.lorentz.leidenuniv.nl/anon/home/bongo/my/Data4You?ticket=hT6Yji111 
 +</code>
  
-Without creating a ticket for a data object, you could login to our //FAIR// storage system place any object you would like to share in a special collection called ''/ilZone/home/public''. Any external collaborator can then access it (READ ONLY) by browsing to https://access.lorentz.leidenuniv.nl/anon in a similar fashion to the example [[#webdav_access|here]]. 
  
 === Custom Lorentz Institute Ingestion Rules === === Custom Lorentz Institute Ingestion Rules ===
Line 261: Line 271:
 </code> </code>
  
-and uploaded it to the IL storage server via DevRODS or iCommands, for instance+and upload it to the IL storage server via DevRODS or iCommands, for instance
  
 <code bash> <code bash>
Line 275: Line 285:
 |version| Data Versioning Number | //0.0.2//| |version| Data Versioning Number | //0.0.2//|
 |tags|Keywords to your Data| //Blackhole Gravity Quantum Computers//| |tags|Keywords to your Data| //Blackhole Gravity Quantum Computers//|
-|doi | If related to published material, Digital object Identifier((https://www.doi.org/))| //10.1103/PhysRevD.97.043511//|+|dois | If related to published material, Digital object Identifier((https://www.doi.org/))| //10.1103/PhysRevD.97.043511//|
 |pi| Name of Principal Investigator| //John B. Smith//| |pi| Name of Principal Investigator| //John B. Smith//|
 |funder| Name organization funding this research | //NWO//| |funder| Name organization funding this research | //NWO//|
Line 288: Line 298:
 === Web Browser Access === === Web Browser Access ===
  
-|URL | https://access.lorentz.leidenuniv.nl|+|URL | https://access.lorentz.leidenuniv.nl/davrods|
 |Username| IL Username| |Username| IL Username|
 |Password| IL Password| |Password| IL Password|
Line 296: Line 306:
 Anonymous access Anonymous access
  
-|URL | https://access.lorentz.leidenuniv.nl/anon|+|URL | https://access.lorentz.leidenuniv.nl/anon/anonymous|
  
  
Line 310: Line 320:
 Menu -> Places -> Connect to Server((This may vary depending on the OS. If in trouble search for Connect To Server.)) Menu -> Places -> Connect to Server((This may vary depending on the OS. If in trouble search for Connect To Server.))
  
-|Server| access.lorentz.leidenuniv.nl|+|Server| access.lorentz.leidenuniv.nl/davrods|
 |Port | 443| |Port | 443|
 |Type|Secure WebDAV (HTTPS)| |Type|Secure WebDAV (HTTPS)|
Line 320: Line 330:
  
 Finder -> Go -> Connect to Server Finder -> Go -> Connect to Server
-|Server Address| https://access.lorentz.leidenuniv.nl|+|Server Address| https://access.lorentz.leidenuniv.nl/davrods|
  
 Specify your IL credentials when prompted and click on Connect. Specify your IL credentials when prompted and click on Connect.
 +
 +==== Rclone Access ====
 +
 +:!: This access method works only on our remote desktop system
 +
 +Open a terminal
 +
 +<code bash>
 +$ rclone config
 +No remotes found - make a new one
 +n) New remote
 +s) Set configuration password
 +q) Quit config
 +n/s/q> n
 +name> iRODS
 +Type of storage to configure.
 +Enter a string value. Press Enter for the default ("").
 +Choose a number from below, or type in your own value
 + 1 / 1Fichier
 +   \ "fichier"
 + 2 / Alias for an existing remote
 +   \ "alias"
 + 3 / Amazon Drive
 +   \ "amazon cloud drive"
 + 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
 +   \ "s3"
 +...
 +36 / Union merges the contents of several upstream fs
 +   \ "union"
 +37 / Webdav
 +   \ "webdav"
 +38 / Yandex Disk
 +   \ "yandex"
 +39 / Zoho
 +   \ "zoho"
 +40 / http Connection
 +   \ "http"
 +41 / premiumize.me
 +   \ "premiumizeme"
 +42 / seafile
 +   \ "seafile"
 +Storage> 37
 +
 +** See help for webdav backend at: https://rclone.org/webdav/ **
 +
 +URL of http host to connect to
 +Enter a string value. Press Enter for the default ("").
 +Choose a number from below, or type in your own value
 + 1 / Connect to example.com
 +   \ "https://example.com"
 +url> https://access.lorentz.leidenuniv.nl/davrods
 +Name of the Webdav site/service/software you are using
 +Enter a string value. Press Enter for the default ("").
 +Choose a number from below, or type in your own value
 + 1 / Nextcloud
 +   \ "nextcloud"
 + 2 / Owncloud
 +   \ "owncloud"
 + 3 / Sharepoint Online, authenticated by Microsoft account.
 +   \ "sharepoint"
 + 4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
 +   \ "sharepoint-ntlm"
 + 5 / Other site/service or software
 +   \ "other"
 +vendor> iRODS
 +User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
 +Enter a string value. Press Enter for the default ("").
 +user> YOUR_IL_USERNAME
 +Password.
 +y) Yes type in my own password
 +g) Generate random password
 +n) No leave this optional password blank (default)
 +y/g/n> y
 +Enter the password:
 +password:
 +Confirm the password:
 +password:
 +Bearer token instead of user/pass (e.g. a Macaroon)
 +Enter a string value. Press Enter for the default ("").
 +bearer_token> 
 +Edit advanced config? (y/n)
 +y) Yes
 +n) No (default)
 +y/n> n
 +Remote config
 +--------------------
 +[iRODS]
 +type = webdav
 +url = https://access.lorentz.leidenuniv.nl/davrods
 +vendor = iRODS
 +user = YOUR_IL_USERNAME
 +pass = *** ENCRYPTED ***
 +--------------------
 +y) Yes this is OK (default)
 +e) Edit this remote
 +d) Delete this remote
 +y/e/d> y
 +Current remotes:
 +
 +Name                 Type
 +====                 ====
 +iRODS                webdav
 +
 +e) Edit existing remote
 +n) New remote
 +d) Delete remote
 +r) Rename remote
 +c) Copy remote
 +s) Set configuration password
 +q) Quit config
 +e/n/d/r/c/s/q> q
 +
 +$
 +
 +
 +</code>
 +
 +The navigate to https://remote.lorentz.leidenuniv.nl/pun/sys/dashboard/files/iRODS to interact with our iRODS storage via a web browser.
institute_lorentz/irods_fair_storage.1666250940.txt.gz · Last modified: 2022/10/20 07:29 by lenocil

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki