User Documentation¶

Accessing NeLS¶

There are two ways you can access data on NeLS:

• Via the NeLS portal online GUI
• Using ssh through command line

Via the NeLS portal¶

For quick browsing and simple file access you can log into NeLS using either their FEIDE identity if you are a member of a Norewegian institution or a NeLS identity which can be requested by contacting the ELIXIR support desk (See I don't have FEIDE login credentials).

Using ssh through command Line¶

After gathering your username and SSH key, you can use ssh to access NeLS with the following command (character $ only indicates beginning of the line and is not a part of the command):

$ ssh -i <SSH_key> -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" <username>@data.nels.elixir.no

Absolute path to the home directory in NeLS is /nels/users/<username>.

Collect connection details from the NeLS portal¶

Access to NeLS using ssh, scp, or sftp (be it via command line or with programs like PuTTY and FileZilla) requires four specific pieces of information: username [1] (this differs differs from either the FEiDE ID and Idp used to access NeLS through the web portal), Login host address [2], Data host address [3] and the SSH private key associated with the username [4]. All the informations can be found in the My Profile page of the NeLS portal. Screenshots of the page with highlighted connection details are shown below.

Download the SSH key to the computer from which you want to authenticate on NeLS, note the host addresses (login.nels.elixir.no and data.nels.elixir.no) and your username. Modify access permissions for the SSH key in such a way that only you can read and write to the file and no other user can access the file. In the linux-based operating systems (including OsX), one can achieve this by typing chmod 600 <SSH_key> into a terminal window. In Windows, one can access the security tab in the Properties option of the <SSH_key> file and keep access granted only to oneself, SYSTEM and Administrators.

Transfer data to and from NeLS¶

Upload/Download data using the Web interface of the NeLS portal¶

For quick browsing and simple file access a user can log into NeLS at https://nels.elixir.no/ using either their FEiDE identity if they are a member of a Norewegian institution or a NeLS identity which can be created by members of the Elixir Helpdesk with access to the NelS admin tools.

This access option is used when data needs to be backed up from NeLS to SBI, for details see section Midterm data storage in NeLS and SBI.

Here is a screenshot of the Personal Area in the NeLS portal. The numbered blocks highlight the following features: [1] Upload File(s), Add New Folder; [2] File and folder manipulation, Rename file or folder; [3] (De)select all/some items; [4] Projects Area - the same functionality available there as in the Personal Area.

Upload/Download via the command line using scp¶

Here is a list of several examples of how to upload files and folders to NeLS using scp command as well as how to download them. In the examples,\ at the end of the lines indicate command split accross multiple lines and can be omitted. Characters $ and > at the beginning of the lines do not belong to the commands, they only mimic most likely response of a terminal window to the command splitting with \ . To be able to follow this section, one needs to gather the connection details specified in the section "Collect the necessary connection details from the NeLS portal".

1. Upload a file into the Personal folder:

   $ scp -i <SSH_key> \
   > -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" \
   > <file-to-send-to-nels> <username>@data.nels.elixir.no:Personal/
   

2. Upload a file into a project folder:

   $ scp -i <SSH_key> \
   > -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" \
   > <file-to-send-to-nels> <username>@data.nels.elixir.no:Projects/<projectname>/
   

3. Upload a folder (recursive upload, -r option):

   $ scp -r -i <SSH_key> \
   > -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" \
   > file-to-send-to-nels <username>@data.nels.elixir.no:Personal/
   

4. Download a file from the Personal folder:

   $ scp -i <SSH_key> \
   > -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" \
   > <username>@data.nels.elixir.no:Personal/<file> \
   > <destination_local>
   

5. Download all files with extension .txt from the Personal folder (wildcard usage):

   $ scp -i <SSH_key> \
   > -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" \
   > <username>@data.nels.elixir.no:Personal/*.txt \
   > <destination_local>
   

6. Download a folder from a project folder (recursive download, -r option):

   $ scp -r -i <SSH_key> \
   > -o IdentitiesOnly=yes -o "ProxyCommand ssh -i <SSH_key> -W %h:%p <username>@login.nels.elixir.no" \
   > <username>@data.nels.elixir.no:Projects/<projectname>/<folder>
   > <destination_local>
   

Import data from e.g. filesender¶

You can import data to NeLS from other systems which are accesible under an URL. Here is an example for the uninett filesender service. Upload files to https://filesender.uninett.no or create a voucher for your collaborator to upload data. Copy the download link for the zip archive of all files, once they are uploaded. Log in to NeLS via ssh.

Import the data to NeLS with:

    $ wget -O tmp.zip "FILESENDER_ZIP_DOWNLOAD_LINK" && unzip tmp.zip && rm tmp.zip

Create checksums to ensure file integrity¶

A simple way to detect file corruption during transfer or storage are MD5 checksums. MD5 checksums are sufficient for detecting accidental corruption, for security related application please use more advanced algorithms such as SHA256 (these are also more compute intensive and slower)

• Linux (multiple files in current dir)

md5sum * > checksums.md5

• Linux (entire directory, recursive; paths relative)

cd /path/to/directory
find . -type f -print0 | xargs -0 md5sum > checksums.md5

• macOS (recursive directory)

cd /path/to/directory
find . -type f -print0 | xargs -0 md5 -r > checksums.md5
# or install coreutils and use md5sum as on Linux

• Windows (PowerShell; recursive, outputs md5sum-compatible "CHECKSUM relative\path")

# run in the directory you want checksummed
Get-ChildItem -Recurse -File | ForEach-Object {
  $h = Get-FileHash -Algorithm MD5 -Path $_.FullName
  "{0}  {1}" -f $h.Hash.ToLower(), ($_.FullName.Substring((Get-Location).Path.Length+1).Replace('\','/'))
} | Out-File -Encoding ASCII checksums.md5

Transfer files + checksum to NeLS (example using scp)¶

• Upload a directory and the checksum file e.g. using scp

Verify checksums on NeLS (after transfer)¶

• Login via ssh and change to the target directory on NeLS and run:

cd ~/Personal/dir   # or /nels/users/<username>/Personal/directory
md5sum -c checksums.md5
# expected output: ./filename.txt: OK   (or filename: OK)

• If you used relative paths with leading "./" (from find), md5sum -c will match them; if checksums use plain filenames ensure current directory matches the paths in checksums.md5.

Notes / troubleshooting (very brief)¶

• Ensure the paths in checksums.md5 match locations on NeLS (use relative paths from the directory where you'll run md5sum -c).
• If macOS produced a different format, recreate checksums with md5 -r or use md5sum-compatible output before transfer.

Transfer data to and from SBI¶

SBI is only connected to NeLS. Data has to reside in NeLS before it can be imported in SBI, and data in SBI can only be exported to NeLS. You need to be member of a SBI project before you can transfer data between NeLS and SBI

Importing data in SBI¶

You will find all SBI projects you are a member of under the StoreBioInfo menu in the NeLS Portal. You will see both your NeLS projects [1] (Personal and Projects) to the left of the screen and the SBI projects to the right [2].

Select the SBI project you would like to import data in. There might already exist datasets in the project [3], or you can create a new dataset for new datasets.

Select the dataset you want to import data into [4] and the subfolder for the data [5]. The subfolder structure is automatically generated when a new dataset is created, and normally contain a folder named rawdata for the data from the data provider (e.g. fastq files prior to analysis) and a folder named analysis for the analysed data (e.g. assemblies and alignment files).

On the NeLS side, browser to the project and folders or files that you want to transfer to SBI [1]. Select single files [2] or all files [3]. Make sure you are in the correct destination folder on the SBI side [4], and press Transfer to SBI [5].

Create new dataset in SBI¶

Within an existing project [1] you can create a new dataset by pressing + New Dataset [2]. In the window that appear, select the Dataset type form the drop-down menu [3]. You also need to give the dataset a Name [5] and a short Description [5]. Finally, press Create [6] to make the new dataset folder in SBI.

The new dataset should appear on the list of datasets [1] in the project you created it in [2]. This should also have created subfolders for raw data and analysed data [3]. It is possible to edit the Name and Description of the dataset [4].

Exporting data from SBI¶

Exporting data from SBI to NeLS is similar to importing files to SBI from NeLS. Make sure you are in the correct destination folder on the NeLS side [1]. On the SBI side, browser to the project and folders or files that you want to transfer to SBI [2], and press Transfer to NeLS [3].

Midterm data storage in NeLS and SBI¶

NeLS is meant for storing and sharing active project data. This is typically data that are being analysed and shared with collaborators during a project. Data normally reside here for months/years.

SBI is meant for more long term storage of project data. This is typically data that are not being analysed at the moment, but is planned to included later in the project. Data normally reside here for years

User roles in NeLS and SBI¶

There are multiple user roles to control read/write permissions and to manage project members. These roles are normally set when the project is made by instructions from the PI, but can be changed during the project.

User role	NeLS	SBI
PI	Manage File system: Add, Rename, Navigate, Download, Delete all Content, Manage members	Add, Navigate, Transfer to NeLS, Manage members
Data manager	Manage File system: Add, Rename, Navigate, Download, Delete all Content, Manage members	Add, Navigate, Transfer to NeLS, Manage members
Power user	Manage File system: Add, Rename, Navigate, Download, Delete all Content	Add, Navigate, Transfer to NeLS, Manage members
Normal user	Add files & folders, Navigate & Download	Navigate & Transfer to NeLS
Help desk	Manage File system: Add, Rename, Navigate, Download, Delete all Content	Add files & folders, Navigate & Transfer to NeLS
Core facility	Add files & folders, Navigate & Download	-
Collaborator	Add files & folders, Navigate & Download	-

Where to deposit data long term¶

When a project is ready for publication, the data should be deposited in public data repositories (deposition databases or data archives) suitable for the type of data. A project can hold different types of data that should be submitted to different data repositories. Many biomolecular data types can be deposited to the ELIXIR Deposition Databases.

ELIXIR Norway is a data broker for the European Nucleotide Archive (ENA) and support Norwegian users to submit sequence data there.

Import and export Galaxy histories to and from NeLS¶

usegalaxy.no is directly connected to your data storage in NeLS. We have made a function in the usegalaxy.no that can export a whole history including the data sets in that history as a single compressed file to NeLS.

This file can be imported into another Galaxy instance and the analysis work can continue there. The advantage of this export is that all provenance data will be kept, meaning all commands, tool versions, database versions, etc that were used prior to the export.

Before getting started¶

• Login to https://usegalaxy.no/ with your FEIDE or NeLS idp
• You need at least one history in usegalaxy.no to export

Exporting a history from usegalaxy.no to NeLS¶

Dataset and histories can also be exported from usegalaxy.no to your local system.

• Make sure that the history you want to export is your current (active) history
• Press the History options [1] and select Export History under the NeLS storage section [2]
• You will be redirected to the NeLS portal where you need to choose the destination folder [3] (shown here a random subfolder in the Personal folder)
• Browse to the folder you want to export the history to and press Use current folder [4].
• You should be redirected back to usegalaxy.no where you can view the transfer progress (you might need to scroll down a bit in the usegalaxy.no main window). When the transfer is complete, the progression bar will turn green [5]

Importing a history from NeLS to usegalaxy.no¶

Both single datasets and complete histories can be imported from your NeLS storage

• Press the History options and select Import History under the NeLS storage section [1]
• You will be redirected to the NeLS storage. browse to the folder containing the data (example here is a random subfolder in the Project folder), select the history (or file) [2] and press Send to Galaxy [3] *You will be redirected back to usegalaxy.no and you can monitor the progress of the import [4]. The progress bar will turn green when the import is complete [5]
• The name of the imported history will be imported from archive: .... The history can be accessed by selecting Histories [6] under the User menu. You can rename the history if you like.

Delete and permanent delete data in usegalaxy.no¶

Galaxy offer multiple ways to delete datasets and histories. The are two major types of deletion:

1. Delete datasets or histories - This option temporary delete data (basically just hide the data)
2. Permanently delete datasets or histories - This option permanently delete data from the disk

Delete datasets¶

Select the dataset(s) you want to delete (temporary) by pressing X next to the dataset in your History panel [1]. The dataset should disappear from you history, but you still access it [2]

You have only temporary deleted the data (equal to moving it to the trash bin on you computer), and it can be undeleted (moved back from trash bin), or permanently deleted by pressing Permanently remove it from disk [3] Press OK to confirm the permanent deletion [4]

You can still see traces of the dataset in your history [5], but the data in the file is deleted from the disk and you have freed up disk space

Delete complete histories¶

Similarly, you can temporary and permanently delete histories. Access the overview of your histories either by selecting Histories [1] under the User menu or by pressing View all histories [2] in the History panel

From the drop-down menu of the history you want to delete (exemplified by the history named imported from archive: ....), press Delete Permanently [3] for permanent deletion.

If you chose the other history view, from the drop-down menu of the history you want to delete press Purge [4]. Both options will ask you to confirm the deletion, and both options will erase the data from disk.

Import and export workflows in usegalaxy.no¶

Galaxy offers a simple way to download a workflow as a single file which can be imported into another Galaxy instance and run there.

Exporting a workflow:¶

Select the Workflow menu [1]

A list of the workflows you have created or imported into your user in usegalaxy.no will be displayed

From the menu that appear by clicking on the name of the workflow you want to export [2], select Download [3] and save the workflow (single .ga file) to your local machine

Importing a workflow¶

Select the Workflow menu [1]

A list of workflows that you have created or imported (if you have any) will be displayed

Pressing Import [2] and a pop-up window with import options will appear. If you have the workflow stored locally, press Browse [3] and select the .ga file and press Import workflow [4]

The imported workflow will appear in the list of workflows

Different tool versions on exporting and importing Galaxy servers may cause the workflow to throw a warning. Click on the name of the workflow and select the Edit [1] to view potential issues, then press Continue [2]

You should now be able to run the workflow