Storage Solutions

Storage
Author

Esther Plomp

Published

January 9, 2023

Overview of Available Data Storage Solutions during the project’s duration at TU Delft. These storage solutions are primarily for internal use, but can be shared with collaborators.

Please see TopDesk for an overview of all the storage solutions available to you. Below follow some more details for each storage solution.

Storage strategy during the project

Recommended storage solutions for research data are:

Cloud based:

  • OneDrive

  • SURFdrive

    These two solutions are personal storage solutions. Ensure that others have access to the data by sharing folders or using a shared Project Data Storage!

Network drive:

  • Project Drive / Project Data Storage

    The Project Data Storage is the most secure solution and intended for long term storage.

TU Delft network drives

Personal Drive (H:) Staff Group Drive (M:) Staff Group Drive (N:) Project Data Storage (U:)
Purpose Private work files only (sharing not possible) Data shared within research group or department Temporary location for large data Data from individual projects, shared with project members
Suitable for confidential data? Yes No No Yes
Size 8 GB > 5TB > 5TB > 5TB
Backed up? Yes Yes NO Yes

Project Data Storage

The Project Data Storage is the recommended storage solution for research data.

  • Project leaders or supervisors should request the drive to ensure that they also have access to the data (especially important for the long term storage of any data).

  • Anyone with a NetID can be added to a Project Data Storage (for external collaborators this NetID will have to be generated so additional information will need to be provided). To provide access to additional people you can add them when you request the drive via Topdesk, you can contact the Service Desk of your faculty (servicepuntTNW [at] tudelft.nl), or you can add them using UMRA (see TU Delft ICT for more UMRA manuals).

    • Note that anyone with access to the Project Data Storage will have viewing or editing rights for the entire Project Data Storage. If you want to easily manage access permissions it may be needed to request multiple Project Data Storage or work with alternative storage solutions such as SURF/OneDrive.
    • Every account has to be traceable to a person (so not a research group).
    • It is helpful to keep a list of users that have access to your Project Data Storage, since there is no interface that provides you this overview.

  • The Project Data Storage requires an internet connection. It may be more convenient to use the Project Data Storage as a backup environment and use another storage solution as your active working environment (SURFdrive/Onedrive).

  • To request additional storage space for an existing project, you can contact the service desk or re-submit the TopDesk form with ‘Change’ selected rather than ‘New’. For >5 TB of space, you may be contacted by the faculty IT manager for approval.

  • Learn more about how to do an additional back up of the Project Data Storage.

  • Mount Project Drive on Server.

Network Drives and Delft HPC

Please see the DelftBlue documentation on the storage of data-transfer recommendations from the HPC clusters. See also the Delft AI Cluster.

SURFdrive

  • Secure cloud storage
  • 30 days backup, synchronize and share data
  • Desktop app for synchronization
  • 1 TB
  • Share data with external collaborators
    • public link + with password protection + expiry date
    • Check the manual
  • Not suitable for long term storage as this is a personal storage space (account expires when you leave TU Delft)
  • Students don’t have accounts but you can give access to folders/files (OneDrive may be more suitable when working with students)
  • Do not use for highly confidential data such as state secrets, sensitive personal data, or highly sensitive IP material.

OneDrive

See the OneDrive manual on how to get started.

  • Secure cloud storage
  • 30 days backup, synchronize and share data
  • 1 TB
  • Share data with external collaborators by specifying users one by one with their email address
  • Not suitable for long term storage account as this is a personal storage space (expires when you leave TU Delft)
  • Students also have accounts
  • Do not use for highly confidential data such as state secrets, sensitive personal data, or highly sensitive IP material.
  • Versioning up to 93 days.

Cloud4Research

Cloud4Research is a TU Delft IT service to facilitate the use of public cloud for research. Contact the Cloud4Research team for an intake meeting to discuss your infrastructure and support needs, as well as the financial requirements.

Sending files

SURFfilesender can be used to share files up to 1 TB (also with end-to-end encryption up to 2 TB)

TU Delft GitLab

TU Delft GitLab is a version control system for code with backup facility. Access to external collaborators, as well as public code sharing, can be provided but if you expect multiple external collaborators GitHub may be more suitable (however, please see Give Up GitHub!). TU Delft GitLab is primarily meant for confidential code/data that cannot be publicly shared. You can create up to 50 repositories/projects. GitLab also allows you to create subgroups to manage your repositories.

Access to a GitLab repository for externals can be requested through TopDesk (Topdesk > Software and Authorisations > IT for Researchers > GitLab TU Delft > Request GitLab).

TU Delft GitLab or GitHub?

TU Delft GitLab is primarily intended or secure code management for confidential code. GitHub is easier to use if you want to collaborate with externals and you are comfortable sharing your code publicly (GitHub allows private repositories but features are more limited). Do not share any confidential or personal information via GitHub, as even information in private repositories may be publicly accessible using GitHub!

Subversion at TU Delft

TU Delft provides version control systems for data with backup facility. Access can be controlled by the repository owner.

Subversion repository can be requested through TopDesk (Topdesk > Software and Authorisations > IT for Researchers > Version Control Tools > Report to Service Desk)

Alternative (non-TU Delft) solutions are Datalad, DVC, lakeFS, git-LFS.

Electronic Lab Notebooks

Electronic Lab Notebooks allow you to capture and organise data digitally, which makes it easier to back up information and search through your files.

Illustration by Connie Clare

Teams

While not necessarily the best solution for research data (especially big data), Teams can provide a solution when you need to collaborate with multiple people/teams and set up tasks for group members.

Files shared via Teams are stored on Sharepoint, which is backed up regularly by TU Delft ICT.

See the Microsoft Teams post for more information.

More Information