Developing and Administering OpenVRE - inab/openVRE GitHub Wiki

3. Developing and Administering OpenVRE

This section describes the core configuration, system setup, and administrative tasks needed to deploy, customize, and maintain openVRE in production or development environments.
It is primarily intended for developers, integrators, and system administrators.

3.1 Core Configuration and Setup

The following topics guide you through the initial configuration and administration of the VRE environment.
They include visual interface customization, manual SGE setup, Keycloak authentication, and HashiCorp Vault integration.

3.1.1 Apply Manual SGE Configuration

Manual configuration is required only for custom SGE cluster setups, where Docker-based automation is not used.

β€’ sgecore username

Before initializing SGE to recognize jobs sent from the frontend, identify the Docker hostname:

docker exec -it sgecore /bin/bash
qconf -as ${FRONT_END_HOSTNAME}
qconf -mconf   # change UID from 1000 to 33

β€’ sgecore usage permissions

Grant SGE proper permissions to use Docker:

groupmod -g 120 docker    # or adjust to your system's Docker group
usermod -aG docker application

chown root:docker /var/run/docker.sock
chmod 660 /var/run/docker.sock

3.1.2 Keycloak Configuration

A sample realm configuration is imported by default. You can customise this realm through the Keycloak UI or create a new one from scratch.

Customise default realm

  1. Go to Clients, select open-vre and change the Root URL, Home URL and Valid redirect URIs with your instance values.
  2. On this same client, go to Credentials tab, click on Regenerate inside the Client Secret section, copy your new Client Secret and paste it on the KEYCLOAK_SECRET of your .env file.

New realm from scratch

  1. Create a new realm with the project name.
  2. Create a new client scope: set Type to Default, Protocol to OpenID Connect and check on Include in token scope.
  3. Create a new client with the id open-vre: set Client type to OpenID Connect, check on Client authentication, Standard flow and Direct access grants, define your URLs (e.g. https://myopenvre.com/) and Valid redirect URIs (e.g. https://myopenvre.com/*).
  4. Go to Realm settings -> Login and check on User registration
  5. Go to Clients, select open-vre, go to Credentials tab, copy your Client Secret and paste it on the KEYCLOAK_SECRET of your .env file.

Troubleshooting

Some systems may require allowing Keycloak access through iptables:

sudo iptables -I INPUT -s {keycloak internal IP} -p tcp --dport 8080 -j ACCEPT

3.1.3 HashiCorp Vault Configuration

The following configuration details apply to the production deployment of openVRE:

Once your Keycloak realm and client are defined, you can start Vault following the steps here: Vault Init.

Vault will be launched following this configuration:

  • Sealed on every restart: It must be unsealed to enable access to secrets. Unseal keys should be securely stored.
  • JWT authentication and verification through OIDC Discovery: Vault allows authentication via a JWT. This JWT is verified against public keys from the issuer, which are fetched from the OIDC Discovery endpoint (i.e. your Keycloak instance). Other validation criteria (e.g. iss, aud, etc.) are also applied. For more details, read the official Vault documentation.
  • RBAC with policies: The default template policy limits the access to user's secrets based on user's sub attribute of the JWT.
  • No secrets versioning: KV secrets engine version 1 is used, meaning only most recently written value is stored. To use versioning use KV version 2. For more details, read the official Vault documentation.
  • End-to-end TLS: The path of the certificates needs to be defined in the .env file.

Development environment note:
The development setup already includes all necessary default settings for Vault. No additional configuration is required in development, and TLS/unseal workflows are simplified for convenience.

3.2 Tools Integration

openVRE integrates modular computational components called Tools, enabling reproducible and extensible research workflows.
These tools can be brought, configured, and executed directly from the platform by researchers or administrators.

3.2.1 Tools Overview

openVRE Tools are modular computational units within the openVRE analysis platform that enable diverse research workflows.
Developed by third-party tool developers, these tools can be parameterized by researchers to suit specific project needs.
Once configured, they are executed by the platform’s compute back end.

Each tool runs within a software container (e.g., Docker, Singularity), ensuring compatibility, portability, and isolation β€” making integration seamless and reproducible.

3.2.2 Types of Tools

There are two main categories of tools in openVRE:

  • Non-Interactive Tools β€” Operate in batch mode without user input. Ideal for large-scale or automated analyses.
  • Interactive Tools β€” Provide a web-based interface for real-time interaction and visualization of results.

3.2.3 How to Bring Your Own Tool

openVRE supports multiple methods for integrating your own tools:

Each guide provides step-by-step instructions for registration, configuration, and validation within the platform.

3.3 Repository and Resources

openVRE supports modular integration of external repositories and resources, enabling smooth data exchange and access to computation or visualization back ends.

3.3.1 Repository Interfaces

openVRE provides pre-built repository interfaces that allow integration with popular research data storage and sharing services. These interfaces enable seamless transfer, management, and publication of data from within openVRE.

3.3.2 Supported Repository Interfaces

  • OpenStack Swift Object Storage β€” Scalable, distributed storage service used for archiving large research datasets.
  • WebDAV-based Integrations:
    • Nextcloud β€” Open-source file sharing and collaboration platform supporting WebDAV for secure access and synchronization.
  • HTTP API-based Integrations:
    • XNAT (eXtensible Neuroimaging Archive Toolkit) β€” Biomedical and neuroimaging data management platform.
    • ArrayExpress β€” EMBL-EBI repository for gene expression and functional genomics datasets.
    • BigNASim β€” Repository for nucleic acid simulations and bioinformatics data involving DNA/RNA structures.
  • European Genome-Phenome Archive (EGA) β€” Secure repository for sensitive genomic and phenotypic data, supporting controlled-access research.
  • B2SHARE β€” EUDAT open data repository for storing, publishing, and sharing research data with metadata compliance.
  • Rclone (in progress) β€” Command-line tool supporting 40+ cloud storage backends, ideal for syncing and secure data transfers.

3.3.3 Puggable Resources

openVRE supports puggable resources, which are modular connectors for extending access to external computational, data, or visualization services. Administrators can configure new back ends or link remote HPC/cloud systems, enabling hybrid or distributed computing environments.

3.4 Sites and Linked Accounts

In openVRE:

  • Sites represent external infrastructures β€” such as HPC clusters, cloud services, or data repositories β€” that can be registered and managed by admins.
  • Linked Accounts securely store user-specific credentials and authentication tokens, enabling access to those connected sites.
  • These connections are handled securely through Vault-based secret management and Keycloak authentication integration.

For detailed instructions on extending openVRE with external infrastructures and user authentication, see the following dedicated guides:

3.4.1 Register a New Linked Site

Steps to define and register a new computational, storage, or data site in openVRE.
See: Add New Linked Site

3.4.2 Adding a New Data Repository

Instructions for integrating new data repositories or catalog interfaces into the platform.
See: Adding a New Data Repository

3.5 Graphic and UI Extensions

These extensions enable developers to enhance and customize the openVRE user interface and visual experience.

3.5.1 Graphic configuration

For a full extensive tutorial on the graphic modules of the VRE, refers to the graphic configuration page

3.5.2 Extending Frontend Components

Guidelines for extending or adapting frontend elements to integrate new components, menus, or UI modules.
Examples include adding menu tabs for external resources or customizing account settings views.
See also: Integrating Front-End References

3.6 Related Topics – Admin Configuration

For further extensions and configuration details on the administrative and development side, see: