THERE IS A NEW VERSION OF THIS BEING DEVELOPED HERE: DataDiVR PLEASE USE THE NEW VERSION IF YOU WANT TO TRY THE APPLICATION, THIS REPOSITORY IS ONLY HERE FOR HISTORIC REASONS

VRNetzer - A Virtual Reality Framework for Network Visualization and Analytics

Networks provide a powerful representation of complex systems of interacting components. In addition to a wide range of available analytical and computational tools, networks also offer a visual interface for exploring large data in a uniquely intuitive fashion.
However, the size and complexity of many networks render static visualizations on common screen or paper sizes impractical and result in proverbial 'hairballs'. Here, we introduce an immersive Virtual Reality (VR) platform that overcomes these limitations and unlocks the full potential of visual, interactive exploration of large networks.
Our platform is designed towards maximal customization and extendability, with key features including import of custom code for data analysis, easy integration of external databases, and design of arbitrary user interface elements. As a proof of concept, we show how our platform can be used to interactively explore genome-scale molecular networks for identifying genetic aberrations responsible for rare diseases and develop hypotheses on their respective pathobiological mechanisms.
Our platform represents a first-of-its-kind, general purpose VR data exploration platform in which human intuition can work seamlessly together with state-of-the-art analysis methods for large and diverse data.

Note: This project is still in an infant state of development. This is a public beta release, meaning it is not fit to be used in any critical applications yet and may still contain some bugs. You can get in touch with us if you would like to contribute to the further development of this project.

Content

Hardware Requirements

Installation: Quick Start

Installation: Stand Alone

Desktop Mode

Data Logging

Take screenshots and record videos

Tutorial 1: Using the Uploader to add your own network

VRNetzer Architecture Overview

Tutorial 2: Creating custom User Interfaces

Tutorial 3: Creating a route on the backend

USE CASE Gene prioritization

VRnet API Documentation

Dependencies

License

Hardware Requirements

We are running the platform on an MSI gaming laptop with the following specs:

• CPU: i7 - 7820HK
• RAM: 16 GB
• GPU: NVIDIA 1070 TI
• OS: WINDOWS 10

and a VR HEADSET - we tested the following:

• HTC VIVE VR Headset + controllers
• OCULUS QUEST (tethered to windows with link) + controllers

Installation: Quick Start

Time required ~ 10min

For a quick start, you can just download the VR Module executable and run VR_Module.exe on your windows computer with a SteamVR compatible headset. Please watch this video that will help to get you started. The VR Module comes preconfigured to connect to the other modules that we already installed on our server to make it easy for you to get a first impression. It has the ability to upload your own datasets, but we don't recommend it. Do NOT upload any sensitive data here, this is only for demonstration purposes and for the whole world to see. We don't guarantee your data's safety! If you want to work with your own data you should use the Stand Alone version.

Installation: Stand Alone

Time required ~ 60min

However, if you want to implement your own datasets and functionalities, you can also run everything localy on your machine (the SAFE option if you are dealing with sensitive data) or run the Analytics Module and UI Module on a networkserver or cloud service (if you need scalability and processing power).
Here is a step by step guide how to do a local installation on a windows computer. Note that the Analytics Module and the UI Module can also run on a (linux) server, while the VR Module needs to run on a windows 10 machine as gaming hardware is required for Virtual Reality which is best supported under windows.

1.) Set up a MySql Database

• download and install MySql Workbench
• download the .dump file of our database from here
• Create a new database, user and password and upload .sql file to it

you have now a clone of our database on your local machine.

2.) Clone the Analytics Module repository into a new folder somewhere on your computer called "VRNetzer"

• install python >3.6 <3.9, make sure its added to path variable

• install pip for python

• edit VRNetzer/Analytics_Module/db_config.py to match the database host and user you created before and save

• right click on VRNetzer/Analytics_Module/runAnalytics_Module.ps1 -> "run with power shell"
  

• if the output of the console window that just opened ends with "Running on http://127.0.0.1:1337/" you are good to go, if there are errors you will need to install dependencies.

3.) Clone the UI Module repository into "VRNetzer"

• install python >3.6, make sure its added to path variable

• install pip for python

• edit VRNetzer/uiserver/static/js/UI_Module_Config.js and set "dbprefix" to the address of your dataServer , here: 'http://127.0.0.1:1337' and save.

• edit VRNetzer/UI_Module/static/js/ UI_Module_Config.js and set "dbprefix" to the address of your Analytics Module, here: 'http://127.0.0.1:1337' and save.

• right click on VRNetzer/UI_Module/run_UI_Module.ps1 -> "run with power shell"

• if the output of the console window that just opened ends with "Running on http://127.0.0.1:5000/" you are good to go, if there are errors you will need to install dependencies.

4.) Download the VR Module executable and extract it to "VRNetzer"

• if you haven't already, make a Steam account and install SteamVR and test your headset
• edit VRNetzer/VRnet/viveNet/Content/data/Config.txt and change the address to the one where your UI Module is, here http://127.0.0.1:5000/
• run VR_Module.exe

Desktop Mode

It is possible to run the VR Module in desktop mode which is useful for development tasks but does not provide the full functionality as with a headset.
To do so, change the value "vr":true in the config file of the VR Module to "vr":false and restart the VR Module.

Data Logging

The VR Module can produce log files that contain a list of all that has happened during a VR session. This produces quite big textfiles and is therefore turned off by default.
To enable logging change the value "logging":false in the config file of the VR Module to "logging":true and restart the VR Module. The log files are generated in VRNetzer/VR_Module/viveNet/Content/data.

Take screenshots and record videos

If you want you can take screenshots and videos of the running application using nvidia geforce experience, which is a free download for nvidia graphics cards, find it here: geforce experience. After installation the default key combination for taking a screenshot is alt+f1 and to start or stop a video recording you press alt+f9. Depending on your setup you might need to enable the ingame overlay and enable desktop recording in the privacy settings in the geforce experience settings if the hotkeys do not work right away.

Tutorial 1: Using the Uploader to add your own network

WINDOWS

• right click on VRNetzer/ Analytics_Module/runAnalyticsModule.ps1 -> "run with power shell" to start the Analytics Module
• right click on VRNetzer/ UI_Module/run_UI_Module.ps1 -> "run with power shell" to start the UI Module
• open the web frontend of the UI Module in a browser /http://127.0.0.1:5000/upload

MAC/LINUX

• open terminal, cd to Analytics_Module and run app.py with python3
• open terminal, cd to UI_Module and run json_io.py with python3

• open the web frontend of the UI Module in a browser http://127.0.0.1:5000/upload
  

• select .csv files to upload, they must be formatted after these guidelines as sketched in the picture above.
• Required is as least a layout file with xyz coordinated (normalized each in [0,1]) and colors and you will obtain a 3D point cloud.
• If you add an edge list that refers to the identifiers given in the 1st column of the layout file you will obtain a network with nodes and edges.
• Uploading a node file with symbols, names and functions/features (optional, can remain empty) you will find the symbols as labels in VR and the node characteristics on the node panel when clicking on a node.
• restart the VR Module and load your project

VRNetzer Architecture Overview

The VRNetzer platform consists of 5 Modules:

VRNet - the virtual reality module written in Unreal Engine

Unreal Engine is one of the industry leaders in the videogame world. We chose it as the base for our VR Module for it's astounding graphics performance, continuous support of upcoming VR hardware and because it's open source.

UI Module - a jQuery and html website

The UI Module is running in the browser of your local machine and can SEND GET and POST requests TO the Analytics Module. Nearly all actions originate here (exept the ones in the VR Module, like if the user touches a node or moves the network with the VR controllers).
Think of the UI Module as the frontend of a website and the Analytics Module as it's backend.
A User clicks on a button on the frontend, this makes it to send a post request to a specific URL (route) on the backend (the Analytics Module), await it's response and finally display the result as text or as a graph on the user interface and/or call functions in the VR Module to alter the appearence of the network displayed in VR.
The Analytics Module can only RESPOND to those requests, meaning the Analytics Module can never send something to the frontend without being asked. Every communication is Initiated by the UI Module.

Now here is what's special about the UI Module:

• It can also SEND api function calls to the VR Module

AND

• It can RECEIVE calls from the VR Module

In contrast to the Analytics Module, the VR Module CAN initiate communication with the UI Module and call special functions set up in the UI Module. This picture illustrates the different routes of communication in the framework.

Analytics Module - a Flask/Python webserver

The Analytics Module is the backend that performs all the data science tasks. It has

• separate routes (URLs) defined for each task
  When the UI Module sends a request to one of these, it parses the input parameters, makes database queries and performs calculations, and finally returns it's response to the UI Module.
• it can run on the local machine or on a powerful cloud server if more power is needed

SQL database

All data is stored within a MySQL database. The following schema shows the minimal set of tables that describes one single network.

The schema and the related tables are auto-generated and auto-populated upon user upload. To upload your own network, see Tutorial 1.

For advanced users:

Additionally, you may want to include more data to contextualize your nodes or edges. For example, in our proof of concept application, we use additional tables to store:

• hierarchical information of our taxonomic node annotations (attribute_taxonomies)
• articles mentioning specific genes (articles, nodes_articles)
• gene expression levels of genes in different tissues (gtex_values)

The possibilities are endless! Functions to query or manipulate the database are in tables.py of the Analytics Module.

Tutorial 2: Creating custom User Interfaces

The User Interface in the virtual reality module is a website made with jQuery. In this tutorial you will learn how to make UI elements like buttons or dropdown lists and how to communicate with the VR and Analytics Module.
Basic JavaScript and HTML skills required.
Prerequisites: install the VRNetzer framework on your local machine Installation: Stand Alone

Step 1: Create a button

• in Visual Studio Code (or the text editor of your choice) File -> Open Folder -> navigate to your UI Module Folder

• right click on VRNetzer/Analytics_Module/runAnalyticsModule.ps1 -> "run with power shell"
  

• right click on VRNetzer/UI_Module/runUIModule.ps1 -> "run with power shell"

• open the VR Module's config.txt file located at VRNetzer/VR_Module/viveNet/Content/data in your text editor and change "vr":true to "vr":false and "UIServerAdress":"....." to "UIServerAdress":"http://127.0.0.1:5000/" or where your UI Module is running and save the file

• run VR_Modul.exe

• it might promt you to install DirectXRuntime, follow the instructions

• Once the VR Module opens, you will see the main UI to which we will add something in the right-most tab in the left upper corner of the 3D view.

• press Alt + Enter to switch from fullscreen to windowed mode.

• click on the user interface in the left corner and hit 'Ctrl + Shift + i' to open the developer tools, click on "Network" and tick the "Disable Cache" checkbox. Note the "Console" window, where debugging output is displayed.
  

• open VRNetzer/UI_Module/templates/main.html in your editor

• at the end of the file, after <div id="tabs-7"> add <button id="MyNewButton"> EXIT </button>

• save changes to main.html and refresh the browser by clicking in the red area and hit F5
  

• if you now navigate to the right most tab again, the button appeared. It looks grey though, not like the other buttons.

The different elements of the page are set up in the html documents. All the logic happens in the JavaScript, in the UI_Module\static\js folder.

VRNetzer_API.js is the most important, it has all the functions to communicate between the Analytics Module and the VR Module.

Then there is a file for each of the html files (main_UI.js). These all have this initialization function $(document).ready(function () {....} and in there are functions that bind to the UI elements created in Html. For our button we do this:

• at the end of main_UI.js, but still inside the document.ready() function put this code:
  

$(function () {
        $("#MyNewButton").button();
        $("#MyNewButton").click(function (event) {
            event.preventDefault();
            logger("click!") 
        });
    });

• save and refresh the browser
  

Now that jQuery is aware of the new button the css styling also works. Click it and the console outputs the message so we know it works.

Step 2: Communicate with the VR Module

• put this code in the MyNewButton.click() function

var args = {
    "content": "somecoolName",
    "route": "tutorial"
};

ue4("GetSelection", args);

Everything that looks like ue4("name","argument") are calls to the VR module, the so called VRnet API documented here.

The variable "args" is a JavaScript object and follows the JSON syntax.

The "GetSelection" call above returns the currently selected nodes in the VR module to this return function ue.interface.getSelection specified in VRNetzer_API.js around line 30

//// FUNCTIONS CALLED BY UE4

ue.interface.getSelection = function (data) {
    logger(data);

    switch (data.route) {
     case "saveSelection":
         SaveSelectionDB(data);
         break;
     case "reLayout":
         ReLayoutSubSet(data);
         break;
     case "GSEA":
         GSEASubSet(data);
         break;

    }

};

Because the GetSelection call is used for different things, it's arguments contain "route": "tutorial" that is used in the return function to determine what to do with the response.

• in the return function above, add a case "tutorial" to the switch statement

    case "tutorial":
        LogOnUIServer(data);
        break;

LogOnUIServer(data); prints out the data we receive from the VR Module on our python console.

That's it! let's start the VR Module and check if everything works.

• start viveNet.exe
• open a layout
  
• load a selection
  
• click on MyNewButton

You should see some printout in the UI Module console window. We have now initiated communication from the UI module to the VR module (by pressing the button) and received a response. In the next step we will see how to do the same with the backend, the Analytics Module.

Tutorial 3: Creating a route on the backend

Basic Python knowledge required
Prerequisites:

• local installation of the VRNetzer framework Installation: Stand Alone

• Tutorial 2 finished

• in a new instance of Visual Studio Code (or the text editor of your choice)
  File -> Open Folder -> navigate to your ** Analytics_Module ** folder

• in app.py, at the end of the document but before if __name__ == "__main__": insert

@app.route('/api/<string:db_namespace>/MyNewRoute', methods=['POST'])
@cross_origin()
def my_new_route(db_namespace):
    data =request.get_json()
    node_ids = [int(x) for x in data['node_ids']]
    print(node_ids)
    return jsonify(node_ids)

• save app.py and bring up the Analytics Module console, it should detect the change and restart automatically
  

That’s it! Now we will go back to the UI Module and forward the output of the VR module that we created previously in tutorial 2.
For this we will use a POST request.

• go to the UI Module project and open it in a new Visual Studio Instance.
• paste this code at the end of VRNetzer_API.js

function MyNewPostRequest(data) {

    payload = JSON.stringify(data);
    //logger(payload);
    path = dbprefix + "/api/" + thisNamespace.namespace + "/MyNewRoute";
    $.ajax({
        type: "POST",
        url: path,
        contentType: "application/json",
        data: payload,
        dataType: "json",
        headers: {
            "Authorization": "Basic " + btoa(dbuser + ":" + dbpw)
        },
        success: function (response) {
        // DO SOMETHING WITH RESPONSE HERE!
            logger(response);
        },
        error: function (err) {
            logger(err);
            logger(data);
        }
    });

}

This is a blank POST request that calls the route we created before.

path = dbprefix + "/api/" + thisNamespace.namespace + "/MyNewRoute" is the target URL, dbprefix comes from UI_Module_Config.js and thisNamespace.namespace is the name of the project. path = dbprefix + "/api/" + thisNamespace.namespace + "/MyNewRoute" is the target URL, dbprefix comes from Analytics_ModuleConfig.js and thisNamespace.namespace is the name of the project.

• also in VRNetzer_API.js, in the function ue.interface.getSelection put MyNewPostRequest(data); instead of LogOnUIServer(data);

CSV file formats

You can find examples of all used .csv formats HERE to to use as templates for the formatting.

Note: You need to assign unique node integer IDs (like EntrezID in case of genes) for your files to upload
Don't use commas in string fields, as they are reserved delimiters.

Layout Lists look like this:

8473,0.4993,0.4544,0.640,188,20,26,100,3dportrait
...

where each line is a node with the following data:

8473	0.4993	0.4544	0.640	188	20	26	100	3dportrait
NodeID	X-Pos	Y-Pos	Z-Pos	R	G	B	A	Namespace

Note: Please provide XYZ coordinates for the layout that are normalised between 0 - 1

RGBA colours range from 0 - 255 Note: "A" (in RGBA) value's should be 100. Bigger values makes nodes glow, smaller values make them darker.

Node Lists

It is recommended to provide a Node file that converts NodeIDs given in the Layout file into readable symbols and names. There is also the possibility to fill the 4th column with functions, descriptions or other properties that should show up on the node panel within the VR. Please make sure that no commas appear within these fields as the are used as delimiters!

The format could look like this:

8473	OGT	O-linked N-acetylglucosamine (GlcNAc) transferase	protein coding	3dportrait
NodeID	Symbol	Name	Function	Namespace

Symbols will appear as labels directly at selected nodes and at nodes in the proximity.

Link Lists

1267,2945
...

where

1267	2945
Start	End

The identifiers refer to the NodeIDs given in the Layout file. For now, unweighted link lists are supported and only one link list per project is allowed.

Attributes

For every node multiple types of attributes can be given. The type can be specified as namespace (e.g. DISEASE). One gene can have multiple attributes (such as a gene can be associated to multiple diseases). And vice versa one specific attribute can be distributed over several nodes.

node_id	attribute_id	namespace	name	description
3543	DOID:2825	DISEASE	nose disease	none
25836	DOID:2825	DISEASE	nose disease	none
4852	DOID:2825	DISEASE	nose disease	none
4524	DOID:1936	DISEASE	atherosclerosis	none
4524	DOID:7148	DISEASE	rheumatoid arthritis	none
4524	DOID:4535	DISEASE	hypotrichosis	none

Note The database structure allows also attributes that are organized as an ontology tree. However it is not part of the uploader functionality yet. We plan to provide it with the next release.

Selection Lists

An important feature are selection lists. They define subsets of your nodes that are of interest for specific task. (Such as a gene subset representing a list of patient variants.) They can be loaded separately in a VR session to always have them highlighted. Moreover they serve as the initial set for several analytical procedures such as seeds for a random walk.

Selection lists are a sub category of attributes and can be uploaded in the following way:

node_id	attribute_id	namespace	name	description
3543	my_ID:1	SELECTION	my_set_1	none
4524	my_ID:1	SELECTION	my_set_1	none
25836	my_ID:2	SELECTION	my_set_2	none
4852	my_ID:2	SELECTION	my_set_2	none

Please provide as attribute_id an user defined ID for your selection. The name can be picked up in the VR session.

Labels

x_loc	y_loc	z_loc	text	namespace
0.5000000	0.5000000	0.5000000	C E L L U L A R C O M P O N E N T S	my_layout

A simple way to add textlabels at certain positions to a specific layout (namespace)

Note

USE CASE Gene prioritization

This refers to the chapter 'Application to prioritization of genomic variants' from the publication describing one specific use case for identifying a single disease causing gene out of a set of given variant genes.

In the VRNetzer release 1.1 candidate and seed genes are already contained as a prepared data set. For this specific task we include an extra patient panel that looks like this

It may include patient and diagnosis data and a list of variants. The second tab shows the manually curated seed genes:

From both lists the genes symbols can be clicked to be highlighted in the network while the gene panel shows the gene characteristics for individual inspection. As described in the publication the sets can be analyzed in terms of their network properties for example. In order to show their interconnectivity or lcc size select the network analytics tab on the control panel (turn your left wrist/controller) and use the isolate and re-layout functions. (NOTE: the LINKS needs to be loaded)

To contextualize the variant genes in the functional set of seeds run the random walk on the 3rd tab of side panel. It will highlight the variants (shiny red), seeds (shiny yellow) and linker genes (shiny blue) in the 3D network. A bar chart within the side panel presents a list of genes resulting from the random walk ranked by their visiting probabilty. Depending on the choice of the restart parameter value (high r means higher localization around initial gene set) you will find DOCK2 among the first few candidates.

A force-graph layout of the resulting random-walk network can be inspected separately in the 5th tab of the side panel. The color legend is the same as in the 3D network and the nodes are interactive so that clicking on them refers directly to the 3D network where the gene panel opens as well.

Saving your results makes them accessible from outside the VR session: http://127.0.0.1:5000/_side

VRnet API Documentation

The following function calls are sent from jQuery to the VR module.

The syntax looks like this:

ue4("rw_result", response);

where "rw_results" is the function name and "response" the parameters.

This list will grow in the future.

function	parameters	type	description
LoadDbNodeList	node list object	json	loads a node list into channel A of the layout
LoadDbNodeListB	node list object	json	loads a node list into channel B of the layout (only xyz, no colors)
LoadDbLinkList	link list object	json	loads a link list from db
LoadDbLabelList	label list object	json	loads a label list for channel A of the layout
LoadDbLabelListB	label list object	json	loads a label list for channel B of the layout
Init	---	---	start initialization routine, OnDocumentReady()
Morph	"A" or "B"	string	Morph between layout channel A and B
GetSelection	{ "content": "Name","route": "saveSelection"}	json	Get active selection and do different things depending on route (save it on db in this case)
SetScale	value	float	set Network Scale
SetNodeSize	value	float	set Network SetNodeSize (linksize is also affected by this)
SetLinkSize	value	float	set Network SetLinkSize ATTENTION: big diameters cause lag
SetLinkAlpha	value	float	set Network Link transperancy
SetLight	value	float	set scene light intensity
ShortestPathPoint	"p1" or "p2"	string	get selected node ID from VR and define it either as start or endpoint
ExitIsolate	---	---	exit Isolate Selection Mode (show all links)
Rw_Result	rw object	json	display the random walk results in VR
Julia	{"seeds":[{"node_id":123}...],"variants":[...],"linker":[...]}	json	Gene Priorization example on side panel
ActivateNode	id	int	select and highlight single node in network
ReLayout	node list object	json	show newly created layout of subset in VR (this resorts the link list)
VRkeyboard	route	string	opens a keyboard in VR - after user presses ENTER, typed string is returned to a .js function by the same name as route - so you need to create this
loadSelection	name	string	Deprecated Load selection from csv file

Dependencies

Analytics Module	UI Module	VR Module
Win, Mac, Linux	Win, Mac, Linux	Win 10
python==3.6 click==7.1.2 decorator==4.4.2 fisher==0.1.9 Flask==1.1.2 Flask-Cors==3.0.8 itsdangerous==1.1.0 Jinja2==2.11.2 MarkupSafe==1.1.1 networkx==2.4 numpy==1.19.4 pandas==1.2.0 PyMySQL==0.9.3 python-dateutil==2.8.1 pytz==2020.5 six==1.15.0 Werkzeug==0.16.1 jquery==3.4.1 jquery-ui	python==3.6 Flask==1.1.2 Flask-Cors==3.0.8 jquery==3.4.1 jquery-ui d3v4	Epic Unreal Engine 4.23 WebUI by Tracer Interactive Direct X Runtime SteamVR

Data availability

source	url	version/realease date
GO taxonomy	http://geneontology.org/docs/download-ontology/	Mar 2019
GO annotations	http://current.geneontology.org/products/pages/downloads.html	Mar 2019
DO taxonomy	https://github.com/DiseaseOntology/HumanDiseaseOntology/ tree/main/src/ontology/releases	Dec 2018
DisGeNET	https://www.disgenet.org/downloads	Dec 2018
OMIM	https://www.omim.org/	Feb 2019
HPO taxonomy	https://hpo.jax.org/app/download/ontology	Sep 2018
HPO annotation	https://hpo.jax.org/app/download/annotation	Sep 2018
KeGG	https://www.genome.jp/kegg/genes.html	Feb 2019
BioGRID	https://downloads.thebiogrid.org/	Jun 2019
REACTOME	https://reactome.org/download-data	Jul 2019
Gtex	https://gtexportal.org/home/datasets	V7
Pubmed articles from INDRA	INDRA module	May 2019
INDRA	https://github.com/sorgerlab/indra	May 2019
HIPPIE (Interactome)	http://cbdm-01.zdv.uni-mainz.de/~mschaefer/hippie/download.php	Jul 2017

Software

Python==3.6
JavaScript==ES6
MySQL==8.0
Apache==2.4.29 (Ubuntu)
EPIC Unreal Engine 4

License

Copyright (c) 2021 Menche Lab

This project is licensed under the terms of the MIT license. Check the LICENSE.md file for details.

This Readme:

UI Module:

Analytics Module: