Support for real time collaboration and model export using celery(#390)

.flake8

@@ -1,6 +1,7 @@
 [flake8]
 max-line-length = 110
-exclude = 
+exclude =
 	./tensorflow_app/caffe-tensorflow,
 	./node_modules/*,
+	*/migrations/*,
 	docs/

.gitignore

@@ -25,4 +25,6 @@ node_modules/
 
 ide/static/bundle/
 
+.vscode/
+
 celerybeat-schedule

.travis.yml

@@ -38,6 +38,8 @@ before_script:
   - psql -c "ALTER ROLE admin SET default_transaction_isolation TO 'read committed'" -U postgres
   - psql -c "ALTER ROLE admin SET timezone TO 'UTC'" -U postgres
   - psql -c "ALTER USER admin CREATEDB" -U postgres
+  - python manage.py makemigrations caffe_app --settings=settings.test
+  - python manage.py migrate --settings=settings.test
 
 script:
   - flake8 ./

README.md

@@ -35,6 +35,39 @@ Setting up Fabrik on your local machine is really easy. You can setup Fabrik usi
     docker-compose up --build
     ```
 
+### Setup Authenticaton for Docker Environment
+1. Go to Github Developer Applications and create a new application. [here](https://github.com/settings/developers)
+
+2. For local deployments the following is what should be used in the options:
+    * Application name: Fabrik
+    * Homepage URL: http://0.0.0.0:8000
+    * Application description: Fabrik
+    * Authorization callback URL: http://0.0.0.0:8000/accounts/github/login/callback/
+
+3. Github will provide you with a Client ID and Secret Key, save these.
+
+4. Create a superuser in django service of docker container
+
+    ```
+    docker-compose run django python manage.py createsuperuser
+    ```
+    Note: Before creating make sure that django service of docker image is running, it can be done by executing ``` docker-compose up ``` followed by ``` Ctrl + C ``` to save docker configuration.
+
+5. Open http://0.0.0.0:8000/admin and login with credentials from step 4.
+
+6. Setting up Social Accounts in django admin
+
+    * Under the ```Social Accounts``` tab open ``` Socialapplications ```, click on ``` Add Social Application ```.
+
+    * Choose  the ``` Provider ``` of social application as ``` Github ``` &  name it ``` Github ```.
+
+    * Add the sites available to the right side, so github is allowed for the current site.
+
+    * Copy and paste your ``` Client ID ``` and ``` Secret Key ``` into the apppropriate fields and Save.
+
+7. Go to ``` Sites ``` tab and update the ``` Domain name ``` to ``` 0.0.0.0:8000 ```.
+
+
 ### Using Virtual Environment
 1. First set up a virtualenv
     ```
@@ -52,11 +85,21 @@ Setting up Fabrik on your local machine is really easy. You can setup Fabrik usi
     ```
     cp settings/dev.sample.py settings/dev.py
     ```
+    Replace the hostname to ``` localhost ``` in settings/dev.py line 14.
 
-4. Install redis server and replace the hostname to 'localhost' in settings/common.py line 99.
+4. Install redis server and replace the hostname to ``` localhost ``` in settings/common.py line 99.
     ```
     sudo apt-get install redis-server
     ```
+    Replace celery result backend in settings/common.py line 122 with
+    ```
+    CELERY_RESULT_BACKEND = 'redis://redis:6379/0'
+    ```
+    Replace celery broker url and result backend hostname to ``` localhost ``` in ide/celery_app.py line 8 with
+    ```
+    broker='redis://redis:6379/0'
+    backend='redis://redis:6379/0'
+    ```
 
 5. If you have Caffe, Keras and Tensorflow already installed on your computer, skip this step
     * For Linux users
@@ -114,6 +157,55 @@ sudo npm install -g webpack
 webpack --progress --watch --colors
 ```
 
+9. To start celery worker
+```
+celery -A ide worker --app=ide.celery_app  --loglevel=info
+```
+
+
+### Setup Authenticaton for Virtual Environment
+1. Go to Github Developer Applications and create a new application. [here](https://github.com/settings/developers)
+
+2. For local deployments the following is what should be used in the options:
+    * Application name: Fabrik
+    * Homepage URL: http://localhost:8000
+    * Application description: Fabrik
+    * Authorization callback URL: http://localhost:8000/accounts/github/login/callback/
+
+3. Github will provide you with a client ID and secret, save these.
+
+4. Create a superuser in django
+
+    ```
+    python manage.py createsuperuser
+    ```
+
+5. Start the application
+
+    ```
+    python manage.py runserver
+    ```
+
+6. Open http://localhost:8000/admin
+
+7. Login with credentials from step
+
+8. Setting up Social Accounts in django admin
+
+    * Under the ```Social Accounts``` tab open ``` Socialapplications ```, click on ``` Add Social Application ```.
+
+    * Choose  the ``` Provider ``` of social application as ``` Github ``` &  name it ``` Github ```.
+
+    * Add the sites available to the right side, so github is allowed for the current site.
+
+    * Copy and paste your ``` Client ID ``` and ``` Secret Key ``` into the apppropriate fields and Save.
+
+9. Go to ``` Sites ``` tab and update the ``` Domain name ``` to ``` localhost:8000 ```.
+
+Note: For testing, you will only need one authentication backend. However, if you want to try out Google's authentication
+then, you will need to follow the same steps as above, but switch out the Github for google.
+
+
 ### Usage
 ```
 python manage.py runserver

backendAPI/admin.py

@@ -0,0 +1,2 @@
+# Register your models here.
+from django.contrib import admin # noqa

backendAPI/apps.py

@@ -0,0 +1,8 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.apps import AppConfig
+
+
+class BackendapiConfig(AppConfig):
+    name = 'backendAPI'

backendAPI/urls.py

@@ -0,0 +1,6 @@
+from django.conf.urls import url
+from views import check_login
+
+urlpatterns = [
+    url(r'^checkLogin$', check_login)
+]

backendAPI/views.py

@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+from django.http import JsonResponse
+from django.contrib.auth.models import User
+
+
+def check_login(request):
+    try:
+        user = User.objects.get(username=request.user.username)
+        user_id = user.id
+        username = 'Anonymous'
+
+        is_authenticated = user.is_authenticated()
+        if (is_authenticated):
+            username = user.username
+
+        return JsonResponse({
+            'result': is_authenticated,
+            'user_id': user_id,
+            'username': username
+        })
+    except Exception as e:
+        return JsonResponse({
+            'result': False,
+            'error': str(e)
+        })

caffe_app/admin.py

@@ -1,5 +1,6 @@
 # Register your models here.
 from django.contrib import admin
-from .models import ModelExport
+from .models import SharedWith, Network
 
-admin.site.register(ModelExport)
+admin.site.register(SharedWith)
+admin.site.register(Network)

caffe_app/consumers.py

@@ -0,0 +1,192 @@
+import json
+import yaml
+import urlparse
+from channels import Group
+from channels.auth import channel_session_user, channel_session_user_from_http
+from caffe_app.models import Network, NetworkVersion, NetworkUpdates
+from ide.views import get_network_version
+from ide.tasks import export_caffe_prototxt, export_keras_json
+
+
+def create_network_version(network, netObj):
+    # creating a unique version of network to allow revert and view hitory
+    network_version = NetworkVersion(network=netObj)
+    network_version.network_def = network
+    network_version.save()
+    return network_version
+
+
+def create_network_update(network_version, updated_data, tag):
+    network_update = NetworkUpdates(network_version=network_version,
+                                    updated_data=updated_data,
+                                    tag=tag)
+    return network_update
+
+
+def fetch_network_version(netObj):
+    network_version = NetworkVersion.objects.filter(network=netObj).order_by('-created_on')[0]
+    updates_batch = NetworkUpdates.objects.filter(network_version=network_version)
+
+    # Batching updates
+    # Note - size of batch is 20 for now, optimization can be done
+    if len(updates_batch) == 2:
+        data = get_network_version(netObj)
+        network_version = NetworkVersion(network=netObj, network_def=json.dumps(data['network']))
+        network_version.save()
+
+        network_update = NetworkUpdates(network_version=network_version,
+                                        updated_data=json.dumps({'nextLayerId': data['next_layer_id']}),
+                                        tag='CheckpointCreated')
+        network_update.save()
+    return network_version
+
+
+@channel_session_user_from_http
+def ws_connect(message):
+    print('connection being established...')
+    message.reply_channel.send({
+        'accept': True
+    })
+    # extracting id of network from url params
+    params = urlparse.parse_qs(message.content['query_string'])
+    networkId = params.get('id', ('Not Supplied',))[0]
+    message.channel_session['networkId'] = networkId
+    # adding socket to a group based on networkId to send updates of network
+    Group('model-{0}'.format(networkId)).add(message.reply_channel)
+
+
+@channel_session_user
+def ws_disconnect(message):
+    networkId = message.channel_session['networkId']
+    Group('model-{0}'.format(networkId)).discard(message.reply_channel)
+    print('disconnected...')
+
+
+@channel_session_user
+def ws_receive(message):
+    print('message received...')
+    # param initialization
+    data = yaml.safe_load(message['text'])
+    action = data['action']
+
+    if ('randomId' in data):
+        randomId = data['randomId']
+
+    if ('networkId' in message.channel_session):
+        networkId = message.channel_session['networkId']
+
+    if (action == 'ExportNet'):
+        # async export call
+        framework = data['framework']
+        net = data['net']
+        net_name = data['net_name']
+        reply_channel = message.reply_channel.name
+
+        if (framework == 'caffe'):
+            export_caffe_prototxt.delay(net, net_name, reply_channel)
+        elif (framework == 'keras'):
+            export_keras_json.delay(net, net_name, False, reply_channel)
+        elif (framework == 'tensorflow'):
+            export_keras_json.delay(net, net_name, True, reply_channel)
+
+    elif (action == 'UpdateHighlight'):
+        add_highlight_to = data['addHighlightTo']
+        remove_highlight_from = data['removeHighlightFrom']
+        user_id = data['userId']
+        highlight_color = data['highlightColor']
+        username = data['username']
+
+        Group('model-{0}'.format(networkId)).send({
+            'text': json.dumps({
+                'addHighlightTo': add_highlight_to,
+                'removeHighlightFrom': remove_highlight_from,
+                'userId': user_id,
+                'action': action,
+                'randomId': randomId,
+                'highlightColor': highlight_color,
+                'username': username
+            })
+        })
+    else:
+        # save changes to database to maintain consistency
+        # get the net object on which update is made
+        netObj = Network.objects.get(id=int(networkId))
+        network_version = fetch_network_version(netObj)
+
+        if (action == 'UpdateParam'):
+            updated_data = {}
+            updated_data['layerId'] = data['layerId']
+            updated_data['param'] = data['param']
+            updated_data['value'] = data['value']
+            updated_data['isProp'] = data['isProp']
+            updated_data['nextLayerId'] = data['nextLayerId']
+
+            network_update = create_network_update(network_version, json.dumps(updated_data), data['action'])
+            network_update.save()
+            # sending update made by one user over all the sessions of open network
+            # Note - conflict resolution still pending
+            Group('model-{0}'.format(networkId)).send({
+                'text': json.dumps({
+                    'layerId': updated_data['layerId'],
+                    'param': updated_data['param'],
+                    'value': updated_data['value'],
+                    'isProp': updated_data['isProp'],
+                    'action': action,
+                    'version_id': 0,
+                    'randomId': randomId
+                })
+            })
+        elif (data['action'] == 'DeleteLayer'):
+            updated_data = {}
+            updated_data['layerId'] = data['layerId']
+            updated_data['nextLayerId'] = data['nextLayerId']
+
+            network_update = create_network_update(network_version, json.dumps(updated_data), data['action'])
+            network_update.save()
+
+            # Note - conflict resolution still pending
+            Group('model-{0}'.format(networkId)).send({
+                'text': json.dumps({
+                    'layerId': updated_data['layerId'],
+                    'action': action,
+                    'version_id': 0,
+                    'randomId': randomId
+                })
+            })
+        elif (action == 'AddLayer'):
+            updated_data = {}
+            updated_data['prevLayerId'] = data['prevLayerId']
+            updated_data['layer'] = data['layer']
+            updated_data['layerId'] = data['layerId']
+            updated_data['nextLayerId'] = data['nextLayerId']
+
+            network_update = create_network_update(network_version, json.dumps(updated_data), data['action'])
+            network_update.save()
+            # sending update made by one user over all the sessions of open network
+            # Note - conflict resolution still pending
+            Group('model-{0}'.format(networkId)).send({
+                'text': json.dumps({
+                    'layer': updated_data['layer'],
+                    'prevLayerId': updated_data['prevLayerId'],
+                    'action': action,
+                    'version_id': 0,
+                    'randomId': randomId
+                })
+            })
+        elif (action == 'AddComment'):
+            updated_data = {}
+            updated_data['layerId'] = data['layerId']
+            updated_data['comment'] = data['comment']
+
+            network_update = create_network_update(network_version, json.dumps(updated_data), data['action'])
+            network_update.save()
+
+            Group('model-{0}'.format(networkId)).send({
+                'text': json.dumps({
+                    'layerId': updated_data['layerId'],
+                    'comment': updated_data['comment'],
+                    'action': action,
+                    'version_id': 0,
+                    'randomId': randomId
+                })
+            })