Skip to content

User Guide

Jump to bottom
flug edited this page Nov 30, 2020 · 23 revisions

The following section describes the usage of the TyphonDL toolset. It is assumed that the user has already downloaded and installed the various tools and plugins needed and created a TyphonML model

For a full description of the Typhon Deployment language see D3.4 and for a full description of the tools see D3.5

A TyphonDL model can be created with the help of the TyphonDL Creation Wizard from the ML model. The wizard uses the previously defined (default or use-case specific) Templates and creates a TyphonDL model file and additional model files for every database that can be edited with the textual editor. When the DL model is ready, the TyphonDL Script Generator can be used to generate technology dependent deployment scripts.

Templates

The TyphonDL plugin comes with a set of default DB and DBType templates, that can be viewed, imported, exported and edited in Eclipse → Window → Preferences → TyphonDL → Templates (see Figure 1). Here, additional templates can be added, or company specific DB settings can be defined and used for creating a new Polystore deployment.

Figure 1 TyphonDL DB Template preferences

TyphonDL Creation Wizard

To create a TyphonDL model from a TyphonML model the TyphonDL Wizard has to be started by selecting the given ML model (.xmi file) and selecting Create TyphonDL model in the Typhon context menu. On the first page of the wizard (see Figure 2) the name for the TyphonDL model has to be entered and a deployment technology such as Docker Compose, or Kubernetes has to be chosen from a dropdown menu. The selected technology will be included in the model in the form of Clustertype which is used when defining a Cluster. If Kubernetes is chosen, the path to a kubeconfig file can be given.

The Analytics component (see D5.5) can be activated and deployment scripts can be created to either run it alongside the other Polystore components, or to run it on a different machine. An already running Analytics component can also be added to the model by giving its URI. The deployment Scripts to run the Natural Language Processing Component on a Docker Swarm can also be generated by selecting the respective checkbox.

Figure 2 TyphonDL Creation Wizard: page one

On the next page the Polystore components API, UI and QL server can be configured (see Figure 3). If Swarm Mode or Kubernetes is used, it is possible to scale the stateless parts of the Polystore, i.e. the API and the QL server. Resource limits and reservations can be given to the API and QL server. And the published port of the API and UI can be adjusted.

Figure 3 TyphonDL Creation Wizard: Configuring the API, UI and QL server

If the Analytics component is to be generated, an optional page (see Figure 4) appears after the first one. Here, the Analytics component can be configured, including the published ports of the Evolution component.

Figure 4 TyphonDL Creation Wizard: Configuring the Analytics component (Docker Compose vs. Kubernetes)

TyphonML provides an XMI representation of the ML model that is parsed by the TyphonDL Wizard and that filters out the databases to be deployed by TyphonDL. For each database the second page of the wizard (see Figure 5) provides the possibility to choose one of the following options:

  1. Use a pre-existing DB model file if a file with the name .tdl exists in the project folder.
  2. Create a new DB model object by choosing a template from the drop down menu.
  3. Use an existing externally running database. A DB model object with the flag external, an URI and the DBType of the selected template is created.
  4. If Kubernetes is chosen on the first page, the option to use a Helm Chart is added. Here, one of the templates already containing a HelmList should be chosen, their names all start with “Helm”. Otherwise a new default HelmList using bitnami as Helm Repo is created. In each of the above cases, the resulting DB model object is cached in the Creation Wizard for further configuration on the next pages.

Figure 5 TyphonDL Creation Wizard: Choosing the DBMS for each database (Docker Compose vs. Kubernetes)

In Figure 6 an example DB configuration is shown. If the DB is not set to external, a Container model object for each database is created and cached together with the DB object in the Wizard. The Container gets an URI object with the value <containerName>:<containerPort>. This URI is parsed by the API to know where to reach each database.

Figure 6 TyphonDL Creation Wizard: Further database configuration (MariaDB container)

Here database settings for VehicleMetadataDB are presented. The previously chosen DBType is shown on the top of the page – here MariaDB (compare to Figure 5).

  1. Credentials: The template (see Figure 1) has a given username (root) and only allows to choose the password. The Wizard provides the possibility to generate a 16 digit password containing small and capital letters and numbers.
  2. Image used: If a different image version should be used, it can be defined in the “Image used” group.
  3. Container Resources: Container resources can be defined by checking the respective checkboxes. This will add a Resources object to the Container. CPU is measured in CPU units, given as the fragment of available processing time (0.2 = 20%). Memory is measured in bytes and is expressed as integer using one of these suffixes: T, G, M, K.
  4. Ports: It’s possible - though not recommended in production - to publish a database container with a given “Published Port” in the “Ports” group. This will add a Ports object to the Container.
  5. Volumes: At the bottom of this page volume specifications can be entered. The template has a default named volume with values for the volume path, volume name and the volume type for each DBMS. To add a volume, clicking on the "Add" button results in the "New Volume" pop-up window. Here, the volume name, the volume type and the path for the volume can be entered. For Docker-Compose, by default the volume type is selected as 'volume'. To add a new volume at least the volume path must be entered. When a (or multiple) volume(s) is/are added, a Volumes object is added to the Container consisting of each added volume.

An example of how to add a volume is illustrated in Figure 7 as part of configuration options for the MongoDB VehicleDataDB container.

Figure 7 TyphonDL Creation Wizard: Further database configuration (MongoDB container) and adding a new volume

  1. Replication: Additionally, to the options above, it’s allowed to replicate the MongoDB if Docker Compose is used. If the Primary/Secondary option is chosen, a Replication object is added to the Container. The number of total Replicas denotes the number of additionally created containers.

On the left side of Figure 8, the database settings for VehicleDataDB, an external MongoDB (compare with the checkbox in Figure 5:left) are presented. Additionally, to setting the Credentials, the user has to give an URI pointing to the database in the “Database Address” group. An example for using Helm charts in the DB AppData (compare with the checkbox in Figure 5:right) is given on the right side of Figure 7. The template for MariaDB Galera already contains the repository settings. The user can specify the use of a custom values file. If the valuesFile field contains the repository name (here “bitnami”), the default values provided by the chart are taken.

Figure 8 TyphonDL Creation Wizard: Further database configuration (MongoDB external database vs. MariaDB Galera Cluster)

When the wizard is finished, the following TyphonDL files get added to the project:

  • TyphonDL model file with the name that was given in the wizard-
  • Properties file needed to generate deployment scripts.
  • One model file for each database.
  • One model file containing the DBTypes.

TyphonDL Editor

Xtext provides a textual editor with syntax highlighting, auto completion and an outline view. If the project that includes the models holds an Xtext nature, the TyphonDL Creation Wizard automatically adds it to the project, and linking between files shown in Figure 9 is also provided. The TyphonDL Creation Wizard already creates a valid TyphonDL model, comprehensive enough to generate polystore deployment scripts, but the user can still add additional information. When Kubernetes is chosen, the Platformtype is automatically set to “minikube ”, a testing environment. A different Platform Type can easily be used by changing the value of Platformtype and adding a “kubeconfig” Key_Values to the Cluster. The “kubeconfig” file can be downloaded from the cluster provider. Example for using AWS:

platformtype AWS
platform platformName : AWS {
	cluster clusterName : Kubernetes {
	    kubeconfig = /path/to/downloaded/kubeconfig.yaml;

Figure 9 TyphonDL textual editor with syntax highlighting and auto completion

Refactoring the clustertype

Renaming the clustertype (by using Shift+Alt+R or Right click -> Rename Element) opens a dialog (see Figure 10:left) in which the user can select a new clustertype and adjust API's, UI's and DB's published ports. If the Analytics component is used, another dialog opens, allowing the user to adjust the analytics settings (see Figure 10:right). Changing the clustertype without using refactoring may result in invalid deployment scripts.

Figure 10 TyphonDL refactor clustertype

TyphonDL Script Generation

To create deployment scripts the TyphonDL Script Generator has to be started by selecting the created and completed DL model (main model file) and choosing Generate Deployment Scripts in the TyphonDL context menu. A folder with the name of the DL model is generated. It contains all files necessary to run the Polystore deployment.

  1. If Docker Compose was chosen, a Service is created for every database and the Polystore can be started by running $ docker-compose up -d. If the DL model contains Resources, the Polystore has to be started by running $ docker stack deploy with Docker running in Swarm Mode. Otherwise the resource definition is ignored. The user can also setup Docker in Swarm Mode using multiple worker nodes and deploy the Polystore as a stack.
  2. If Kubernetes was chosen, a Deployment and a Service to connect to the Pod(s) created by the Deployment is created for every database and the Polystore can be started by executing $ sh deploy.sh.
Clone this wiki locally