Skip to content

Commit

Permalink
Update info
Browse files Browse the repository at this point in the history
  • Loading branch information
carlosribas committed Jul 26, 2021
1 parent 04a65c4 commit ec6c889
Showing 1 changed file with 19 additions and 64 deletions.
83 changes: 19 additions & 64 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,31 +27,20 @@ running unit-tests on local machine only, it is not using any database or networ
**Requirements**

- [Terraform](https://www.terraform.io)
- [Terraform inventory](https://github.com/adammck/terraform-inventory)
- [Virtual environment](https://virtualenv.pypa.io/en/latest/) with installed dependencies
- [Memcached](https://memcached.org/)

1. Create `terraform/providers.tf` using the `providers.tf.template` file.

2. Generate `sequence_search_rsa` key:

`cd terraform && ssh-keygen -t rsa -b 4096`

See: https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

3. Follow steps in `redeploy.jenkinsfile`.

- Install SSH keys
- Run `terraform init && terraform apply` to create the infrastructure
- Run ansible to create postgres database
- Run ansible to create producer instance
- Run ansible to create consumer instances
3. Run `terraform init && terraform apply` to create the infrastructure

### Development workflow

1. Choose terraform workspace:

`terraform workspace select <env>` where `env` can be `default` or `test`.
`terraform workspace select <env>` where `env` can be `default`, `test` or `covid`.
Once the workspace is selected, terraform will choose the correct `tfstate`
file and will know how to configure ssh keys.

Expand All @@ -60,53 +49,31 @@ running unit-tests on local machine only, it is not using any database or networ
This will check that the infrastructure is up and running, configure ssh keys,
and update ansible inventory on each run.

3. To apply python or ansible changes, run the appropriate ansible playbook:

`ansible-playbook -i hosts ...`

---------------------------

### Additional notes

#### How to upload image with databases to openstack

1. Create an .iso image from `databases` folder on Mac:

`cd sequence_search/consumer`
`hdiutil makehybrid -o databases.iso databases -iso -joliet`

2. To upload image to the cloud first download `openstack.rc` (v2) from Horizon dashboard

3. Source it: `source openstack.rc`, enter your user's password
The installation and configuration of VMs in production is done via Ansible.

4. Create a glance image:
#### How to generate the hosts file used by Ansible

`glance image-create --name sequence_search_databases --disk-format=iso --container-format=bare --file databases.iso`
1. Run `terraform show -json > current_state`

See: https://matt.berther.io/2008/12/14/creating-iso-images-from-a-folder-in-osx/
2. Run this python script to create the `ansible/hosts` file:

#### How to dynamically generate ansible inventory from terraform state
`python hosts.py`

1. (a) Install terraform inventory (https://github.com/adammck/terraform-inventory):
3. You can now run ansible commands with:

`brew install terraform-inventory`
`pushd ansible; ansible-playbook -i hosts <file>`

1. (b) Alternatively, download and install platform-specific distribution from here:
#### Restart the sequence search

https://github.com/adammck/terraform-inventory/releases
In case you need to restart the sequence search, run these commands:

2. Create terraform infrastructure:

`pushd terraform; terraform apply; popd`

3. Generate the ansible inventory from terraform state and save it to hosts file:

`terraform-inventory -inventory terraform/terraform.tfstate > ansible/hosts`

4. You can run ansible commands now with:

`pushd ansible; ...`
`ansible-playbook -i hosts producer.yml --tag restart`

`ansible-playbook -i hosts consumers.yml --tag restart`

#### Frontend

Expand All @@ -117,32 +84,30 @@ The code is available here: https://github.com/RNAcentral/rnacentral-sequence-se

#### Using terraform workspaces for blue-green release

1. Choose a workspace: `terraform workspace select <environment>`
1. Choose a workspace: `terraform workspace select <env>`

Environment can be `default` or `test`. Al subsequent terraform commands
will apply only to that environment.
Env can be `default`, `test` or `covid`. All subsequent terraform commands will apply only to that environment.

2. Apply terraform changes: `terraform apply`

The `main.tf` will automatically configure the correct IP depending on the workspace.

3. Update local ssh config: `ansible-playbook -i hosts localhost.yml`

The IP address will be set depending on the current `terraform.tfstate` which
is enabled by the terraform workspace.
The IP address will be set depending on the current `terraform.tfstate` which is enabled by the terraform workspace.

4. Run any other ansible playbooks: `ansible-playbook -i hosts producer.yml`


#### How to create a load balancer and do blue-green release
#### How to create a load balancer and move from one environment to another

1. `pushd terraform_load_balancer; terraform apply; popd`
1. `pushd terraform/load_balancer; terraform apply; popd`

2. Run the command bellow passing the IPs as variables on the command line:

```
cd ansible_load_balancer
ansible-playbook -i hosts --private-key=../terraform_load_balancer/load_balancer_rsa --extra-vars "main_ip=main.ip.address fallback_ip=fallback.ip.address" load_balancer.yml
ansible-playbook -i hosts --private-key=../terraform/load_balancer/load_balancer_rsa --extra-vars "main_ip=main.ip.address fallback_ip=fallback.ip.address" load_balancer.yml
```
The load balancer is an nginx server that proxies http requests to the currently selected producer machine's `8002`
Expand All @@ -152,16 +117,6 @@ port. If you want to change the port of producer machine, go to `load_balancer.y
If you want to update nginx configuration, make changes in
`ansible_load_balancer/roles/ansible_load_balancer/templates/upstream.conf.js`.
#### Installation in production using Jenkins
To configure Jenkins deployment:
- Install and configure Jenkins
- In Jenkins interface create a one-branch pipeline for each `*.jenkinsfile` in `/jenkins` folder
- In Jenkins upload secret file `openstack.rc` copied from RedHat Horizon dashboard
(Project -> Compute -> Access & Security -> API Access)
- Install python-based dependencies via `pip install -r requirements.txt` (possibly, using a virtualenv)
#### Manual installation in local environment
1. `git clone https://github.com/RNAcentral/rnacentral-sequence-search.git`
Expand Down

0 comments on commit ec6c889

Please sign in to comment.