Skip to content

Commit

Permalink
[dy] Staging/production GitHub Actions docs (mage-ai#2609)
Browse files Browse the repository at this point in the history 
* [dy] Add docs

* [dy] Add docs for staging/production github actions

* [dy] Separate to staging and production task
  • Loading branch information
@dy46
dy46 authored May 9, 2023
1 parent 634cf76 commit 44028b0
Show file tree
Hide file tree
Showing 3 changed files with 232 additions and 0 deletions.
1 change: 1 addition & 0 deletions docs/mint.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,7 @@
"pages": [
"production/ci-cd/local-cloud/repository-setup",
"production/ci-cd/local-cloud/github-actions",
"production/ci-cd/staging-production/github-actions",
"production/ci-cd/local-cloud/gitlab-ci-cd",
"production/ci-cd/local-cloud/buildkite"
]
Expand Down
102 changes: 102 additions & 0 deletions docs/production/ci-cd/staging-production/github-actions.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
title: "GitHub Actions"
description: "Development (local), staging (cloud), and production (cloud) using GitHub Actions"
sidebarTitle: "Deploy with staging environment"
---

## Mage project setup

Follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).

---

## GitHub Actions setup

1. Create a new repository on GitHub.
2. Open your repository on GitHub, then click the tab labeled **Settings**.
3. Click the section labeled **Secrets and variables** on the left hand side to expand it.
4. Create separate staging and production GitHub environments in the **Environments** section.
1. You can also choose to require approval before running jobs in your production environment.
More information [here](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers).
2. You may need to change the `environment` variable in the jobs section of the Github Actions below
based on the name of your GitHub environments.
5. Click the link labeled **Actions**.
6. Click the button labeled **New repository secret** in the top right corner.
7. Follow the instructions below for your specific cloud provider:

---

## AWS

1. If you haven’t already, create a new AWS ECR repository.
2. You’ll need AWS credentials with the following policy permissions:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:RegisterTaskDefinition",
"ecs:UpdateService",
"iam:PassRole"
],
"Resource": "*"
}
]
}
```
3. In the field labeled **Name**, enter the value `AWS_ACCESS_KEY_ID`.
4. In the field labeled **Secret**, enter your AWS Access Key ID.
5. Click the button labeled **Add secret** to save.
6. Add a 2nd secret by clicking the button labeled **New repository secret** in
the top right corner.
7. In the field labeled **Name**, enter the value `AWS_SECRET_ACCESS_KEY`.
8. In the field labeled **Secret**, enter your AWS Secret Access Key.
9. Click the button labeled **Add secret** to save.
10. Click on the tab labeled **Actions**.
11. On the left side, click the button labeled **New workflow**.
12. Find the link labeled **`set up a workflow yourself`** and click it.
13. Copy the contents from the GitHub Action YAML file for AWS at
[templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml),
and paste it into the textarea.
14. Change the following values under the key labeled `env`:

```yaml
env:
AWS_REGION: ...
CONTAINER_NAME: ...
ECR_REPOSITORY: ...
ECS_CLUSTER: ...
ECS_STAGING_SERVICE: ...
ECS_PRODUCTION_SERVICE: ...
ECS_STAGING_TASK_DEFINITION: ...
ECS_PRODUCTION_TASK_DEFINITION: ...
```
| Key | Description | Sample value |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `AWS_REGION` | Region of your AWS ECS cluster. | `us-west-2` |
| `CONTAINER_NAME` | Set this to the name of the container in the containerDefinitions section of your task definition. | `mage-data-production-container` |
| `ECR_REPOSITORY` | The name of the AWS ECR repository you created to store your Docker images. | `mage-data` |
| `ECS_CLUSTER` | The name of your AWS ECS cluster. | `mage-production-cluster` |
| `ECS_STAGING_SERVICE` | The name of your AWS ECS staging service. | `mage-production-cluster` |
| `ECS_PRODUCTION_SERVICE` | The name of your AWS ECS production service. | `mage-production-ecs-service` |
| `ECS_STAGING_TASK_DEFINITION` | Go to your AWS ECS task definition for the staging service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
| `ECS_PRODUCTION_TASK_DEFINITION` | Go to your AWS ECS task definition for the production service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |

1. Click the button labeled **Start commit** in the top right corner.
1. Click the button labeled **Commit new file**.
1. Every time you merge a pull request into the master branch, this GitHub
Action will run, building a Docker image using your GitHub code, then
updating AWS ECS to use the new image with the updated code.
129 changes: 129 additions & 0 deletions templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
name: Deploy to Amazon ECS staging and production

on:
push:
branches:
- master

env:
AWS_REGION: MY_AWS_REGION # set this to your preferred AWS region, e.g. us-west-1
ECR_REPOSITORY: MY_ECR_REPOSITORY # set this to your Amazon ECR repository name
ECS_STAGING_SERVICE: MY_STAGING_ECS_SERVICE # set this to your Amazon ECS staging service name
ECS_PRODUCTION_SERVICE: MY_PROD_ECS_SERVICE # set this to your Amazon ECS production service name
ECS_CLUSTER: MY_ECS_CLUSTER # set this to your Amazon ECS cluster name
ECS_STAGING_TASK_DEFINITION: MY_ECS_TASK_DEFINITION # set this to the path to your Amazon ECS staging task definition
# file, e.g. .aws/task-definition.json
ECS_PRODUCTION_TASK_DEFINITION: MY_ECS_TASK_DEFINITION # set this to the path to your Amazon ECS production task definition
# file, e.g. .aws/task-definition.json
CONTAINER_NAME: MY_CONTAINER_NAME # set this to the name of the container in the
# containerDefinitions section of your task definition

jobs:
push-to-ecr:
name: Push image to AWS ECR
runs-on: ubuntu-latest

outputs:
image: ${{ steps.build-image.outputs.image }}

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
mask-aws-account-id: 'no'

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@aaf69d68aa3fb14c1d5a6be9ac61fe15b48453a2

- name: Build, tag, and push image to Amazon ECR
id: build-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ github.sha }}
run: |
# Build a docker container and
# push it to ECR so that it can
# be deployed to ECS.
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
echo "::set-output name=image::$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"
deploy-staging:
name: Deploy staging
runs-on: ubuntu-latest
environment: staging
needs: push-to-ecr

outputs:
image: ${{ steps.build-image.outputs.image }}

steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}

- name: Download task definition
run: |
aws ecs describe-task-definition --task-definition ${{ env.ECS_STAGING_TASK_DEFINITION }} \
--query taskDefinition > task-definition.json
- name: Fill in the new image ID in the Amazon ECS task definition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@v1
with:
task-definition: task-definition.json
container-name: ${{ env.CONTAINER_NAME }}
image: ${{ needs.push-to-ecr.outputs.image }}

- name: Deploy Amazon ECS staging task definition
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: ${{ env.ECS_STAGING_SERVICE }}
cluster: ${{ env.ECS_CLUSTER }}
wait-for-service-stability: true

deploy-production:
name: Deploy production
runs-on: ubuntu-latest
environment: production
needs: [push-to-ecr, deploy-staging]

steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}

- name: Download task definition
run: |
aws ecs describe-task-definition --task-definition ${{ env.ECS_PRODUCTION_TASK_DEFINITION }} \
--query taskDefinition > task-definition.json
- name: Fill in the new image ID in the Amazon ECS task definition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@v1
with:
task-definition: task-definition.json
container-name: ${{ env.CONTAINER_NAME }}
image: ${{ needs.push-to-ecr.outputs.image }}

- name: Deploy Amazon ECS production task definition
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: ${{ env.ECS_PRODUCTION_SERVICE }}
cluster: ${{ env.ECS_CLUSTER }}
wait-for-service-stability: true

0 comments on commit 44028b0

Please sign in to comment.