rolling-scopes-school#9: Application deployment via Elastic Beanstalk…

… (Docker platform) (#2)

* feat: compile server and serverless implementations

* feat: add Dockerfile to build server

details: use Docker Compose to run cart-api comprising of postgres and nest.js app

* refactor: create prisma:deploy and prisma:seed scripts to run with diff envs

* feat: compile prisma seed script

* docs: add comments to .dockerignore

* feat: create Dockerfile to build image with cart api server

* fix: export port as part of Elastic Beanstalk platform Docker requirements

* feat: add cloudwatch streaming config for Elastic Beanstalk application

* feat: add .ebignore as part of Elastic Beanstalk application config

* feat: add npm scripts for Elastic Beanstalk application

details:
1. env-cmd vars expansion is used to pass DATABASE_URL to env vars of EB application
2. Dockerfile is used to build image on EC2 instance (docker-compose.yml is ignored by .ebignore)

* fix: include .ebextensions into Elastic Beanstalk deploy

* rolling-scopes-school#8: Integrate wiht AWS RDS (Postgres) (#1)

* fix: add missing properties to a user's cart

* feat: use esbuild to compile project

* feat: wrap nest app as serverless function

details: create aws cdk stack to deploy lambda handler

* feat: add Prisma ORM to the project

details: define cart and cart_items db schemas

* feat: add postgres docker image for local testing

* feat: add Product model to db schema

* feat: add Prisma seed script to fill db with initial data

* feat: add users table with relation to cart

* fix: indicate correct entry point for cart lambda handler

* refactor: rename prisma models properties

* test: actualize e2e test

* refactor: remove rows in cascade on removing relations

* refactor: drop 'count' column in products table

* feat: use Postgres to store data in cart service

* tests: add integration test for Prisma service

* feat: users service uses prisma service to store/retrieve data

* fix: does not include cdk into source files

* feat: cart controller uses basic auth for get/put methods

details: add e2e tests for cart controller

* feat: add possibility to create orders

details:
1. create 'orders' table
2. update 'checkout' endpoint in cart controller
3. create order via prisma transaction
4. add integration tests for order creation
5. add e2e tests for order creation

* feat: create rest api with proxy method to pass all the data to cart handler

details: reference already existing db instance by its attributes

* fix: include prisma linux query lib into cart lambda source code

* refactor: drop 'products' table

details: cart service should not save products data as its already present in 'products' service, so it will operate 'product_id'/'count' only

* refactor: remove jwt/local auth strategies (use basic only)

* refactor: bootstrap nest application in separate module

details: clearly split server (listening to a port) and serverless implementations

* feat: give final shape to input/output of cart controller/service

details: cart creation/update/checkout implemented fully

* refactor: remove 'calculateCartTotal' as there's no info about a product price

* tests: update integration/e2e tests of cart service

* refactor: squash prisma migrations

* refactor: resolve dest of prisma query lib relatively to output dir

* refactor: use webpack to build nest.js application

details: esbuild does not respect 'experimentalDecorators' which are heavily used by nest.js for injection (see details at https://esbuild.github.io/content-types/#tsconfig-json)

* refactor: remove PrismaModule import in AuthModule

* feat: create rds db instance (postgres) via aws cdk

* fix: correct supertest import in tests

* feat: ignore 'cdk*' files and folders with prettier

* feat: enable source maps for aws cdk build

* feat: create network construct for managing connections between lambda and rds

details:
1. network constructs creates security groups to establish connection between lambda and rds db instance
2. db instance endpoint host/port are used to create database url passed as end to lambda handler
3. lambda handler is deployed to one AZ only (the same availability zone is used by db instance)

* refactor: take server port from env var

* refactor: rename db env vars to conform with aws cdk

* feat: add scripts to deploy/seed local/remote db instance with prisma

* docs: add documentation for aws cdk stack

* refactor: move aws cdk account id/region env vars to config

* tests: actualize aws cdk unit tests

* fix: correct formatting in cdk/README.md

* fix: use .env.local or .env.remote depending where db changes are required

* docs: add documentation for cart service

* feat: add method to delete cart of a user

* docs: add mentioning of .env file used for local development by default

* tests: add Postman collection to test Cart service

details: update README with info about Postman collection

* feat: remove public access to RDS DB instance

* refactor: remove internet <-> RDS security group

* refactor: change eb instance type to t2.micro

details: t2.micro falls under Free Tier 750h/month rule in eu-central-1 (Frankfurt) region

* feat: add http api to proxy requests from HTTPS endpoint to EB HTTP endpoint

* fix: add default headers to http proxy api to support CORS

* docs: add information about docker/eb npm scripts

.dockerignore

@@ -0,0 +1,20 @@
+.git/
+# Contains AWS CDK stack declaration, not used during docker build
+cdk/
+# Contains compiled project files, will be compiled during docker build
+dist/
+node_modules/
+# Contains Postman collection for REST API testing, not used during docker build
+postman/
+# Contains tests, not run during docker build
+test/
+.dockerignore
+.DS_Store
+.env*
+.eslintrc.js
+.gitignore
+.prettierrc
+docker-compose.yml
+Dockerfile
+nest-cli.json
+README.md

.ebextensions/log-streaming.config

@@ -0,0 +1,5 @@
+option_settings:
+  aws:elasticbeanstalk:cloudwatch:logs:
+    StreamLogs: true
+    DeleteOnTerminate: true
+    RetentionInDays: 1

.ebignore

@@ -0,0 +1,15 @@
+# Ignore everything
+*
+# Except
+!.ebextensions
+!migrations
+!prisma
+!src
+!.dockerignore
+!Dockerfile
+!package*
+!tsconfig*
+!webpack.config*
+
+cdk/
+node_modules/

.env.example

@@ -1,3 +1,4 @@
 DATABASE_USERNAME=admin
 DATABASE_PASSWORD=123456
-DATABASE_URL=postgresql://admin:123456@localhost:5432/admin?schema=public
+DATABASE_URL=postgresql://admin:123456@localhost:5432/admin?schema=public
+SERVER_PORT=4000

Dockerfile

@@ -0,0 +1,39 @@
+# syntax=docker/dockerfile:1
+FROM node:20-alpine AS base
+
+FROM base AS install-dev
+WORKDIR /tmp/dev
+COPY package.json package-lock.json ./
+RUN npm ci
+
+FROM base AS install-prod
+WORKDIR /tmp/prod
+COPY package.json package-lock.json ./
+RUN npm ci --omit dev
+
+FROM install-dev AS prisma
+COPY prisma ./prisma
+COPY migrations ./migrations
+RUN npm run prisma:generate
+
+FROM prisma AS build
+ENV NODE_ENV=production
+COPY tsconfig* webpack.config.js ./
+COPY src ./src
+RUN npm run build
+
+FROM base
+WORKDIR /app
+COPY --from=install-prod /tmp/prod/node_modules/@prisma ./node_modules/@prisma/
+COPY --from=install-prod /tmp/prod/node_modules/prisma ./node_modules/prisma/
+COPY --from=install-prod /tmp/prod/node_modules/.bin ./node_modules/.bin/
+
+COPY --from=prisma /tmp/dev/node_modules/.prisma ./node_modules/.prisma/
+COPY --from=prisma /tmp/dev/migrations ./migrations/
+COPY --from=prisma /tmp/dev/prisma ./prisma/
+
+COPY --from=build /tmp/dev/dist/main.js ./dist/
+COPY --from=build /tmp/dev/dist/seed.js ./dist/
+COPY --from=build /tmp/dev/package.json ./
+EXPOSE 4000
+CMD ["dist/main.js"]

README.md

@@ -5,31 +5,28 @@ Cart Service is responsible for creation carts/orders.
 ## NPM scripts
 
 - `build` - builds cart handler;
-
 - `test:integration` - runs integration tests for Prisma service;
-
 - `test:e2e` - runs e2e tests;
-
 - `prisma:generate` - generates Prisma client based on schema files stored in [prisma/schema](./prisma/schema/);
-
 - `prisma:deploy:local` - creates databases based on migration files stored in [migrations](./migrations/);
-
 - `prisma:deploy:remote` - same but uses `.env.remote` to connect to remote Postgres database;
-
 - `prisma:seed:local` - populates Postgres database with initial data (see [seed.ts](./prisma/seed.ts));
-
 - `prisma:seed:remote` - same but uses `.env.remote` to connect to remote Postgres database;
-
 - `docker:up` - Starts DynamoDB local service (see [docker-compose.yml](./docker-compose.yml) for implementation details);
-
-- `docker:down` - Stops DynamoDB local service.
+- `docker:down` - Stops DynamoDB local service;
+- `docker:build` - Builds `hazardsoft/cart-service` Docker image;
+- `docker:push` - Pushes `hazardsoft/cart-service` Docker image to Docker Hub;
+- `eb:init` - Initializes `hazardsoft-cart-api` Elastic Beanstalk application;
+- `eb:create` - Creates `prod` environment with `hazardsoft-cart-api-prod` subdomain for Elastic Beanstalk applciation;
+- `eb:deploy` - Deploys changes to `prod` environment of Elastic Beanstalk application;
+- `eb:destroy` - Destroys all environments of Elastic Beanstalk application.
 
 ## Environment
 
 Copy/paste [.env.example](./.env.example) file and rename it to the following:
 
-1. `.env.local`/`.env` - used to work with local instance of Postgres database;
-2. `.env.remote` - used to work with AWS RDS DB instance of Postgres database;
+1. `.env.local`/`.env` - contains credentials for local instance of Postgres database;
+2. `.env.remote` - contains credentials for AWS RDS DB instance of Postgres database.
 
 ### Populate Postgres database
 
@@ -44,17 +41,21 @@ Prisma consumes `DATABASE_URL` env var in order to connect to the database.
 
 Docker compose is used to spin up docker container with Postgres in order to run integration/e2e tests.
 To prepare database in docker container perform the next steps:
+
 1. run `npm run docker:up` command to spin up docker container with Postges database;
 2. run `npm run prisma:deploy:local` command to create tables in docker container;
 3. run `npm run prisma:seed:local` command to fill Postgres database with initial data.
 
 #### Run Tests
 
 1. [db.integration-spec.ts](./test/db.integration-spec.ts) contains integration tests for [Prisma service](./src/db/prisma.service.ts).
- -  `npm run test:integration` command to run integration tests;
+
+- `npm run test:integration` command to run integration tests;
+
 2. [test/cart.e2e-spec.ts](./test/cart.e2e-spec.ts) contains e2e tests for [Cart controller](./src/cart/cart.controller.ts).
- - `npm run test:e2e` command to run e2e tests.
+
+- `npm run test:e2e` command to run e2e tests.
 
 ### Postman
 
-Use [postman_collection.json](./postman/Cart%20Service.postman_collection.json) collection to test REST API endpoints with [Postman](https://www.postman.com)
+Use [postman_collection.json](./postman/Cart%20Service.postman_collection.json) collection to test REST API endpoints with [Postman](https://www.postman.com)

cdk/.env.example

@@ -1,2 +1,3 @@
 DATABASE_USERNAME=
-DATABASE_PASSWORD=
+DATABASE_PASSWORD=
+CART_SERVICE_URL=

cdk/README.md

@@ -22,6 +22,14 @@ The following env var is passed into cart handler in order to provide access to
 
 - `DATABASE_URL` - generated dynamically based on db instance endpoint host/port values + username/password passed via `.env` file.
 
+### HTTP API
+
+Elastic Beanstalk deploy provides HTTP endpoint, but to integrate it with FE it is necessary to have HTTPS endpoint.
+In order to provide HTTPS endpoint additional HTTP API is created to proxy requests to EC2 instance of EB application.
+The following env var is used to define EB endpoint:
+
+- `CART_SERVICE_URL`
+
 ## NPM scripts
 
 - `lint` - runs ESLint with fix option

cdk/src/config.ts

@@ -9,6 +9,11 @@ export const config = {
       password: process.env.DATABASE_PASSWORD ?? ''
     }
   },
+  servers: {
+    cart: {
+      url: process.env.CART_SERVICE_URL ?? ''
+    }
+  },
   handlers: {
     timeout: 10
   }

cdk/src/constructs/db.ts

@@ -55,7 +55,7 @@ export class CartServiceDatabase extends Construct {
       vpc: props.network.vpc,
       vpcSubnets: props.network.vpcSubnets,
       securityGroups: props.network.securityGroups,
-      publiclyAccessible: true,
+      publiclyAccessible: false,
       port: Number(Port.POSTGRES.toString()),
       performanceInsightRetention: PerformanceInsightRetention.DEFAULT,
       backupRetention: Duration.seconds(0),

cdk/src/constructs/httpApi.ts

@@ -0,0 +1,41 @@
+import { CfnOutput } from 'aws-cdk-lib'
+import { Cors } from 'aws-cdk-lib/aws-apigateway'
+import { CorsHttpMethod, HttpApi, HttpMethod } from 'aws-cdk-lib/aws-apigatewayv2'
+import { HttpUrlIntegration } from 'aws-cdk-lib/aws-apigatewayv2-integrations'
+import { Construct } from 'constructs'
+
+interface CartServiceHttpApiProps {
+  serverUrl: string
+}
+
+export class CartServiceHttpApi extends Construct {
+  constructor(scope: Construct, id: string, props: CartServiceHttpApiProps) {
+    super(scope, id)
+
+    const integration = new HttpUrlIntegration(
+      'CartServiceHttpUrlIntegration',
+      `${props.serverUrl}/{proxy}`
+    )
+
+    const api = new HttpApi(this, 'CartServiceHttpApi', {
+      description: 'HTTP API to proxy requests to EC2 instance of Elastic Beanstalk application',
+      corsPreflight: {
+        allowOrigins: Cors.ALL_ORIGINS,
+        allowHeaders: Cors.DEFAULT_HEADERS,
+        allowMethods: [
+          CorsHttpMethod.GET,
+          CorsHttpMethod.OPTIONS,
+          CorsHttpMethod.POST,
+          CorsHttpMethod.PUT
+        ]
+      }
+    })
+    api.addRoutes({
+      path: '/{proxy+}',
+      methods: [HttpMethod.ANY],
+      integration: integration
+    })
+
+    new CfnOutput(this, 'CartServiceHttpApiUrl', { value: api.apiEndpoint })
+  }
+}

cdk/src/constructs/network.ts

@@ -1,5 +1,4 @@
 import {
-  Peer,
   Port,
   SecurityGroup,
   Vpc,
@@ -12,7 +11,6 @@ import { Construct } from 'constructs'
 export class CartServiceNetwork extends Construct {
   public readonly vpc: IVpc
   public readonly availabilityZone: string
-  private readonly internet2RDSPublicSecurityGroup: ISecurityGroup
   private readonly rds2LambdaPrivateSecurityGroup: ISecurityGroup
   private readonly lambda2RDSPrivateSecurityGroup: ISecurityGroup
 
@@ -31,11 +29,7 @@ export class CartServiceNetwork extends Construct {
       allowAllOutbound: false,
       vpc: this.vpc
     })
-    this.internet2RDSPublicSecurityGroup = new SecurityGroup(this, 'Internet2RDSSecurityGroup', {
-      description: 'Security group for cart service (Internet to RDS)',
-      allowAllOutbound: false,
-      vpc: this.vpc
-    })
+
     this.lambda2RDSPrivateSecurityGroup = new SecurityGroup(this, 'Lambda2RDSSecurityGroup', {
       description: 'Security group for cart service (Lambda to RDS)',
       allowAllOutbound: false,
@@ -52,11 +46,16 @@ export class CartServiceNetwork extends Construct {
       Port.POSTGRES,
       'Allow outgoing traffic to RDS'
     )
+    /* this.internet2RDSPublicSecurityGroup = new SecurityGroup(this, 'Internet2RDSSecurityGroup', {
+      description: 'Security group for cart service (Internet to RDS)',
+      allowAllOutbound: false,
+      vpc: this.vpc
+    })
     this.internet2RDSPublicSecurityGroup.addIngressRule(
       Peer.anyIpv4(),
       Port.POSTGRES,
       `Allow incoming traffic to Postgres from any IP v4`
-    )
+    ) */
   }
 
   getSubnetsForLambda(): SubnetSelection {
@@ -70,7 +69,9 @@ export class CartServiceNetwork extends Construct {
   }
 
   getSecurityGroupsForRDS(): ISecurityGroup[] {
-    return [this.rds2LambdaPrivateSecurityGroup, this.internet2RDSPublicSecurityGroup]
+    // remove public access to RDS DB instance
+    // return [this.rds2LambdaPrivateSecurityGroup, this.internet2RDSPublicSecurityGroup]
+    return [this.rds2LambdaPrivateSecurityGroup]
   }
 
   getSecurityGroupsForLambda(): ISecurityGroup[] {

cdk/src/constructs/api.ts → cdk/src/constructs/restApi.ts

@@ -2,12 +2,12 @@ import { Cors, LambdaIntegration, RestApi } from 'aws-cdk-lib/aws-apigateway'
 import type { IFunction } from 'aws-cdk-lib/aws-lambda'
 import { Construct } from 'constructs'
 
-interface CartServiceApiProps {
+interface CartServiceRestApiProps {
   cartHandler: IFunction
 }
 
-export class CartServiceApi extends Construct {
-  constructor(scope: Construct, id: string, props: CartServiceApiProps) {
+export class CartServiceRestApi extends Construct {
+  constructor(scope: Construct, id: string, props: CartServiceRestApiProps) {
     super(scope, id)
 
     const api = new RestApi(this, 'CartServiceApi')

cdk/src/stack.ts

@@ -3,9 +3,10 @@ import { CartServiceHandlers } from './constructs/handlers'
 import type { Construct } from 'constructs'
 import { config } from './config'
 import { CartServiceDatabase } from './constructs/db'
-import { CartServiceApi } from './constructs/api'
+import { CartServiceRestApi } from './constructs/restApi'
 import { CartServiceNetwork } from './constructs/network'
 import { createDatabaseUrl } from './helpers/db'
+import { CartServiceHttpApi } from './constructs/httpApi'
 
 export class CartService extends Stack {
   constructor(scope: Construct, id: string) {
@@ -52,8 +53,12 @@ export class CartService extends Stack {
 
     db.db.grantConnect(handlers.cartHandler, config.database.credentials.username)
 
-    new CartServiceApi(this, 'CartServiceApi', {
+    new CartServiceRestApi(this, 'CartServiceApi', {
       cartHandler: handlers.cartHandler
     })
+
+    new CartServiceHttpApi(this, 'CartServiceHttpApi', {
+      serverUrl: config.servers.cart.url
+    })
   }
 }

cdk/tests/cdk.test.ts

@@ -14,16 +14,13 @@ describe('Cart Service AWS CDK Stack Tests', () => {
   })
 
   test('should have 3 security groups with ingress/egress rules', () => {
-    template.resourceCountIs('AWS::EC2::SecurityGroup', 3)
+    template.resourceCountIs('AWS::EC2::SecurityGroup', 2)
     template.resourceCountIs('AWS::EC2::SecurityGroupIngress', 1)
     template.resourceCountIs('AWS::EC2::SecurityGroupEgress', 1)
 
     template.hasResourceProperties('AWS::EC2::SecurityGroup', {
       GroupDescription: 'Security group for cart service (RDS to Lambda)'
     })
-    template.hasResourceProperties('AWS::EC2::SecurityGroup', {
-      GroupDescription: 'Security group for cart service (Internet to RDS)'
-    })
     template.hasResourceProperties('AWS::EC2::SecurityGroup', {
       GroupDescription: 'Security group for cart service (Lambda to RDS)'
     })
@@ -39,7 +36,7 @@ describe('Cart Service AWS CDK Stack Tests', () => {
       DBInstanceClass: 'db.t3.micro',
       Engine: 'postgres',
       MultiAZ: false,
-      PubliclyAccessible: true
+      PubliclyAccessible: false
     })
   })
 
@@ -58,6 +55,21 @@ describe('Cart Service AWS CDK Stack Tests', () => {
     template.hasResourceProperties('AWS::ApiGateway::Method', {
       HttpMethod: 'OPTIONS'
     })
+  })
+
+  test('should have HTTP API', () => {
+    template.resourceCountIs('AWS::ApiGatewayV2::Api', 1)
+
+    template.resourceCountIs('AWS::ApiGatewayV2::Route', 1)
+    template.hasResourceProperties('AWS::ApiGatewayV2::Route', {
+      RouteKey: 'ANY /{proxy+}'
+    })
+
+    template.resourceCountIs('AWS::ApiGatewayV2::Integration', 1)
+    template.hasResourceProperties('AWS::ApiGatewayV2::Integration', {
+      IntegrationMethod: 'ANY',
+      IntegrationType: 'HTTP_PROXY'
+    })
 
   })
 })