Skip to content

Latest commit

 

History

History
276 lines (220 loc) · 8.38 KB

architecture.md

File metadata and controls

276 lines (220 loc) · 8.38 KB

Architecture Journey

This document outlines how to grow a project from a monolithic REST API to a microservices architecture.

Beginnings

This is the typical web application starting point. A single codebase that handles all the web traffic, business logic, and data access.

flowchart LR
  subgraph Frontend
    NextJS
  end
  subgraph Backend
    API
    DB
  end
  NextJS --> API
  API --> DB
Loading

v0.1.0 Monolithic REST API

Next up I have added a simple CLI to list feeds. At this point it is important to notice that we have multiple consumers of the API. Writing and maintaining clients by hand is time consuming and error prone. To ease this burden I will also introduce openapi-generator, which can generate clients in multiple languages. We also get the benefit of strong typing in languages like TypeScript and Go moving some errors from runtime to compile time.

flowchart LR
  subgraph Consumers
    UI
    CLI
  end
  subgraph RestClients
    TypeScriptRestClient
    GolangRestClient
  end
  UI --> TypeScriptRestClient
  CLI --> GolangRestClient
  TypeScriptRestClient --> REST
  GolangRestClient --> REST
  REST --> DB
Loading

v1.0.0 Introduce Domain Layer

The domain layer will ensure that business logic is separate from any single service. This will allow standing up new services much easier.

flowchart LR
  subgraph Consumers
    UI
    CLI
  end
  subgraph RestClients
    TypeScriptRestClient
    GolangRestClient
  end

  UI --> TypeScriptRestClient
  CLI --> GolangRestClient
  TypeScriptRestClient --> REST
  GolangRestClient --> REST
  REST --> DomainLayer
  DomainLayer --> DB
Loading

v1.1.1 Add GraphQL and gRPC

This update adds the next iteration of boundaries in our system: GraphQL and gRPC. GraphQL will be our public facing API Gateway while gRPC will be used for internal communication between services.

GraphQL is an excellent choice as it has the ability to generate clients in multiple languages. We also have the benefit of strong typing and the ability to request only the data we need. As the stack evolves, GraphQL will provide a consistent interface to backend services.

GraphQL doesn't require strongly typed clients, but generating them ensures requests and responses are well formed. This also ensures developers are alerted of deprecations or changes in the API at compile time.

flowchart LR
  subgraph Consumers
    UI
    CLI
  end
  subgraph RestClients
    TypeScriptRestClient
    GolangRestClient
  end
  subgraph EdgeVPC
    REST
    GraphQL
  end
  subgraph InternalVPC
    gRPC
    DomainLayer
    DB
  end

  UI --> TypeScriptRestClient
  CLI --> GolangRestClient
  TypeScriptRestClient --> REST
  GolangRestClient --> REST
  REST --> DomainLayer
  DomainLayer --> DB

  GraphQL --> gRPC
  gRPC --> DomainLayer
Loading

v1.3.2 Refactor REST to use GraphQL backend

Phase 3: Refactor REST, Introduce GraphQL Clients.

Now we are in an interesting position where we have a legacy REST API. While the internals have been ported to GraphQL, there are still data transforms that occur at each boundary.

Options:

  • Continue maintaining OpenAPI spec
  • Figure out how to generate OpenAPI spec from GraphQL schema
  • Possibly use gRPC gateway to replace REST API
flowchart LR
  subgraph Consumers
    UI
    CLI
    LegacyCLI
  end
  subgraph GeneratedClients
    subgraph RestClients
      TypeScriptRestClient
      GolangRestClient
    end
    subgraph GraphQLClients
      TypeScriptGraphQLClient
      GolangGraphQLClient
    end
  end

  subgraph EdgeVPC
    REST
    GraphQL
  end

  subgraph InternalVPC
    gRPC
    DomainLayer
    DB
  end

  UI --> TypeScriptGraphQLClient
  CLI --> GolangGraphQLClient

  TypeScriptGraphQLClient --> GraphQL
  GolangGraphQLClient --> GraphQL

  TypeScriptRestClient --> REST
  LegacyCLI --> GolangRestClient
  GolangRestClient --> REST
  REST --> GraphQL

  GraphQL --> gRPC
  gRPC --> DomainLayer
  DomainLayer --> DB
Loading

v1.4.0 GraphQL Gateway powered by Data Pipelines

As of v1.4.0, the system has stabilized around GraphQL as our API Gateway. It is possible to make sweeping changes to the internal system without having to change our external contract. For instance, we can now delete the API service. We can also split up any internal service to support horizontal scaling.

Note: in prod there would be an ingress controller in front of the GraphQL service. This would handle things like rate limiting, authentication, and other concerns.

Components:

Ancillary Services:

---
title: System Overview
---
flowchart LR
  subgraph Clients
    Browser
    CLI
  end

  Browser --> UI
  Browser --> GraphQL
  CLI --> GraphQL
  GraphQL --> RPC
  RPC --> DB

  subgraph ExternalCDN
    UI
  end

  subgraph Services
    GraphQL
    RPC
  end

  subgraph ExternalServices
    DB
  end

  subgraph DataPipelines
    direction RL
    Tasks
    FetchFeeds
    FetchFeeds --> RPC
    RPC <--> Tasks
  end
Loading

In this revision I have added Temporal as the workflow orchestrator. The FetchFeeds workflow is a simple ETL. It fetches rss, stores, transforms into articles, and loads into the database (via domain layer, not directly).

Feed Tasks are ad-hoc, created from RPC. At time of writing this was used to show how to generate fake data.

Fetch Feeds:

  • runs on a continuous schedule
  • mimics a simple Extract Transform Load (ETL) pipeline
  • uses Object Storage locations as input and output

Next Steps

Next the goal will be to start splitting the gRPC backend into separate services to allow for horizontal scaling.

By using GraphQL as our gateway, we can radically refactor the backend without having to update any downstream consumers.

flowchart LR
  GraphQL --> FeedService
  GraphQL --> ArticleService
  GraphQL --> UserService

  subgraph Database
    RMDBS
    NoSQL
  end

  subgraph CloudProvider
    GoogleFirebase
  end

  FeedService --> RMDBS
  ArticleService --> NoSQL
  UserService --> GoogleFirebase
Loading