Add a Docker compose file (#215)

* feat: Add docker-compose file

Adds a docker compose file which can be used for easily getting up and
running with a dev environment. Adds material pulling into the
Dockerfile with controllable behaviour via the MATERIAL_METHOD build-arg,
either 'copy' in a local folder or pull the repo described in a local
config yaml file. Also moves database migration to the entrypoint script
so that a working database is not required at build-time.

* fix: Remove wip comments from compose file

* fix: Refer to correct .material file with volume mount config

* fix: Ran prettier on README

* feat: Adds cache-busting for git pull

Adds support for cache-busting the git-pull of the material repo,
meaning that the pull section of the dockerfile will not be run
from the cached layer if the CACHE_BUST environment variable is
changed between calls to `docker build` or `docker-compose build`.

So far there seems no way to dynamically set the environment
variable from within the compose file, so a script is included to
do this manually via export, equally a prepended env variable could
be used when calling `docker-compose build` or `... up --build`.

* feat: Add nginx and qdrant to docker-compose

* move docker compose instructions to docs

* clean up nginx conf

* added workflow to test docker compose build

* sigh prettier

* removed mount material to allow pulled material in container

fixed some issues with the shell scripts

---------

Co-authored-by: Alasdair Wilson <[email protected]>

.env

@@ -8,7 +8,7 @@ MATERIAL_DIR=".material"
 YAML_TEMPLATE="config/oxford.yaml"
 NEXT_PUBLIC_MATERIAL_URL="https://github.com/UNIVERSE-HPC/course-material"
 NEXT_PUBLIC_BASEPATH=""
-NEXTAUTH_URL=http://localhost:3002/gutenberg/api/auth
+NEXTAUTH_URL=http://localhost/gutenberg/api/auth
 NEXTAUTH_SECRET="secret here"
 
 GITHUB_SECRET="github_secret"
@@ -19,4 +19,8 @@ NEXT_PUBLIC_ENABLE_SEARCH=true
 OPENAI_API_KEY="openai_api_key"
 QDRANT_COLLECTION_NAME="gutenberg"
 QDRANT_API_KEY=""
-QDRANT_DB_URL=""
+QDRANT_DB_URL=""
+
+NGINX_HOST="localhost"
+NGINX_PORT="80"
+NGINX_IMAGE_TAG="latest"

.github/workflows/test.yml

@@ -87,6 +87,19 @@ jobs:
       - name: Run component tests
         run: ${{ env.MANAGER }} component
 
+  docker-build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: docker compose build
+        run: docker-compose -f docker-compose.yml build
+      - name: docker compose up
+        run: docker-compose -f docker-compose.yml up -d
+      - name: docker compose ps
+        run: docker-compose -f docker-compose.yml ps
+      - name: docker compose down
+        run: docker-compose -f docker-compose.yml down
+
   deploy:
     name: Deploy to staging
     if: github.event_name == 'push' && github.ref == 'refs/heads/main'
@@ -103,7 +116,7 @@ jobs:
         run: corepack enable
       - uses: actions/setup-python@v5
         with:
-          python-version: '3.12'
+          python-version: "3.12"
       - name: Checkout course material
         shell: bash
         run: |

Dockerfile

@@ -1,9 +1,76 @@
-# Install dependencies only when needed
-FROM node:20.9-alpine AS builder
+# This dictates the branch of the Dockerfile to use, i.e. to pull, copy or use a 
+# placeholder for mounting the course material in. The appropriate values for 
+# $MATERIAL_METHOD are therefore ["pull", "copy", "placeholder"]
+ARG MATERIAL_METHOD=copy
+
+# Define the Python version to use
+ARG PYTHON_VERSION=3.12.3-slim
+ARG NODE_VERSION=20.9-alpine
+
+####
+# MATERIAL OPTION: PULL
+# Pull course material into the app directory from a defined repo. Default is
+# the Oxford course material repo defined in the config/oxford.yaml file.
+FROM python:${PYTHON_VERSION} AS pull_material
+
+# Variables for the material pull script
+ARG YAML_TEMPLATE=config/oxford.yaml
+ARG MATERIAL_DIR=.material
+ARG CACHE_BUST=20240514
+
+# Copy the scripts and config files into the container
+ONBUILD WORKDIR /app
+ONBUILD COPY ../scripts/ /app/scripts/
+ONBUILD COPY ../config/ /app/config/
+
+# Install dependencies and clean caches afterwards 
+ONBUILD RUN \
+    apt-get update && \
+    apt-get install -y git && \
+    apt-get clean && \
+    pip install --upgrade pip setuptools wheel && \
+    pip install -r scripts/python_requirements.txt && \ 
+    pip cache purge
+# pull the material into the container with git
+ONBUILD RUN \
+    echo ${CACHE_BUST} && \
+    python scripts/pull_material.py
+
+####
+# MATERIAL OPTION: COPY
+# Copy existing course material into this container for the build
+FROM node:${NODE_VERSION} AS copy_material
+
+ARG MATERIAL_DIR=.material
+# Material dir may not exist yet, so we pass COPY a file which we know will be 
+# in context (i.e. the entrypoint script) to ensure the COPY command doesn't 
+# fail 
+ONBUILD COPY ../entrypoint.sh ../${MATERIAL_DIR}* /app/${MATERIAL_DIR}/
+
+
+####
+# MATERIAL OPTION: PLACEHOLDER
+# Make a placeholder directory for the material so we can mount over it later on
+FROM node:${NODE_VERSION} AS placeholder_material
+
+ARG MATERIAL_DIR=.material
+ONBUILD RUN \
+    mkdir -p /app/${MATERIAL_DIR}/HPCu && \
+    touch /app/${MATERIAL_DIR}/HPCu/index.md
+
+
+# Here we use the MATERIAL_METHOD arg to determine which method we use to get 
+# material into our container for build time.
+FROM ${MATERIAL_METHOD}_material as material
+
+#### 
+# BUILDER
+FROM node:${NODE_VERSION} AS builder
 # Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
 RUN apk add --no-cache libc6-compat
 WORKDIR /app
 RUN echo node --version
+# Install dependencies only when needed
 COPY package.json .
 COPY yarn.lock .
 COPY .yarnrc.yml .
@@ -29,7 +96,10 @@ ARG DATABASE_URL
 ENV DATABASE_URL ${DATABASE_URL}
 ARG NEXT_PUBLIC_PLAUSIBLE_DOMAIN
 ENV NEXT_PUBLIC_PLAUSIBLE_DOMAIN ${NEXT_PUBLIC_PLAUSIBLE_DOMAIN}
-RUN npx prisma migrate deploy
+
+# Copy the course material into this container for the build
+ARG MATERIAL_DIR=.material
+COPY --from=material /app/${MATERIAL_DIR} /app/${MATERIAL_DIR}
 
 RUN yarn build
 
@@ -45,6 +115,7 @@ ENV NEXT_TELEMETRY_DISABLED 1
 RUN addgroup --system --gid 1001 nodejs
 RUN adduser --system --uid 1001 nextjs
 COPY --from=builder --chown=nextjs:nextjs /app ./
+COPY ./prisma /app/prisma
 
 USER nextjs
 

README.md

@@ -1,4 +1,4 @@
-# Gutenberg
+# Gutenberg
 
 [![Build and Test](https://github.com/OxfordRSE/gutenberg/actions/workflows/test.yml/badge.svg)](https://github.com/OxfordRSE/gutenberg/actions/workflows/test.yml)
 
@@ -20,7 +20,7 @@ To see our full documentation, please visit [our documentation site](https://blo
 
 To get a development environment running, follow the instructions below:
 
-## Getting started
+## Getting started (installing locally)
 
 Setting up a local development requires the following software:
 
@@ -29,7 +29,7 @@ Setting up a local development requires the following software:
 - `node` version v18.19.1 or later - packaged with `npm`
 - `corepack` version `0.28.0` or later - install with `npm install -g corepack`
 - `yarn` version `3.3.0` or later - installed via `corepack` with `yarn --version`
-- For developing the renderer rather than just materials, you'll also need a `postgres` instance
+- For developing the full web application functionality rather than just previewing materials, you'll also need a `postgres` instance
 
 ### 1. Clone repository
 
@@ -104,6 +104,10 @@ This means that your edits will be reflected instantly.
 
 Now everything should be set up for local development of the core systems.
 
+## Getting started (using Docker-compose)
+
+Gutenberg also has a deployable dev environment using docker-compose. If you wish to use this alternative then see [getting started using docker compose](blog.oxrse.uk/gutenberg/getting-started-docker-compose).
+
 ## Developing material
 
 You can develop material by editing the `.material` folder.

docker-compose.dev.yml

@@ -0,0 +1,37 @@
+version: '3.9'
+
+x-vol-args: &args
+  volumes: 
+    - ./.material:/app/.material
+
+services:
+  db:
+    image: postgres
+    restart: always
+    shm_size: 128mb
+    environment:
+      POSTGRES_PASSWORD: super-secret-password
+      POSTGRES_USER: postgres
+    ports:
+      - "5432:5432"
+
+  gutenberg:
+    image: gutenberg
+    build:
+      context: .
+      dockerfile: Dockerfile
+      args: 
+        CACHE_BUST: ${CACHE_BUST}
+        MATERIAL_METHOD: "pull"
+        MATERIAL_DIR: ".material"
+        YAML_TEMPLATE: "config/oxford.yaml"
+    ports:
+      - "3000:3000"
+    links: 
+      - db
+    <<: *args
+    depends_on:
+      db:
+        condition: service_started
+    environment:
+      DATABASE_URL: postgresql://postgres:super-secret-password@db:5432

docker-compose.yml

@@ -0,0 +1,61 @@
+# Use postgres/example user/password credentials
+version: "3.9"
+
+services:
+  nginx:
+    image: nginx:${NGINX_IMAGE_TAG} # stable-alpine3.19
+    volumes:
+      - ./nginx:/etc/nginx/templates
+    ports:
+      - "80:8080"
+    environment:
+      - NGINX_HOST=localhost
+      - NGINX_PORT=8080
+      - NGINX_PROXY_PASS=http://gutenberg:3000
+    depends_on:
+      gutenberg:
+        condition: service_started
+
+  db:
+    image: postgres
+    restart: always
+    shm_size: 128mb
+    environment:
+      POSTGRES_PASSWORD: super-secret-password
+      POSTGRES_USER: postgres
+    ports:
+      - "5432:5432"
+    volumes:
+      - db-data:/var/lib/postgresql/data
+
+  gutenberg:
+    image: gutenberg
+    build:
+      context: .
+      dockerfile: Dockerfile
+      args:
+        CACHE_BUST: ${CACHE_BUST}
+        MATERIAL_METHOD: "pull"
+        MATERIAL_DIR: ".material"
+        YAML_TEMPLATE: "config/oxford.yaml"
+    links:
+      - db
+    depends_on:
+      db:
+        condition: service_started
+    environment:
+      DATABASE_URL: postgresql://postgres:super-secret-password@db:5432
+
+  qdrant:
+    image: qdrant
+    build:
+      context: qdrant/.
+      dockerfile: Dockerfile
+    ports:
+      - "6333:6333"
+    volumes:
+      - qdrant_data:/qdrant/storage
+
+volumes:
+  db-data:
+  qdrant_data:

docs/development/docker.md

@@ -0,0 +1,42 @@
+---
+Title: Deploying with Docker-compose
+permalink: /gutenberg/docker
+---
+
+Gutenberg can be deployed both locally and to production using docker-compose.
+
+- `git` version 2.43.0 or later
+- `docker` version 18.03 or later
+
+Both of which will need to be installed following instructions for your system.
+
+Then we can get started by cloning the repository with git and entering into it with `cd`:
+
+```bash
+git clone https://github.com/OxfordRSE/gutenberg.git`
+cd gutenberg
+```
+
+From here we can simply run:
+
+```
+docker-compose up
+```
+
+which should make gutenberg available to you at http://localhost:3000. This will
+build a Docker image the first time you run it (assuming you don't already have
+an image called `gutenburg` in your local registry) from the local source code -
+this might take a few minutes. If you want to force a rebuild after you've made
+some changes you can append the `--build` flag after `up`, i.e.
+
+```
+docker-compose up --build
+```
+
+Although this isn't particularly quick. By default the compose setup will pull
+the repo from the source defined in `config/oxford.yaml`, though you can change
+this from inside the `dev-compose.yml`, either by changing the `YAML_TEMPLATE`
+value (under `services.gutenberg.build.args`) to a different config yaml file or
+changing `MATERIAL_METHOD` to `"copy"` to copy a locally checked out folder into
+the container at build-time instead of pulling fresh - for more details on how
+to pull material to edit locally follow the below section.

docs/development/index.md

@@ -7,6 +7,8 @@ permalink: /development/
 
 This will guide you through the process of setting up a development environment.
 
+If you instead wish to build the application via docker then please refer to the [docker guide]({{"/development/docker" | relative_url }})
+
 To see how to to build the docs, please see the [documentation guide]({{"/development/docs" | relative_url }}).
 
 <!-- prettier-ignore -->

entrypoint.sh

@@ -1,3 +1,7 @@
 #!/bin/bash
+
+# Run migrations on the database
+npx prisma migrate deploy
+
 yarn cron &
 yarn start

nginx/default.conf.template

@@ -0,0 +1,14 @@
+server {
+    listen       ${NGINX_PORT};
+    server_name  ${NGINX_HOST};
+
+    #access_log  /var/log/nginx/host.access.log  main;
+
+    location / {
+        proxy_pass ${NGINX_PROXY_PASS};
+    }
+
+    #error_page  404              /404.html;
+    }
+
+

run-compose.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+export CACHE_BUST=$(date +%s)
+docker-compose build --build-arg CACHE_BUST=$CACHE_BUST