Initial public version of the Mongo-Initializr (#1)

Added the `Mongo-Initializr` scripts and some helper scripts for building the Docker images.

README.md

@@ -1 +1,133 @@
 # Mongo-Initializr
+
+Docker image for building MongoDB databases using the `Mongo-Initializr` scripts.
+
+## Docker image
+
+By default the database is initialized on startup of the container using the Docker [ENTRYPOINT](docker/docker-entrypoint.sh) which is executing the [Mongo-Initializr](docker/mongo-initializr.sh) script. The behavior of this script can be customized based on the `MI_*`-variables.
+
+### Variables
+
+* `MONGO_INITDB_ROOT_USERNAME`, `MONGO_INITDB_ROOT_PASSWORD`: Username and password used for initializing and building the database (both default set to aerius).
+* `MONGO_INITDB_DATABASE`: The name of the database to initialize (default set to mongo).
+* `MI_INPUT_FILE`: The file which contains a list of dbdata-files and the corresponding database collection. If set the build and sync are executed.
+* `MI_NEXUS_BASE_URL`, `MI_NEXUS_REPOSITORY`: The base-url and nexus repository where the dbdata-files are located.
+* `MI_DATABASE_VERSION`: The version of the database, which will be added to the database constants. 
+* `MI_RUN_SCRIPT_FOLDER`: The folder there the `_run.js` file is located. If *not* set, the runner wil *not* be executed.
+* `MI_DUMP_DATABASE`: Set to `true` if a (gzipped) binary export of the database needs to be created.
+* `MI_INITIALIZE_ON_BUILD`: Set to `true` if the database is going to be initialized during the Docker image build. You need to add `RUN /usr/local/bin/docker-entrypoint.sh mongod` to you `Dockerfile` in order to start the initialization during the Docker build. 
+* `MI_SKIP_DBDATA_SYNC`: Set to `true` if the dbdata sync should be skipped.
+* `MI_SKIP_UNSET_ENVS`: Set to `true` if all the `MI_*` environment variables should be kept after the `docker-entrypoint.sh` entrypoint script is runned.
+* `MI_BIN_FOLDER_CLEANUP`: Set to `true` if the bin-folder should be removed.
+* `MI_SOURCE_FOLDER_CLEANUP`: Set to `true` if the source-folder should be removed.
+* `MI_DBDATA_FOLDER_CLEANUP`: Set to `true` if the dbdata-folder should be removed.
+* `HTTPS_DATA_USERNAME`, `HTTPS_DATA_PASSWORD`: The username and password of the nexus repository used for syncing the dbdata-files.
+
+### (Docker) Folders
+* `MI_BIN_FOLDER` (`/mi/bin`): Folder where you can find the `Mongo-Initializr` scripts.
+* `MI_SOURCE_FOLDER` (`/mi/source`): Folder for the database source code.
+* `MI_DBDATA_FOLDER` (`/mi/dbdata`): Folder for all dbdata files.
+* `MI_DUMP_FOLDER` (`/mi/dump`): Folder for all database dumps. All dumps in this folder will be restored by the [Mongo-Initializr](docker/mongo-initializr.sh) script.
+
+### Examples of how to use the aerius-mongo-initializr image
+
+#### Initialize the example-project database using a Docker run.
+``` bash
+docker run \
+	--name example-project \
+	--volume /projects/example-project/git/example-project/source/:/mi/source \
+	--env MI_INPUT_FILE="/mi/source/example-project/src/data/initdb.json" \
+	--env MI_RUN_SCRIPT_FOLDER="/mi/source/example-project/src/main" \
+	--env MONGO_INITDB_DATABASE=example \
+	--env MI_DATABASE_VERSION=0.0.1 \
+	--env MI_NEXUS_BASE_URL=https://nexus.example-project.com \
+	--env MI_NEXUS_REPOSITORY=dbdata \
+	--env HTTPS_DATA_USERNAME=${HTTPS_DATA_USERNAME} \
+	--env HTTPS_DATA_PASSWORD=${HTTPS_DATA_PASSWORD} \
+	aerius-mongo-initializr:0.1-SNAPSHOT-7.0.6
+```
+
+<br>
+
+#### Initialize the example-project database with local dbdata files using a Docker run.
+``` bash
+docker run \
+	--name example-project \
+	--volume /projects/example-project/git/example-project/source/:/mi/source \
+	--volume /projects/example-project/dbdata/:/mi/dbdata \
+	--env MI_INPUT_FILE="/mi/source/example-project/src/data/initdb.json" \
+	--env MI_RUN_SCRIPT_FOLDER="/mi/source/example-project/src/main" \
+	--env MONGO_INITDB_DATABASE=example \
+	--env MI_DATABASE_VERSION=0.0.1 \
+	--env MI_SKIP_DBDATA_SYNC=true \
+	aerius-mongo-initializr:0.1-SNAPSHOT-7.0.6
+```
+
+<br>
+
+#### Dockerfile containing all the information to be able to initialize the example-project database on startup of the container.
+```bash
+#syntax = docker/dockerfile:1
+FROM aerius-mongo-initializr:0.1-SNAPSHOT-7.0.6 
+
+ENV MONGO_INITDB_DATABASE=example \
+    MONGO_INITDB_ROOT_USERNAME=example \
+    MONGO_INITDB_ROOT_PASSWORD=passwd \
+    MI_DATABASE_VERSION=0.01 \
+    MI_INPUT_FILE="/mi/source/example-project/src/data/initdb.json" \
+    MI_RUN_SCRIPT_FOLDER="/mi/source/example-project/src/main" \
+    MI_INITIALIZE_ON_BUILD=false \
+    MI_NEXUS_BASE_URL="https://nexus.example-project.com" \
+    MI_NEXUS_REPOSITORY=dbdata
+
+# Copy all necessary scripts
+COPY ./source /mi/source
+```
+
+```bash
+docker build --tag example-project .
+```
+
+```bash
+docker run \
+    --name example-project \
+    --volume ./data:/data/db \
+    --publish 27017:27017 \
+    --env HTTPS_DATA_USERNAME=user \
+    --env HTTPS_DATA_PASSWORD=passwd \
+    example-project:latest
+```
+
+Note: the Mongo docker-entrypoint will not initialize the database again if there is a known path in the dbPath folder. See the  Mongo `docker-entrypoint`.
+
+<br>
+
+#### Dockerfile to initialize the example-project database during the Docker build.
+```bash
+#syntax = docker/dockerfile:1
+FROM aerius-mongo-initializr:0.1-SNAPSHOT-7.0.6 
+
+ARG MONGO_INITDB_DATABASE=example-project \
+	MONGO_INITDB_ROOT_USERNAME=example \
+	MONGO_INITDB_ROOT_PASSWORD=passwd \
+	MI_DATABASE_VERSION=0.01 \
+	MI_INPUT_FILE="/mi/source/example-project/src/data/initdb.json" \
+	MI_RUN_SCRIPT_FOLDER="/mi/source/example-project/src/main" \
+	MI_INITIALIZE_ON_BUILD=true \
+	MI_NEXUS_BASE_URL="https://nexus.example-project.com" \
+	MI_NEXUS_REPOSITORY=dbdata \
+	HTTPS_DATA_USERNAME \
+	HTTPS_DATA_PASSWORD
+
+# Copy all necessary scripts
+COPY ./source /mi/source
+
+# Run the docker-entrypoint in order to initialize the database
+RUN /usr/local/bin/docker-entrypoint.sh mongod
+```
+
+## Image build
+
+There are two scripts for building the Docker images.
+* [`update.sh`](update.sh) - Creates the Dockerfile files for the specified Mongo versions.
+* [`build_images.sh`](build_images.sh) - Builds all generated Dockerfile files.

VERSION

@@ -0,0 +1 @@
+0.1-SNAPSHOT

bin/include.functions.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+
+# Exit on error
+set -e
+
+# Echo output colors
+GREEN='\033[0;32m'
+COLOR_OFF='\033[0m'
+
+# Method to displaying messages
+_log() {
+    echo -e "${GREEN}[$(date '+%Y-%m-%d %H:%M:%S')] $@${COLOR_OFF}"
+}
+
+# Helper function to call mongoimport
+_mongoimport() { 
+    mongoimport --uri "mongodb://${MONGO_HOST}:${MONGO_PORT}/${DATABASE_NAME}" --username "${MONGO_USER}" --password "${MONGO_PASS}" --authenticationDatabase admin "$@"
+}
+
+# Helper function to call mongosh
+_mongosh() {
+    mongosh "${MONGO_HOST}:${MONGO_PORT}/${DATABASE_NAME}" --username "${MONGO_USER}" --password "${MONGO_PASS}" --authenticationDatabase admin "$@"
+}

bin/mi-dbdata-import.sh

@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+
+# Exit on error
+set -e
+
+# Default values
+DEFAULT_INPUT_FILE="/initdb.json"
+DEFAULT_DATA_FOLDER="/dbdata"
+DEFAULT_MONGO_HOST="localhost"
+DEFAULT_MONGO_PORT="27017"
+
+
+INPUT_FILE="${DEFAULT_INPUT_FILE}"
+DATA_FOLDER="${DEFAULT_DATA_FOLDER}"
+MONGO_HOST="${DEFAULT_MONGO_HOST}"
+MONGO_PORT="${DEFAULT_MONGO_PORT}"
+MONGO_USER=""
+MONGO_PASS=""
+DATABASE_NAME=""
+DATABASE_VERSION=""
+
+
+SCRIPT_PATH=$(readlink -f "${0}")
+SCRIPT_DIR=$(dirname "${SCRIPT_PATH}")
+
+
+# Include functions
+source "${SCRIPT_DIR}/include.functions.sh"
+
+
+# Function to display the banner
+display_banner() {
+    _log "------------------------------------------"
+    _log "- Mongo-Initializr - DbData Import        "
+    _log "------------------------------------------"
+    _log
+}
+
+# Function to display script usage
+display_help() {
+    _log
+    _log "Usage: $(basename -- $0) [OPTIONS]"
+    _log "Options:"
+    _log "  -i, --input-file [arg]        Specify input file (default: ${DEFAULT_INPUT_FILE})"
+    _log "  -d, --data-folder [arg]       Specify data folder (default: ${DEFAULT_DATA_FOLDER})"
+    _log
+    _log "      --mongo-hostname [arg]    Specify MongoDB hostname (default: ${DEFAULT_MONGO_HOST})"
+    _log "      --mongo-port [arg]        Specify MongoDB port (default: ${DEFAULT_MONGO_PORT})"
+    _log "      --mongo-username [arg]    Specify MongoDB username"
+    _log "      --mongo-password [arg]    Specify MongoDB password"
+    _log
+    _log "      --database-name [arg]     Specify database name"
+    _log "      --database-version [arg]  Specify database version"
+    _log
+    _log "  -h, --help                    Display this help message"
+    exit 1
+}
+
+# Function to parse command line arguments
+parse_arguments() {
+    while [[ $# -gt 0 ]]; do
+        case "${1}" in
+            -i|--input-file)
+                INPUT_FILE="$2"
+                shift 2
+                ;;
+            -d|--data-folder)
+                DATA_FOLDER="$2"
+                shift 2
+                ;;
+            --mongo-host)
+                MONGO_HOST="$2"
+                shift 2
+                ;;
+            --mongo-port)
+                MONGO_PORT="$2"
+                shift 2
+                ;;
+            --mongo-username)
+                MONGO_USER="$2"
+                MONGO_NOAUTH=false
+                shift 2
+                ;;
+            --mongo-password)
+                MONGO_PASS="$2"
+                shift 2
+                ;;
+            --database-name)
+                DATABASE_NAME="$2"
+                shift 2
+                ;;
+            --database-version)
+                DATABASE_VERSION="$2"
+                shift 2
+                ;;
+            -h|--help)
+                display_help
+                ;;
+            *)
+                _log "Unknown option: ${1}"
+                display_help
+                ;;
+        esac
+    done
+}
+
+# Function for validating the command line arguments
+validate_arguments() {
+    # Check if all mongo settings are set
+    if [[ -z "${MONGO_USER}" ]] || [[ -z "${MONGO_PASS}" ]]; then
+        _log "Error: Mongo credentials are required."
+        display_help
+    fi
+
+    # Check if database-name / -version are set
+    if [[ -z "${DATABASE_NAME}" ]] || [[ -z "${DATABASE_VERSION}" ]]; then
+        _log "Error: Database name and version are required."
+        display_help
+    fi
+
+    # Check if initdb input file exists
+    if [[ ! -f "${INPUT_FILE}" ]]; then
+        _log "Error: The input file '${INPUT_FILE}' does not exist."
+        display_help
+    fi
+}
+
+# Function to clean a MongoDB database
+clean_database() {
+    _mongosh --eval "db.getCollectionNames().forEach(function(n){db[n].drop({})});"
+}
+
+# Function to add handle all entries in the input file
+handle_input_file() {
+    jq -r '.[] | "\(.collection) \(.path)"' "${INPUT_FILE}" | while read collection path; do
+        add_json_to_collection "${collection}" "${path}"
+    done
+}
+
+# Function to add JSON data to a collection
+add_json_to_collection() {
+    local collection=${1}
+    local filename="${DATA_FOLDER}/${2}"
+
+    _log "Processing file '${2}'."
+
+    # Check if the JSON filename exists
+    if [[ -f "${filename}" ]]; then
+        # Add JSON data to the collection. Use mongoimport to create the collection and add data
+        _mongoimport --collection ${collection} --file $filename --jsonArray --upsert --upsertFields "_id"
+
+        _log "'${filename}' is added to '${collection}'"
+    else
+        _log "Error: JSON file '$filename' not found."
+        exit 1
+    fi
+}
+
+# Function to add the constants to the database
+add_constants() {
+    constants=$(jq -n -c --arg name "${DATABASE_NAME}" --arg version "${DATABASE_VERSION}" \
+    '{
+      _id: 1,
+      database: $ARGS.named
+    }')
+    _mongosh --eval "db.constants.insertOne(${constants})"
+}
+
+
+# Main script logic
+display_banner
+
+parse_arguments "$@"
+
+validate_arguments
+
+_log "Clean database '${DATABASE_NAME}'"
+clean_database
+
+_log "Handle inputfile '${INPUT_FILE}'"
+handle_input_file
+
+_log "Add constants to database '${DATABASE_NAME}'"
+add_constants
+
+_log "Done!"