Initial working version

This is a working prototype that can be used to create working images from valid config files. Fixes: - Specify the card/project configuration format & workflow #4 - Implement Basic program structure #7 - Implement image file manipulation #10 Starts without completing: - Implement Progress Reporting #8: text UI (cli) is done and usable. Machine-readable JSON file is not but can be implemented via the logger and StepMachine. - Implement Downloader #9: a basic requests-based downloader is present. Needs more work to support transfer issues and being interupted
offspot · Dec 2, 2022 · 1f477d2 · 1f477d2
1 parent 34fc45a
commit 1f477d2
Show file tree

Hide file tree

Showing 26 changed files with 2,422 additions and 93 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -0,0 +1,68 @@
+name: Build
+
+on:
+  push:
+    branches:
+      - main
+      - initial
+
+env:
+  SSH_KEY: /tmp/id_rsa
+
+jobs:
+  build-and-upload:
+    # building on ubuntu-20.04 so we'll link to glibc 2.31
+    # which is version in Debian bullseye as well while Ubuntu 22.04 has 2.35
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout
+        uses: actions/[email protected]
+      - name: Setup Python
+        uses: actions/[email protected]
+        with:
+          python-version: "3.10"
+          architecture: x64
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install --no-install-recommends -y python3-dev patchelf ccache build-essential
+      - name: Install python dependencies
+        run: |
+          pip install -U pip
+          pip install -U ordered-set zstandard nuitka
+          pip install -r requirements.txt
+      - name: Update environ (filename)
+        run: |
+          import os
+          ref = os.getenv("GITHUB_REF", "").split("/")[-1]
+          env = {"FILENAME": f"image-creator_{ref}"}
+          with open(os.getenv("GITHUB_ENV"), "a") as fh:
+            for name, value in env.items():
+              fh.write(f"{name}={value}\n")
+        shell: python
+      - name: Build image-creator
+        run: |
+          cd src
+          python -m nuitka \
+            --onefile \
+            --python-flag="no_site,no_warnings,no_asserts,no_docstrings" \
+            --include-package=image_creator \
+            --show-modules \
+            --warn-implicit-exceptions \
+            --warn-unusual-code \
+            --assume-yes-for-downloads \
+            --output-filename=$(dirname $PWD)/$FILENAME \
+            --remove-output \
+            --no-progressbar \
+            image_creator/
+      - name: Dump SSH Key to file
+        shell: bash
+        run: |
+          echo "${{secrets.ssh_key}}" > $SSH_KEY
+          chmod 600 $SSH_KEY
+      - name: Upload build to CI
+        run: scp -rp -P 30022 -i $SSH_KEY -o StrictHostKeyChecking=no $PWD/$FILENAME [email protected]:/data/tmp/ci/
+      - name: Set job summary
+        run: |
+          echo '### Artefacts' >> $GITHUB_STEP_SUMMARY
+          echo '- [${{ env.FILENAME }}](http://tmp.kiwix.org/ci//${{ env.FILENAME }})' >> $GITHUB_STEP_SUMMARY
diff --git a/.github/workflows/qa.yml b/.github/workflows/qa.yml
@@ -1,21 +1,17 @@
 name: QA
 
-on: [push, pull_request]
-
-env:
-  # black default
-  MAX_LINE_LENGTH: 88
+on: [push]
 
 jobs:
   check-qa:
     runs-on: ubuntu-22.04
     steps:
       - name: Checkout
-        uses: actions/checkout@v2.5.0
+        uses: actions/checkout@v3.1.0
       - name: Setup Python
-        uses: actions/setup-python@v2.3.3
+        uses: actions/setup-python@v4.3.0
         with:
-          python-version: 3.10
+          python-version: "3.10"
           architecture: x64
       - name: Check black formatting
         run: |
@@ -25,9 +21,9 @@ jobs:
 
       - name: Check flake8 linting
         run: |
-          pip install flake8==6.0.0
+          pip install flake8==6.0.0 mccabe==0.7.0
           flake8 --version
-          flake8 . --count --max-line-length=$MAX_LINE_LENGTH --statistics
+          flake8 . --count --max-line-length=88 --max-complexity 12 --statistics
 
       - name: Check import order with isort
         run: |

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -6,17 +6,17 @@ repos:
     hooks:
     -   id: trailing-whitespace
     -   id: end-of-file-fixer
+-   repo: https://github.com/psf/black
+    rev: "22.10.0"
+    hooks:
+    -   id: black
 -   repo: https://github.com/pycqa/isort
     rev: "5.10.1"
     hooks:
     -   id: isort
         args: ["--profile", "black", "--filter-files"]
--   repo: https://github.com/psf/black
-    rev: "22.10.0"
-    hooks:
-    -   id: black
 -   repo: https://github.com/PyCQA/flake8
     rev: "6.0.0"
     hooks:
     -   id: flake8
-        args: ["--max-line-length", "88"]
+        args: ["--max-line-length", "88", "--max-complexity", "12"]
diff --git a/README.md b/README.md
@@ -1,80 +1,94 @@
 # image-creator
 
-Hotspot image creator to build OLIP or Kiwix Offspot off [`base-image`](https://github.com/offspot/base-image).
-
-## Scope
-
-- Validate inputs
-- Download base image
-- Resize image to match contents
-- Download contents into mounted `/data`
-- Post-process downloaded contents
-- Configure from inputs
-- Re-generate SSH server keys
-- *Pull* application images
-- Prepares JSON config
-
-## Inputs
-
-- Target system (OLIP or Offspot)
-- Image name
-- Hostname
-- domain name
-- SSID
-- WiFi AP password (if any)
-- WiFi Country code
-- WiFi channel
-- Timezone
-- SSH Public keys to add
-- VPN configuration (tinc)
-- Contents
-
-## App Containers
-
-- **OLIP**
- - API
- - Frontend
- - Stats
- - Controller
-- **Offspot**
- - Kiwix-serve
- - WikiFundi (en/fr/es)
- - Aflatoun (en/fr)
- - Surfer
-- IPFS daemon
-- Captive portal
-
-## data partition
-
-
-| /data subfolders | Usage |
-|---|---|
-| `offspot/zim` | Offspot Kiwix serve ZIM files|
-| `offspot/wikifundi` | Offspot WikiFundi data |
-| `offspot/files` | Offspot Surfer data |
-| `offspot/xxx` | Offspot data for other apps |
-| `olip` | OLIP data |
-
-
-## JSON Configurator
-
-JSON config file at `/boot/config.json` is read and parsed on startup by the boot-time config script.
-It looks for the following properties. Dotted ones means nested.
-
-Behavior is to adjust configuration only if the property is present. Script will remove property from JSON once applied.
-
-Configurator is also responsible for resizing `/data` partition to device size on first boot but this is not configurable via JSON.
-
-| Property| Type | Usage |
-|---|---|---|
-| `hostname` | `string` | Pi host name |
-| `domain` | `string` | FQDN to answer to on DNS |
-| `wifi.ssid` | `string` | WiFi SSID |
-| `wifi.password` | `string` | WiFi password (clear). If `null`, auth not required |
-| `wifi.country-code` | `string` | ISO-639-2 Country code for WiFI |
-| `wifi.channel` | `int` | 1-11 channel for WiFi |
-| `timezone` | `string` | Timezone to configure date with |
-| `ssh-keys` | `string[]` | List of public keys to add to user |
-| `tinc-vpn` | `string` | tinc-VPN configuration |
-| `env.all`  | `string[]` | List of `KEY=VALUE` environment variables to pass to **all applications** |
-| `env.xxx`  | `string[]` | List of `KEY=VALUE` environment variables to pass **containers matching _xxx_** |
+RaspberryPi image creator to build OLIP or Kiwix Hotspot off [`base-image`](https://github.com/offspot/base-image).
+
+[![CodeFactor](https://www.codefactor.io/repository/github/offspot/image-creator/badge)](https://www.codefactor.io/repository/github/offspot/image-creator)
+[![Build Status](https://github.com/offspot/image-creator/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/offspot/image-creator/actions/workflows/build.yml?query=branch%3Amain)
+[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
+
+
+## Usage
+
+`image-creator` is to be **ran as `root`**.
+
+```
+❯ image-creator --help
+usage: image-creator [-h] [--build-dir BUILD_DIR] [-C] [-K] [-X] [-T CONCURRENCY] [-D] [-V] CONFIG_SRC OUTPUT
+
+create an Offspot Image from a config
+
+positional arguments:
+  CONFIG_SRC            Offspot Config YAML file path or URL
+  OUTPUT                Where to write image to
+
+options:
+  -h, --help            show this help message and exit
+  --build-dir BUILD_DIR
+                        Directory to store temporary files in, like files that needs to be extracted. Defaults to some place within /tmp
+  -C, --check           Only check inputs, URLs and sizes. Don't download/create image.
+  -K, --keep            [DEBUG] Don't remove output image if creation failed
+  -X, --overwrite       Don't fail on existing output image: remove instead
+  -T CONCURRENCY, --concurrency CONCURRENCY
+                        Nb. of threads to start for parallel downloads (at most one per file). `0` (default) for auto-selection based on CPUs.
+                        `1` to disable concurrency.
+  -D, --debug
+  -V, --version         show program's version number and exit
+```
+
+
+## Configuration
+
+Image configuration is done through a YAML file which must match the following format. Only `base` is required.
+
+
+
+| Member           | Kind           | Function                                                                                                                                                           |
+|------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `base`           | `string`       | Version ([official releases](https://drive.offspot.it/base/)) or URL to a base-image file. Accepts `file://` URLs. Accepts lzma encoded images using `.xz` suffix  |
+| `output.size`    | `string`/`int` | Requested size of output image. Accepts `auto` for an power-of-2 sized that can fit the content (⚠️ TBI)                                                           |
+| `oci_images`     | `string[]`     | List of OCI Image names. More specific the better. Prefer ghcr.io if possible                                                                                      |
+| `files`          | `file[]`       | List of files to include on the data partition. See below. One of `url` or `content` must be present                                                               |
+| `files[].url`    | `string`       | URL to download file from                                                                                                                                          |
+| `files[].to`     | `string`       | [required] Path to store file at. Must be a descendent of `/data`                                                                                                  |
+| `files[].content`| `string`       | Text content of the file to write. Replaces `url` if present                                                                                                       |
+| `files[].via`    | `string`       | For `url`-based files, transformation to apply on downloaded file: `direct` (default): simple download, `bztar`, `gztar`, `tar`, `xztar`, `zip` to expand archives |
+| `files[].size`   | `string`/`int` | **Only for `untar`/`unzip`** should file be compressed. Specify expanded size. Assumes File-size (uncompressed) if not specified. ⚠️ Fails if lower than file size |
+| `write_config`   | `bool`         | Whether to write this file to `/data/conf/image.yaml`                                                                                                              |
+| `offspot`        | `dict`         | [runtime-config](https://github.com/offspot/runtime-config) configuration. Will be parsed and dumped to `/boot/offspot.yaml`                                       |
+
+### Sample
+
+```yaml
+---
+base: 1.0.0
+output:
+  size: 8G
+oci_images:
+- ghcr.io/offspot/kiwix-serve:dev
+files:
+- url: http://download.kiwix.org/zim/wikipedia_fr_test.zim
+  to: /data/contents/zims/wikipedia_fr_test.zim
+  via: direct
+- to: /data/conf/message.txt
+  content: |
+    hello world
+wite_config: true
+offspot:
+  timezone: Africa/Bamako
+  ap:
+    ssid: Kiwix Offspot
+    as-gateway: true
+    domain: demo
+    tld: offspot
+  containers:
+    services:
+      kiwix:
+        container_name: kiwix
+        image: ghcr.io/offspot/kiwix-serve:dev
+        command: /bin/sh -c "kiwix-serve /data/*.zim"
+        volumes:
+          - "/data/content/zims:/data:ro"
+        ports:
+          - "80:80"
+
+```
diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,6 @@
+requests==2.28.1
+PyYAML==6.0
+cli-ui==0.17.2
+humanfriendly==10.0
+progressbar2==4.2.0
+docker_export==0.4
diff --git a/src/image_creator/__init__.py b/src/image_creator/__init__.py
@@ -0,0 +1 @@
+__version__ = "1.0.0.dev0"
diff --git a/src/image_creator/__main__.py b/src/image_creator/__main__.py
@@ -0,0 +1,3 @@
+from image_creator.entrypoint import main
+
+main()
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		from image_creator.entrypoint import main

		main()