Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify Compose environments for testing AM packages #355

Merged
merged 8 commits into from
Jun 20, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 49 additions & 28 deletions .github/workflows/test-am-debs.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,34 @@
name: Archivematica DEB Packages Test
on:
workflow_dispatch:
inputs:
build_packages:
description: "Build local packages"
required: true
default: true
type: "boolean"
pull_request:
paths:
- "debs/jammy/archivematica/**"
- "debs/jammy/archivematica-storage-service/**"
- "debs/jammy-testing/**"
- "tests/archivematica/**"
- "!tests/archivematica/README.md"
- "!tests/archivematica/EL9/**"
push:
branches:
- "stable/**"
- "qa/**"
paths:
- "debs/jammy/archivematica/**"
- "debs/jammy/archivematica-storage-service/**"
- "debs/jammy-testing/**"
- "tests/archivematica/**"
- "!tests/archivematica/README.md"
- "!tests/archivematica/EL9/**"
jobs:
build-am-deb:
name: Build Archivematica Debian packages
build-am-packages:
name: Build Archivematica packages
runs-on: ubuntu-latest
if: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
steps:
- name: Check out code
uses: actions/checkout@v4
@@ -27,12 +38,13 @@ jobs:
- name: Save artifacts
uses: actions/upload-artifact@v4
with:
name: archivematica-deb
name: archivematica-packages
path: |
${{ github.workspace }}/debs/jammy/archivematica/repo
build-ss-deb:
name: Build Storage Service Debian package
build-ss-packages:
name: Build Storage Service packages
runs-on: ubuntu-latest
if: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
steps:
- name: Check out code
uses: actions/checkout@v4
@@ -42,28 +54,29 @@ jobs:
- name: Save artifacts
uses: actions/upload-artifact@v4
with:
name: archivematica-storage-service-deb
name: archivematica-storage-service-packages
path: |
${{ github.workspace }}/debs/jammy/archivematica-storage-service/repo
create-deb-repo:
name: Create Debian repository
create-package-repo:
name: Create package repository
runs-on: ubuntu-latest
if: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
needs:
- build-am-deb
- build-ss-deb
- build-am-packages
- build-ss-packages
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Restore Archivematica packages
uses: actions/download-artifact@v4
with:
name: archivematica-deb
name: archivematica-packages
path: |
${{ github.workspace }}/debs/jammy/archivematica/repo
- name: Restore Storage Service package
- name: Restore Storage Service packages
uses: actions/download-artifact@v4
with:
name: archivematica-storage-service-deb
name: archivematica-storage-service-packages
path: |
${{ github.workspace }}/debs/jammy/archivematica-storage-service/repo
- name: Create repository
@@ -72,20 +85,28 @@ jobs:
- name: Save package repository
uses: actions/upload-artifact@v4
with:
name: repository-deb
name: package-repository
path: |
${{ github.workspace }}/debs/jammy/_deb_repository
test-deb:
name: Test Debian packages
needs: create-deb-repo
test:
name: Test packages
needs: create-package-repo
runs-on: ubuntu-latest
if: "${{ always() }}"
env:
build_packages: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
steps:
- name: Check if the repository creation job failed
if: "${{ needs.create-package-repo.result != 'skipped' && ! env.build_packages }}"
run: |
exit 1
- name: Check out code
uses: actions/checkout@v4
- name: Restore package repository
if: needs.create-package-repo.result == 'success'
uses: actions/download-artifact@v4
with:
name: repository-deb
name: package-repository
path: ${{ github.workspace }}/debs/jammy/_deb_repository
- name: "Upgrade crun (supports Ubuntu's systemd in the Dockerfile)"
run: |
@@ -98,35 +119,35 @@ jobs:
python-version: "3.10"
cache: "pip"
cache-dependency-path: |
debs/jammy-testing/requirements.txt
tests/archivematica/requirements.txt
- name: "Cache the virtual environment"
id: "venv-cache"
uses: "actions/cache@v4"
with:
path: |
debs/jammy-testing/.venv/
key: "os-${{ runner.os }}-python_version-${{ env.python_version }}-hash-${{ hashFiles('debs/jammy-testing/requirements.txt') }}"
tests/archivematica/.venv/
key: "os-${{ runner.os }}-python_version-${{ env.python_version }}-hash-${{ hashFiles('tests/archivematica/requirements.txt') }}"
- name: "Set up the virtual environment"
if: "steps.venv-cache.outputs.cache-hit == false"
working-directory: "${{ github.workspace }}/debs/jammy-testing"
working-directory: "${{ github.workspace }}/tests/archivematica"
run: |
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
- name: "Add virtual environment to PATH"
working-directory: "${{ github.workspace }}/debs/jammy-testing"
working-directory: "${{ github.workspace }}/tests/archivematica"
run:
echo "$PWD/.venv/bin" >> $GITHUB_PATH
- name: "Start the Compose environment"
working-directory: "${{ github.workspace }}/debs/jammy-testing"
working-directory: "${{ github.workspace }}/tests/archivematica"
env:
DOCKER_IMAGE_NAME: "ubuntu"
DOCKER_IMAGE_TAG: "22.04"
run: |
podman-compose up --detach
- name: Install packages
working-directory: ${{ github.workspace }}/debs/jammy-testing
working-directory: ${{ github.workspace }}/tests/archivematica
run: |
podman-compose exec --env LOCAL_REPOSITORY="yes" --user ubuntu archivematica /am-packbuild/debs/jammy-testing/install.sh
podman-compose exec --env LOCAL_REPOSITORY="${{ env.build_packages }}" --user ubuntu archivematica /am-packbuild/tests/archivematica/jammy/install.sh
- name: Test AM API - Get processing configurations
run: |
test $( \
77 changes: 49 additions & 28 deletions .github/workflows/test-am-rpms.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,34 @@
name: Archivematica RPM Packages Test
on:
workflow_dispatch:
inputs:
build_packages:
description: "Build local packages"
required: true
default: true
type: "boolean"
pull_request:
paths:
- "rpms/EL9/archivematica/**"
- "rpms/EL9/archivematica-storage-service/**"
- "rpms/EL9-testing/**"
- "tests/archivematica/**"
- "!tests/archivematica/README.md"
- "!tests/archivematica/jammy/**"
push:
branches:
- "stable/**"
- "qa/**"
paths:
- "rpms/EL9/archivematica/**"
- "rpms/EL9/archivematica-storage-service/**"
- "rpms/EL9-testing/**"
- "tests/archivematica/**"
- "!tests/archivematica/README.md"
- "!tests/archivematica/jammy/**"
jobs:
build-am-rpm:
name: Build Archivematica RPM packages
build-am-packages:
name: Build Archivematica packages
runs-on: ubuntu-latest
if: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
steps:
- name: Check out code
uses: actions/checkout@v4
@@ -27,12 +38,13 @@ jobs:
- name: Save artifacts
uses: actions/upload-artifact@v4
with:
name: archivematica-rpm
name: archivematica-packages
path: |
${{ github.workspace }}/rpms/EL9/archivematica/*.rpm
build-ss-rpm:
name: Build Storage Service RPM package
build-ss-packages:
name: Build Storage Service packages
runs-on: ubuntu-latest
if: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
steps:
- name: Check out code
uses: actions/checkout@v4
@@ -42,28 +54,29 @@ jobs:
- name: Save artifacts
uses: actions/upload-artifact@v4
with:
name: archivematica-storage-service-rpm
name: archivematica-storage-service-packages
path: |
${{ github.workspace }}/rpms/EL9/archivematica-storage-service/*.rpm
create-rpm-repo:
name: Create RPM repository
create-package-repo:
name: Create package repository
runs-on: ubuntu-latest
if: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
needs:
- build-am-rpm
- build-ss-rpm
- build-am-packages
- build-ss-packages
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Restore Archivematica packages
uses: actions/download-artifact@v4
with:
name: archivematica-rpm
name: archivematica-packages
path: |
${{ github.workspace }}/rpms/EL9/archivematica/
- name: Restore Storage Service package
- name: Restore Storage Service packages
uses: actions/download-artifact@v4
with:
name: archivematica-storage-service-rpm
name: archivematica-storage-service-packages
path: |
${{ github.workspace }}/rpms/EL9/archivematica-storage-service
- name: Create repository
@@ -72,20 +85,28 @@ jobs:
- name: Save package repository
uses: actions/upload-artifact@v4
with:
name: repository-rpm
name: package-repository
path: |
${{ github.workspace }}/rpms/EL9/_yum_repository
test-rpm:
name: Test RPM packages
needs: create-rpm-repo
test:
name: Test packages
needs: create-package-repo
runs-on: ubuntu-latest
if: "${{ always() }}"
env:
build_packages: "${{ github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.build_packages) }}"
steps:
- name: Check if the repository creation job failed
if: "${{ needs.create-package-repo.result != 'skipped' && ! env.build_packages }}"
run: |
exit 1
- name: Check out code
uses: actions/checkout@v4
- name: Restore package repository
if: needs.create-package-repo.result == 'success'
uses: actions/download-artifact@v4
with:
name: repository-rpm
name: package-repository
path: ${{ github.workspace }}/rpms/EL9/_yum_repository
- name: "Upgrade crun (supports Ubuntu's systemd in the Dockerfile)"
run: |
@@ -98,35 +119,35 @@ jobs:
python-version: "3.10"
cache: "pip"
cache-dependency-path: |
rpms/EL9-testing/requirements.txt
tests/archivematica/requirements.txt
- name: "Cache the virtual environment"
id: "venv-cache"
uses: "actions/cache@v4"
with:
path: |
rpms/EL9-testing/.venv/
key: "os-${{ runner.os }}-python_version-${{ env.python_version }}-hash-${{ hashFiles('rpms/EL9-testing/requirements.txt') }}"
tests/archivematica/.venv/
key: "os-${{ runner.os }}-python_version-${{ env.python_version }}-hash-${{ hashFiles('tests/archivematica/requirements.txt') }}"
- name: "Set up the virtual environment"
if: "steps.venv-cache.outputs.cache-hit == false"
working-directory: "${{ github.workspace }}/rpms/EL9-testing"
working-directory: "${{ github.workspace }}/tests/archivematica"
run: |
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
- name: "Add virtual environment to PATH"
working-directory: "${{ github.workspace }}/rpms/EL9-testing"
working-directory: "${{ github.workspace }}/tests/archivematica"
run:
echo "$PWD/.venv/bin" >> $GITHUB_PATH
- name: "Start the Compose environment"
working-directory: "${{ github.workspace }}/rpms/EL9-testing"
working-directory: "${{ github.workspace }}/tests/archivematica"
env:
DOCKER_IMAGE_NAME: "rockylinux"
DOCKER_IMAGE_TAG: "9"
run: |
podman-compose up --detach
- name: Install packages
working-directory: ${{ github.workspace }}/rpms/EL9-testing
working-directory: ${{ github.workspace }}/tests/archivematica
run: |
podman-compose exec --env LOCAL_REPOSITORY="yes" --user ubuntu archivematica /am-packbuild/rpms/EL9-testing/install.sh
podman-compose exec --env LOCAL_REPOSITORY="${{ env.build_packages }}" --user ubuntu archivematica /am-packbuild/tests/archivematica/EL9/install.sh
- name: Test AM API - Get processing configurations
run: |
test $( \
8 changes: 8 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -10,3 +10,11 @@ repos:
rev: v0.39.0
hooks:
- id: markdownlint
- repo: https://github.com/thlorenz/doctoc
rev: v2.2.0
hooks:
- id: doctoc
files: |
(?x)^(
tests/archivematica/README\.md
)
6 changes: 1 addition & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -66,8 +66,4 @@ managing Ubuntu repositories using reprepro can be found

## Test package

Rocky Linux 9 packages: see the [rpms/EL9-testing](rpms/EL9-testing) directory
for more details.

Ubuntu 22.04 Jammy packages: see the [debs/jammy-testing](debs/jammy-testing)
directory for more details.
See the [tests/archivematica](tests/archivematica) directory for more details.
3 changes: 0 additions & 3 deletions debs/jammy-testing/.gitignore

This file was deleted.

59 changes: 0 additions & 59 deletions debs/jammy-testing/README.md

This file was deleted.

3 changes: 0 additions & 3 deletions rpms/EL9-testing/.gitignore

This file was deleted.

32 changes: 0 additions & 32 deletions rpms/EL9-testing/Dockerfile

This file was deleted.

59 changes: 0 additions & 59 deletions rpms/EL9-testing/README.md

This file was deleted.

22 changes: 0 additions & 22 deletions rpms/EL9-testing/compose.yaml

This file was deleted.

1 change: 0 additions & 1 deletion rpms/EL9-testing/requirements.txt

This file was deleted.

1 change: 1 addition & 0 deletions tests/archivematica/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.venv
17 changes: 15 additions & 2 deletions debs/jammy-testing/Dockerfile → tests/archivematica/Dockerfile
Original file line number Diff line number Diff line change
@@ -24,14 +24,27 @@ ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
ENV TZ=UTC

RUN ln -snf /usr/share/zoneinfo/${TZ} /etc/localtime && echo ${TZ} > /etc/timezone

FROM rockylinux:9 AS install_rockylinux_9

RUN set -ex \
&& dnf -y update \
&& dnf -y install coreutils --allowerasing \
&& dnf -y install \
glibc-langpack-en \
openssh-server \
python-unversioned-command \
selinux-policy \
sudo \
&& dnf clean all

FROM install_${DOCKER_IMAGE_NAME}_${DOCKER_IMAGE_TAG} as server

RUN useradd --home-dir /home/ubuntu --system ubuntu

RUN mkdir -p /etc/sudoers.d/ && echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers.d/ubuntu

RUN ln -snf /usr/share/zoneinfo/${TZ} /etc/localtime && echo ${TZ} > /etc/timezone

COPY --chown=ubuntu:ubuntu . /src

EXPOSE 22
20 changes: 12 additions & 8 deletions rpms/EL9-testing/install.sh → tests/archivematica/EL9/install.sh
Original file line number Diff line number Diff line change
@@ -48,7 +48,7 @@ else
name=archivematica
baseurl=https://packages.archivematica.org/1.16.x/rocky9
gpgcheck=1
gpgkey=https://packages.archivematica.org/1.16.x/key.asc
gpgkey=https://packages.archivematica.org/GPG-KEY-archivematica-sha512
enabled=1
EOF'
fi
@@ -72,8 +72,8 @@ sudo -u root yum-config-manager --enable crb
#

if [ $(getenforce) != "Disabled" ]; then
sudo semanage port -m -t http_port_t -p tcp 81
sudo semanage port -a -t http_port_t -p tcp 8001
sudo semanage port -m -t http_port_t -p tcp 80
sudo semanage port -a -t http_port_t -p tcp 8000
sudo setsebool -P httpd_can_network_connect_db=1
sudo setsebool -P httpd_can_network_connect=1
sudo setsebool -P httpd_setrlimit 1
@@ -156,8 +156,12 @@ sudo -u root systemctl enable archivematica-mcp-server
sudo -u root systemctl start archivematica-mcp-server
sudo -u root systemctl enable archivematica-dashboard
sudo -u root systemctl start archivematica-dashboard
sudo -u root systemctl reload nginx

# Update nginx site configurations to match standard Archivematica ports
sudo -u root sed -i -e 's/80;/90;/g' /etc/nginx/nginx.conf
sudo -u root sed -i -e 's/listen 8001/listen 8000/g' /etc/nginx/conf.d/archivematica-storage-service.conf
sudo -u root sed -i -e 's/listen 81/listen 80/g' /etc/nginx/conf.d/archivematica-dashboard.conf
sudo -u root systemctl reload nginx

#
# MCPClient
@@ -185,8 +189,8 @@ sudo -u root systemctl start clamd@scan
systemctl -q is-enabled firewalld || rc1=$?
systemctl -q is-active firewalld || rc2=$?
if [ ${rc1} -eq 0 ] && [ ${rc2} -eq 0 ]; then
sudo firewall-cmd --zone=public --add-port=81/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8001/tcp --permanent
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent
sudo systemctl restart firewalld || true
fi

@@ -217,8 +221,8 @@ sudo -u archivematica bash -c " \
--org-name="test" \
--org-id="test" \
--api-key="apikey" \
--ss-url="http://localhost:8001" \
--ss-url="http://localhost:8000" \
--ss-user="admin" \
--ss-api-key="apikey" \
--site-url="http://localhost:81"
--site-url="http://localhost"
";
126 changes: 126 additions & 0 deletions tests/archivematica/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Instructions

## Table of contents

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [Software requirements](#software-requirements)
- [Set up a Python virtual environment](#set-up-a-python-virtual-environment)
- [Set up EL9 packages](#set-up-el9-packages)
- [Install EL9 packages from archivematica.org](#install-el9-packages-from-archivematicaorg)
- [Install EL9 packages from a local repository](#install-el9-packages-from-a-local-repository)
- [Set up Ubuntu 22.04 Jammy packages](#set-up-ubuntu-2204-jammy-packages)
- [Install jammy packages from archivematica.org](#install-jammy-packages-from-archivematicaorg)
- [Install jammy packages from a local repository](#install-jammy-packages-from-a-local-repository)
- [Test the Archivematica installation](#test-the-archivematica-installation)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Software requirements

- Podman
- crun >= 1.14.4
- Python 3

This environment has been tested with Podman 3.4.4 and podman-compose 1.1.0 and
it is based on the official Docker images:

- rockylinux:9
- ubuntu:22.04

## Set up a Python virtual environment

Create a virtual environment and activate it:

```shell
python3 -m venv .venv
source .venv/bin/activate
```

Install the Python requirements:

```shell
python3 -m pip install -r requirements.txt
```

## Set up EL9 packages

Start the Compose environment:

```shell
export DOCKER_IMAGE_NAME=rockylinux
export DOCKER_IMAGE_TAG=9
podman-compose up --detach
```

### Install EL9 packages from archivematica.org

Install `EL9` packages from the published Archivematica repository:

```shell
podman-compose exec --user ubuntu archivematica /am-packbuild/tests/archivematica/EL9/install.sh
```

### Install EL9 packages from a local repository

Alternatively, test using a local repository built from the `/rpms/EL9`
directory of this repository.

Create the local repository:

```shell
make -C ../../rpms/EL9/archivematica
make -C ../../rpms/EL9/archivematica-storage-service
make -C ../../rpms/EL9 createrepo
```

Install `EL9` packages using the local repository:

```shell
podman-compose exec --env LOCAL_REPOSITORY="yes" --user ubuntu archivematica /am-packbuild/tests/archivematica/EL9/install.sh
```

## Set up Ubuntu 22.04 Jammy packages

Start the Compose environment:

```shell
export DOCKER_IMAGE_NAME=ubuntu
export DOCKER_IMAGE_TAG=22.04
podman-compose up --detach
```

### Install jammy packages from archivematica.org

Install `jammy` packages from the published Archivematica repository:

```shell
podman-compose exec --user ubuntu archivematica /am-packbuild/tests/archivematica/jammy/install.sh
```

### Install jammy packages from a local repository

Alternatively, test using a local repository built from the `/debs/jammy`
directory of this repository.

Create the local repository:

```shell
make -C ../../debs/jammy/archivematica
make -C ../../debs/jammy/archivematica-storage-service
make -C ../../debs/jammy createrepo
```

Install `jammy` packages using the local repository:

```shell
podman-compose exec --env LOCAL_REPOSITORY="yes" --user ubuntu archivematica /am-packbuild/tests/archivematica/jammy/install.sh
```

## Test the Archivematica installation

Once installation finishes you should be able to access to the web interfaces:

- Access to Dashboard: <http://localhost:8000>
- Access to Storage Service: <http://localhost:8001>
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: archivematica-jammy-testing
name: archivematica-package-testing

services:

@@ -9,6 +9,10 @@ services:
DOCKER_IMAGE_NAME: "${DOCKER_IMAGE_NAME:-ubuntu}"
DOCKER_IMAGE_TAG: "${DOCKER_IMAGE_TAG:-22.04}"
TARGET: server
# The nginx configuration files included in the EL9 RPM specs reference
# this hostname in their proxy_pass.
extra_hosts:
- "localhost4:127.0.0.1"
ports:
- "2222:22"
- "8000:80"
File renamed without changes.
File renamed without changes.