Skip to content

Commit

Permalink
Adding support for Debian 10 (Buster).
Browse files Browse the repository at this point in the history
Added support for creating images for Debian 10 (Buster).
Also changed build script from hooking into dockerhub environment to
stand-alone shell script. Unfortunately dockerhub has too short a runtime
limit (4 hours) and too slow VMs to be able to build these images.
  • Loading branch information
ArturKlauser committed Apr 7, 2019
1 parent 33c3082 commit f7dd025
Show file tree
Hide file tree
Showing 7 changed files with 431 additions and 165 deletions.
58 changes: 32 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ Makoto's excellent write-up of [RStudio installation on Raspberry
Pi](http://herb.h.kobe-u.ac.jp/raspiinfo/rstudio_en.html).

## Building the Debian Packages
Use Dockerfile.build_env to create the build environment in which the code for
RStudio Server and Desktop is going to be compiled. The build procedure has
been adapted to the Raspbian environment (Debian v9 Stretch) using the native
version of the Boost library and the native QT libraries. Once the build
environment is created it is used by Dockerfile.server_deb and
Dockerfile.desktop_deb to build the RStudio Server and Desktop Debian packages
respectively.

The build process is fairly lengthy. Expect several hours (4-8ish) both
Use Dockerfile.build-env to create the build environment in which the code for
RStudio Server and Desktop is going to be compiled. The build procedure has
been adapted to the Raspbian environment using the native version of the Boost
library and the native QT libraries. It has been tested on Debian 9 (Stretch)
and Debian 10 (Buster). Once the build environment is created, it is used by
Dockerfile.server-deb and Dockerfile.desktop-deb to build the RStudio Server and
Desktop Debian packages respectively.

The build process is fairly lengthy. Expect several hours (5-10ish) both
natively and in cross-build. You'll also need at least 1 GB of RAM on the build
machine, but more is better, which precludes native build on smaller Raspberry
Pis with less memory. In addition make sure to configure at least 1 GB of swap
Expand All @@ -49,29 +49,32 @@ space. Under Raspbian you can configure swap space like this:
An attempt was made to have the build run on dockerhub autobuild, but its VMs
take about 3 times longer than a native build on a Raspberry Pi 3 B+ and run
into the 4 hour time limit imposed for autobuilds. Overcoming this would have
required to split the Dockerfile.\*\_deb build files into 3 sequential builds
each, which was not considered worthwhile.
required to split the Dockerfile.\*\-deb build files into 3 sequential builds
each, which was not considered worthwhile. So the infrastructure to hook into
dockerhub autobuilds has been removed again from this repository.

The Dockerfiles are set up for cross-build by default. To build natively, first
comment out the cross-build commands:
```
perl -i -pe 's/(.*cross-build-(start|end).*)/# $1/' docker/Dockerfile.*
```

To build, run the hooks/build script:
Since the Dockerfiles depend on a number of ARGs that must be passed in on the
command line, the preferred way of building the images is to use the wrapper
shell script `build.sh`.

To build, run:
```
cd docker
IMAGE_NAME=arturklauser/raspberrypi-rstudio-build-env DOCKERFILE_PATH=Dockerfile.build_env hooks/build
IMAGE_NAME=arturklauser/raspberrypi-rstudio-server-deb DOCKERFILE_PATH=Dockerfile.server_deb hooks/build
IMAGE_NAME=arturklauser/raspberrypi-rstudio-desktop-deb DOCKERFILE_PATH=Dockerfile.desktop_deb hooks/build
for stage in build-env server-deb desktop-deb; do
./build.sh stretch "${stage}"
done
```

To reduce the memory pressure, the build uses only a parallelism of 2 by
default. If you are running out of memory you can try reducing that to 1 by
adding `--build-arg BUILD_PARALLELISM=1` to the docker build command line. On
the other hand, if you are cross-building on a host with sufficient memory you
can increase this, e.g. on a host with >= 8 GB of RAM add `--build-arg
BUILD_PARALLELISM=4` to the docker build command line.
changing it in the build.sh script. On the other hand, if you are cross-building
on a host with sufficient memory you can increase this, e.g. on a host with >= 8
GB of RAM you can use `BUILD_PARALLELISM=4`.

Once the build of each Debian package is done, the build environment is
jettisoned and the packages is copied into an empty container's root directory.
Expand All @@ -87,14 +90,14 @@ basic R programs in RStudio Server but doesn't have any extras installed. For
this you would add the `--target install-minimal` to the docker build command.
The default is to build a more fully featured runtime with:
```
cd docker
IMAGE_NAME=arturklauser/raspberrypi-rstudio-server DOCKERFILE_PATH=Dockerfile.server hooks/build
./build.sh stretch server
```
The full runtime also has the necessary system and R packages installed to
support working with .Rmd files including latex for generating PDF, as well as
source code version control. It also contains compile environments for C, C++
and Fortran which are often used when you install additional R source packages
from CRAN in your R user environment.
from CRAN in your R user environment, as well as R packages for data cleaning,
manipulation, and plotting.

## Running RStudio Server
Once you have the raspberrypi-rstudio-server created you can start an RStudio
Expand All @@ -114,9 +117,12 @@ your web browser.
Most likely you will want to keep the results of your work around across
container restarts. For this, the server image is expected to be used with a
working directory from your host (`$PWD/work` above) mounted into the home
directory `/home/rstudio` of the user in the `rserver` container. If you have
specified a different `USER` at build time, you have to adjust the `/home`
directory accordingly.
directory `/home/rstudio` of the user in the `rserver` container. Make sure that
the directory on the host exists before you pass it into the docker container,
otherwise container engine will probably create is as root which will give you
permission problems when trying to create files in it from within the container.
If you have specified a different `USER` at build time, you have to adjust the
`/home` directory accordingly.

## Getting the .deb Package Files
If what you want to do is not running RStudio in a Docker container but
Expand Down
104 changes: 104 additions & 0 deletions build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
#!/bin/bash
#
# Build an RStudio docker image.

# Account name prefix for docker image tags.
readonly DOCKERHUB_USER='arturklauser'

# Print usage message with error and exit.
function usage() {
if [ "$#" != 0 ]; then
(echo "$1"; echo) >&2
fi
cat - >&2 << END_USAGE
Usage: build.sh <debian-version> <build-stage>
debian-version: stretch ... Debian version 9
buster .... Debian version 10
build-stage: build-env ..... build environment
server-deb .... server Debian package
desktop-deb ... desktop Debian package
server ........ server runtime environment
END_USAGE
exit 1
}

# Print standardized timestamp.
function timestamp() {
date -u +'%Y-%m-%dT%H:%M:%SZ'
}

function main() {
if [[ "$#" != 2 ]]; then
usage "Invalid number ($#) of command line arguments."
fi

# Define build environment.
readonly DEBIAN_VERSION="$1"
readonly BUILD_STAGE="$2"

echo "Start building at $(timestamp) ..."

# Define RStudio source code version to use and the package release tag.
case "${DEBIAN_VERSION}" in
'stretch')
# As of 2019-04-06 v1.1.463 is the latest version 1.1 tag.
readonly VERSION_MAJOR=1
readonly VERSION_MINOR=1
readonly VERSION_PATCH=463
readonly PACKAGE_RELEASE='1~r2r'
;;
'buster')
# As of 2019-04-06 v1.2.1335 is the latest version 1.2 tag.
readonly VERSION_MAJOR=1
readonly VERSION_MINOR=2
readonly VERSION_PATCH=1335
readonly PACKAGE_RELEASE='1~r2r'
;;
*)
usage "Unsupported Debian version '${DEBIAN_VERSION}'"
;;
esac

# Define image tag and dockerfile depending on requested build stage.
case "${BUILD_STAGE}" in
'build-env' | 'server-deb' | 'desktop-deb' | 'server')
readonly IMAGE_NAME="${DOCKERHUB_USER}/raspberrypi-rstudio-${BUILD_STAGE}"
readonly DOCKERFILE="docker/Dockerfile.${BUILD_STAGE}"
;;
*)
usage "Unsupported build stage '${BUILD_STAGE}'"
;;
esac

readonly VERSION_TAG=${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}
readonly BUILD_PARALLELISM=2

# If we're running on ARM we comment out the cross-build lines.
if [[ $(uname -m) =~ 'arm' ]]; then
readonly CROSS_BUILD_FIX='s/^(.*cross-build-.*)/# $1/'
else
readonly CROSS_BUILD_FIX=''
fi

# Build the docker image.
set -x
time \
perl -pe "${CROSS_BUILD_FIX}" "${DOCKERFILE}" \
| docker build \
--build-arg DEBIAN_VERSION="${DEBIAN_VERSION}" \
--build-arg VERSION_TAG="${VERSION_TAG}" \
--build-arg VERSION_MAJOR="${VERSION_MAJOR}" \
--build-arg VERSION_MINOR="${VERSION_MINOR}" \
--build-arg VERSION_PATCH="${VERSION_PATCH}" \
--build-arg PACKAGE_RELEASE="${PACKAGE_RELEASE}" \
--build-arg BUILD_PARALLELISM="${BUILD_PARALLELISM}" \
--build-arg VCS_REF=$(git log --pretty=format:'%H' HEAD~..HEAD) \
--build-arg BUILD_DATE=$(timestamp) \
-t "${IMAGE_NAME}:${VERSION_TAG}-${DEBIAN_VERSION}" \
-
set +x

echo "Done building at $(timestamp)"
}

main "$@"
110 changes: 74 additions & 36 deletions docker/Dockerfile.build_env → docker/Dockerfile.build-env
Original file line number Diff line number Diff line change
Expand Up @@ -3,69 +3,89 @@
# This image can either be cross-built on an x86 host or built natively on a
# Raspberry Pi3 B+ (RP3) with 1 GB of RAM.
#
# Run the build like so:
# docker build -t raspberrypi-rstudio-build-env -f Dockerfile.build_env .
# This will create a build environment image that can be used for compiling
# RStudio Server and Desktop Debian packages.
#
# Some of this work is based on the excellent write-up of RStudio installation at
# http://herb.h.kobe-u.ac.jp/raspiinfo/rstudio_en.html
#
# Run the build with the build.sh script to make sure all required ARGs are
# passed in. This will create a build environment image that can be used for
# compiling RStudio Server and Desktop Debian packages.

ARG DEBIAN_VERSION
ARG VERSION_TAG

# See https://www.balena.io/docs/reference/base-images/base-images/
FROM balenalib/raspberrypi3-debian:stretch-build as build_env
FROM balenalib/raspberrypi3-debian:${DEBIAN_VERSION}-build as build_env

# Balenalib images for arm can be cross-built on x86.
# Comment out next line for native build.
RUN [ "cross-build-start" ]

# Define RStudio source code version to use and the package release tag.
# Override these ARGs on the command line:
# docker build ... \
# --build-arg VERSION_MAJOR=123 ...
# Define Debian version, RStudio version to use and the package release tag.
ARG DEBIAN_VERSION
ARG VERSION_MAJOR
ARG VERSION_MINOR
ARG VERSION_PATCH
ARG VERSION_TAG
ARG PACKAGE_RELEASE

# Automatically computed variables - don't override on command line.
ARG VERSION_TAG=${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}
ARG BUILD_TAG=${VERSION_TAG}-${PACKAGE_RELEASE}

RUN apt-get update && apt-get upgrade -y && apt-get install -y \
galternatives \
git \
openjdk-8-jdk \
pandoc \
pandoc-citeproc \
python-dev \
r-recommended
#--- begin buster
# Debian 10 (Buster) repositories are not signed yet, so override apt
# security checks.
RUN set -x; \
if [ "${DEBIAN_VERSION}" = 'buster' ]; then \
perl -i -pe 's/^deb (\[([^\]]*)\](\s*))?/deb [trusted=yes$3$2] /' /etc/apt/sources.list; \
fi
# Debian 10 (Buster) ca-certificates package misses all cert-hash links in
# /etc/ssl/certs, causing SSL (e.g. curl https://...) to fail. Repopulate those
# symlinks.
RUN set -x; \
if [ "${DEBIAN_VERSION}" = 'buster' ]; then \
update-ca-certificates --fresh; \
fi
#--- end buster

# Upgrade base image.
RUN set -x \
&& apt-get update \
&& apt-get upgrade -y \
&& apt autoremove

RUN mkdir -p /home/pi/Downloads
WORKDIR /home/pi/Downloads
# Install some basic build dependencies.
RUN set -x \
&& apt-get install -y \
galternatives \
git \
openjdk-8-jdk \
pandoc \
pandoc-citeproc \
python-dev \
r-recommended

# Get the RStudio sources. We only download the version of the code we need,
# instead of HEAD. HEAD has dependencies that RaspberryPi3 Debian (v9 Stretch)
# Get the RStudio sources. We only make a shallow clone of the version we
# need, instead of cloning the whole repository.
# Note that HEAD has dependencies that RaspberryPi3 Debian 9 (Stretch)
# currently doesn't satisfy, e.g. QT >= 5.10.
RUN set -x \
&& wget https://github.com/rstudio/rstudio/archive/v${VERSION_TAG}.tar.gz -O rstudio.tgz \
&& tar xzf rstudio.tgz \
&& rm rstudio.tgz \
&& ln -s rstudio-${VERSION_TAG} rstudio
# Alternatively, you could download the whole repository (~ 10x larger) and
# check out the specific version tag.
# RUN git clone https://github.com/rstudio/rstudio.git && \
# cd rstudio && \
# git checkout v${VERSION_TAG}
&& mkdir -p /home/pi/Downloads \
&& cd /home/pi/Downloads \
&& git clone -b v${VERSION_TAG} --depth 1 https://github.com/rstudio/rstudio.git \
&& ln -s rstudio rstudio-${VERSION_TAG}

WORKDIR /home/pi/Downloads/rstudio/dependencies/linux
# We're going to use the system version of the boost library (version
# 1.62.0) instead of installing and building the RStudio version of boost
# (version 1.63.0) which takes several hours of build time.
# Comment out boost source installation.
RUN perl -i -pe 's/(^.*install-boost$)/# \1/s' ../common/install-common
RUN apt-get install libboost-all-dev
# Script installs qt-sdk only for x86 but we need armhf, so skip qt install.
# Install system boost.
RUN apt-get install libboost1.62-all-dev
# We use the system QT libraries instead of RSTudio's.
# Script installs qt-sdk binaries for x86 but we need armhf, so skip it.
RUN ./install-dependencies-debian --exclude-qt-sdk
# Extra dependencies to build rstudio-desktop: use the system QT libraries instead.
# Install system QT libraries.
RUN apt-get install -y \
libqt5opengl5-dev \
libqt5sensors5-dev \
Expand All @@ -82,18 +102,36 @@ WORKDIR /home/pi/Downloads/rstudio
# RStudio desktop .deb package. But bundling of system QT doesn't work due
# to path differences and is unnecessary anyway.
RUN sed -i -e 's#\(set(RSTUDIO_BUNDLE_QT\) TRUE#\1 FALSE#' src/cpp/desktop/CMakeLists.txt
# Add the package dependencies to system QT libraries.

#--- begin buster
# We need libssl1.0.2. No other version will work since we're compiling against
# that version and have a dynamic library depenency on it.
RUN sed -i -e 's#libssl1.0.0 | libssl1.0.2 | libssl1.1, #libssl1.0.2, #' package/linux/CMakeLists.txt
#--- end buster

# For Desktop, add the package dependencies to system QT libraries.
#--- begin stretch
RUN sed -i -e 's#\(^.*set(RSTUDIO_DEBIAN_DEPENDS "${RSTUDIO_DEBIAN_DEPENDS}libgstreamer0.10.*$\)#\1\nset(RSTUDIO_DEBIAN_DEPENDS "${RSTUDIO_DEBIAN_DEPENDS}libqt5webchannel5, libqt5webkit5, libqt5positioning5, libqt5sensors5, libqt5svg5, libqt5xml5, libqt5xmlpatterns5, ")#' package/linux/CMakeLists.txt
#--- end stretch
#--- begin buster
RUN sed -i -e 's#\(^.*set(RSTUDIO_DEBIAN_DEPENDS "psmisc,.*$\)#\1\nset(RSTUDIO_DEBIAN_DEPENDS "${RSTUDIO_DEBIAN_DEPENDS}libqt5webchannel5, libqt5webkit5, libqt5positioning5, libqt5sensors5, libqt5svg5, libqt5xml5, libqt5xmlpatterns5, ")#' package/linux/CMakeLists.txt
#--- end buster

# Add our own package release (aka. Debian revision number) to the .deb file.
RUN sed -i -e 's#\(^.*\)-\(${PACKAGE_ARCHITECTURE}\)#set(CPACK_DEBIAN_PACKAGE_RELEASE "'${PACKAGE_RELEASE}'")\n\1-${CPACK_DEBIAN_PACKAGE_RELEASE}_\2#' package/linux/CMakeLists.txt

# Constrain gwtc compiler to use a single worker - gwtc is a memory hog and
# running multiple workers is getting too tight with 1 GB RAM + 1 GB swap.
RUN perl -0777 -i -pe 's#(<arg value="-localWorkers"/>\s+<arg value=)"\d+"(/>)#\1"1"\2#s' src/gwt/build.xml
# Reduce heap size to 600 MB to avoid vitual memory thrashing on our 1 GB
# RAM RPi3. Below 600 MB we'd run the jvm out of heap space.
RUN perl -i -pe 's#-Xmx1536#-Xmx600#' src/gwt/build.xml

#--- begin stretch
# Change "cmake_policy(SET CMP0020 OLD)" to "cmake_policy(SET CMP0043 OLD)"
# to avoid a build time complaint.
RUN sed -i -e s/CMP0020/CMP0043/g src/cpp/desktop/CMakeLists.txt
#--- end stretch

# We build in a separate directory.
RUN mkdir build
Expand Down
Loading

0 comments on commit f7dd025

Please sign in to comment.