Skip to content

Commit

Permalink
BUG Use /tmp for joblib temp files instead of /shm (#37)
Browse files Browse the repository at this point in the history
Instruct joblib to use disk for temporary files. Joblib defaults to /shm when that directory is present. In the Docker container, /shm is present but defaults to 64 MB. That's not enough space to handle even moderate-sized computations. The temporary directory is a bit slower, but has much more space.

 https://github.com/joblib/joblib/blob/0.11/joblib/parallel.py#L328L342
  • Loading branch information
Stephen Hoover authored Jun 20, 2017
1 parent e8e81a3 commit 5289ae8
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 0 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ Version number changes (major.minor.micro) in this package denote the following:

## Unreleased

### Fixes
- Use /tmp for joblib temporary files instead of /shm


## [3.0.1] - 2017-05-25
### Package Updates
- muffnn 1.1.1 -> 1.1.2
Expand Down
6 changes: 6 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,12 @@ RUN mkdir -p ${HOME}/.config/matplotlib && \
echo "backend : Agg" > ${HOME}/.config/matplotlib/matplotlibrc && \
python -c "import matplotlib.pyplot"

# Instruct joblib to use disk for temporary files. Joblib defaults to
# /shm when that directory is present. In the Docker container, /shm is
# present but defaults to 64 MB.
# https://github.com/joblib/joblib/blob/0.11/joblib/parallel.py#L328L342
ENV JOBLIB_TEMP_FOLDER=/tmp

ENV VERSION=3.0.1 \
VERSION_MAJOR=3 \
VERSION_MINOR=0 \
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ VERSION_MICRO
VERSION contains the full version string, e.g. "1.0.3". VERSION_MAJOR,
VERSION_MINOR, and VERSION_MICRO each contain a single integer.

## Joblib Temporary Files

The [`joblib`](https://pythonhosted.org/joblib/) library enhances multiprocessing
capabilities for scientific Python computing. In particular, the `scikit-learn`
library uses `joblib` for parallelization. This Docker image sets `joblib`'s
default location for staging temporary files to the /tmp directory.
The normal default is /shm. /shm is a RAM disk which defaults to a 64 MB size
in Docker containers, too small for typical scientific computing.

# Creating Equivalent Local Environments

The `environment.yml` file in this repo can be used to create a python environment that is
Expand Down

0 comments on commit 5289ae8

Please sign in to comment.