rajewsky-lab · danilexn · Jan 22, 2024 · Jan 22, 2024 · Jan 22, 2024 · Jan 22, 2024
diff --git a/.coveragerc b/.coveragerc
@@ -0,0 +1,4 @@
+[run]
+concurrency = multiprocessing
+parallel = true
+sigterm = true
diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@
 
 # Spacemake: processing and analysis of large-scale spatial transcriptomics data
 ### [🌐 docs](https://spacemake.readthedocs.io/en/latest/) | [📜 paper](https://doi.org/10.1093/gigascience/giac064) | [💬 discussions](https://github.com/rajewsky-lab/spacemake/discussions)
-<img src="https://raw.githubusercontent.com/rajewsky-lab/spacemake/master/docs/graphical_abstract_twitter.png" width="400" />
+<img src="https://raw.githubusercontent.com/rajewsky-lab/spacemake/master/docs/smk_logo.png" width="200" />
 
 Spacemake is a modular, robust, and scalable spatial transcriptomics pipeline built in `Snakemake` and `Python`. Spacemake is designed to handle all major spatial transcriptomics datasets and can be readily configured for other technologies. It can process and analyze several samples in parallel, even if they stem from different experimental methods. Spacemake's unified framework enables reproducible data processing from raw sequencing data to automatically generated downstream analysis reports. Spacemake is built with a modular design and offers additional functionality such as sample merging, saturation analysis, and analysis of long reads as separate modules.
 

diff --git a/docs/config.rst b/docs/config.rst
@@ -1,13 +1,7 @@
 Configuration
 =============
 
-Once installed, spacemake needs to be configured.
-
-.. include:: shared/spacemake_init.rst
-
-Optionally, you can also provide the ``--download_species`` flag, which will download Gencode genomes and
-annotations for ``mouse`` and ``human``, and place them under ``project\_root/species\_data/<species>``,
-where <species> is either mouse or human.
+Once installed and initialized, spacemake needs to be configured.
 
 .. include:: shared/shared_sample_variables.rst
 
@@ -167,7 +161,7 @@ Each ``run_mode`` can have the following variables:
    If a one of the ``run_mode`` variables is missing, the variable of the parent will be used.
    If parent is not provided, the ``default`` ``run_mode`` will be the parent. 
 
-Provided run\_mode(s)
+Provided run\_modes
 ^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: yaml

diff --git a/docs/graphical_abstract_twitter.png b/docs/graphical_abstract_twitter.png
diff --git a/docs/install.rst b/docs/install.rst
@@ -5,34 +5,32 @@ Step 1: create conda environment
 --------------------------------
 
 The most straightforward way to install spacemake, is to first create conda environment with the above packages.
-
-To do this, we highly recommend using `mamba <https://github.com/mamba-org/mamba>`_, a much faster conda package manager than conda itself.
-
-After mamba is installed, download the `environment.yaml <https://raw.githubusercontent.com/rajewsky-lab/spacemake/master/environment.yaml>`_. This environment.yaml, contains all the dependencies required by spacemake.
+We highly recommend using `mamba <https://github.com/mamba-org/mamba>`_, a much faster conda package manager than conda itself.
+After mamba is installed, download the `environment.yaml <https://raw.githubusercontent.com/rajewsky-lab/spacemake/master/environment.yaml>`_.
+This file contains all dependencies required by spacemake.
 
 Once downloaded, to install all spacemake dependencies type::
 
     mamba env create -f environment.yaml
 
-This will create a conda environment called ``spacemake``.
-
-Too activate the newly created environment type::
+This will create a conda environment called ``spacemake``. To activate the newly created environment type::
 
    conda activate spacemake
 
 Step 2: download Dropseq-tools
 ------------------------------
 
-To work with spacemake, currently it is needed to download `Dropseq-tools from here <https://github.com/broadinstitute/Drop-seq>`_.
-This packages is a collection of processing tools originally written for `Drop-seq <https://www.cell.com/cell/fulltext/S0092-8674(15)00549-8>`_. Spacemake uses several functions from this package during pre-processing and processing and without it it is impossible to run spacemake.
+Spacemake currently requires to download `Dropseq-tools <https://github.com/broadinstitute/Drop-seq>`_.
+This package is a collection of processing tools originally written for `Drop-seq <https://www.cell.com/cell/fulltext/S0092-8674(15)00549-8>`_. Spacemake uses several functions from this package during pre-processing and processing, and without it it is impossible to run spacemake.
 
 Simply download one of the releases (we recommend using `2.5.1 <https://github.com/broadinstitute/Drop-seq/releases/download/v2.5.1/Drop-seq_tools-2.5.1.zip>`_) and place it somewhere in your filesystem.
 
 
 Step 3: install spacemake
 -------------------------
 
-**After creating the conda environment and downloading Dropseq-tools** (as described above) spacemake can be installed via ``pip``::
+**After creating the conda environment and downloading Dropseq-tools** (as described above),
+spacemake can be installed via ``pip``::
 
    pip install spacemake
 
@@ -49,4 +47,4 @@ To make sure spacemake has been properly installed, run::
 
    spacemake --version
 
-This should output the latest spacemake version available in pip.
+This should output the latest spacemake version available in ``pip``.
diff --git a/docs/projects/index.rst b/docs/projects/index.rst
@@ -1,11 +1,12 @@
 Manage projects and samples
 ===========================
 
-In spacemake each sample, and it's settings, are stored in the ``project_df.csv`` under the root
+In spacemake, each sample and its settings are stored in the ``project_df.csv`` under the root
 directory of the spacemake project.
 
-Each sample will have exactly one row in this ``project_df.csv`` file. In the back-end, spacemake uses a ``pandas.DataFrame`` to load, and save this ``.csv`` file on disk. This data-frame
-will be indexed by key ``(project_id, sample_id)``
+Each sample has exactly one row in this ``project_df.csv`` file. In the back-end, spacemake uses 
+a ``pandas.DataFrame`` to load and save this ``.csv`` file on disk. This data frame
+is indexed by key ``(project_id, sample_id)``
 
 The spacemake class responsible for this back-end logic is the :ref:`ProjectDF<ProjectDF>` class.
 
@@ -312,8 +313,8 @@ To use this functionality, type::
 
 The sample-sheet columns have to obey certain conventions for spacemake to parse it properly:
 
-* ``Sample_ID`` contains the ``sample_id``s in the project.
-* ``Sample_Project`` contains the ``project_id``s in the project.
+* ``Sample_ID`` contains the ``sample_id`` in the project.
+* ``Sample_Project`` contains the ``project_id`` in the project.
 * ``Description`` must end with ``_species``, where species is the one configured for the samples in the project, e.g. ``HEK293_wt_human``.
 
 Spacemake will also parse the fields ``Investigator``, ``Date``, and ``Experiment`` from the sample-sheet and add them to the project metadata.

diff --git a/docs/shared/shared_sample_variables.rst b/docs/shared/shared_sample_variables.rst
@@ -1,18 +1,17 @@
 One of the most important parts of spacemake are the so-called 'shared sample-variables'.
 These are reusable, user-definable variables, which we can assign to several samples.
-
 They can be shortly defined as follows:
 
-species
+``species``
    a collection of genome, annotation and rRNA\_genome. There is no default species, and each sample can have exactly one species.
 
-barcode\_flavor
+``barcode_flavor``
    the variable which specifies the structure of Read1 and Read2, namely how the cell\_barcode and UMI should be extracted. If no value provided for a sample, the default will be used.
 
-run\_mode
+``run_mode``
    each sample can have several ``run_mode``-s, all of which are user definable. If no ``run_mode``-s are specified, a sample will be processed using ``default`` ``run_mode`` settings.
 
-puck (for spatial samples only)
+``puck`` (spatial only)
    if a sample is spatial, it has to have a puck variable. If no puck is specified, a default puck will be used.  
 
 

diff --git a/docs/shared/spacemake_init.rst b/docs/shared/spacemake_init.rst
@@ -1,7 +1,6 @@
 After you have installed spacemake as specified :ref:`here <installation>`, you are ready to process and analyze spatial samples.
 
 To initialize spacemake ``cd`` into the directory in which you want to start spacemake. This directory will be your ``project_root``.
-
 Then simply type::
 
    spacemake init \

diff --git a/docs/smk_logo.png b/docs/smk_logo.png
diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
@@ -7,6 +7,50 @@ always possible to resolve some of these issues.
 Bumped on another problem which is not documented here? Feel free to `open
 an issue on Github. <https://github.com/rajewsky-lab/spacemake/issues>`_
 
+
+GLIBCXX_xxx not found
+^^^^^^^^^^^^^^^^^^^^^
+In certain environments you might run into the following error:
+
+.. code-block:: console
+
+    ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /.../envs/spacemake/lib/python3.10/site-packages/matplotlib/_path.cpython-310-x86_64-linux-gnu.so)
+
+Certain dependencies (e.g. scipy) might affect this. To overcome it, try the following:
+
+.. code-block:: console
+
+   export LD_LIBRARY_PATH=/conda_installation_folder/envs/spacemake/lib:$LD_LIBRARY_PATH
+
+For more details on this and further troubleshooting visit `stackoverflow <https://stackoverflow.com/questions/72540359/glibcxx-3-4-30-not-found-for-librosa-in-conda-virtual-environment-after-tryin>`_.
+
+
+Issues with memory usage
+^^^^^^^^^^^^^^^^^^^^^^^^
+Certain steps of the spacemake workflow might currently result in excessive memory
+usage. These occur in samples where a large number of barcodes exist in the data,
+such as for ``stero-seq`` or ``open-ST``. The excessive memory usage is due to using
+the ``Drop-seq`` tools, where a specific memory size is allocated for ``java``.
+
+We are working on removing the depndency to the ``Drop-seq`` tools altogether,
+which will also speed up several steps. If you run into memory errors, however, you
+can solve them by modifying the ``main.smk`` file inside your spacemake installation,
+which should be somewhere in
+
+.. code-block:: console
+
+    /path_to_conda/envs/spacemake/lib/python3.10/site-packages/spacemake/snakemake
+
+inside your conda installation folder. Simply modify the following lines
+
+.. code-block:: console
+
+    {dropseq_tools}/BamTagHistogram -m 32g
+    {dropseq_tools}/DigitalExpression -m 16g
+
+by increasing the value of ``-m`` accordingly.
+
+
 Issues with STAR
 ^^^^^^^^^^^^^^^^
 To reduce memory usage when running several samples at the same time, 
@@ -30,20 +74,3 @@ In case the shared memory cannot be released (Linux), try the following:
 .. code-block:: console
 
     ipcs -m | grep `whoami` | awk '{ print $2 }' | xargs -n1 ipcrm -m
-
-
-GLIBCXX_xxx not found
-^^^^^^^^^^^^^^^^^^
-In certain environments you might run into the following error:
-
-.. code-block:: console
-
-    ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /.../envs/spacemake/lib/python3.10/site-packages/matplotlib/_path.cpython-310-x86_64-linux-gnu.so)
-
-Certain dependencies (e.g. scipy) might affect this. To overcome it, try the following:
-
-.. code-block:: console
-
-   export LD_LIBRARY_PATH=/conda_installation_folder/envs/spacemake/lib:$LD_LIBRARY_PATH
-
-For more details on this and further troubleshooting visit `stackoverflow <https://stackoverflow.com/questions/72540359/glibcxx-3-4-30-not-found-for-librosa-in-conda-virtual-environment-after-tryin>`_.
diff --git a/environment.yaml b/environment.yaml
@@ -3,7 +3,7 @@ channels:
     - bih-cubi
     - conda-forge
     - bioconda
-    - defaults
+    - nodefaults
 dependencies:
     - python>=3.6,<3.12
     - snakemake>=5.32.0,<6.4.0
@@ -27,7 +27,12 @@ dependencies:
     - openjdk==11.0.15
     - pigz
     - pip:
-        - pandas>=1.3.0,<=1.5.1
+        - setproctitle
+        - isal
+        - pytest
+        - pytest-cov
+        - mrfifo>=0.3.0
+        - pandas>2
         - scanpy>=1.8.1
         - leidenalg>=0.8.1
         - numpy>=1.18.1
@@ -40,5 +45,5 @@ dependencies:
         - novosparc
         - opencv-python
         - jinja2>=3.1.3
-        - pytest
+        - matplotlib==3.8.4
 #        - pytest-optional-tests
diff --git a/setup.cfg b/setup.cfg
@@ -45,3 +45,4 @@ console_scripts =
 testpaths = tests
 markers = 
     big_download: needs to download large-ish files
+addopts = --cov=spacemake --cov-report html