uopdate readmes

README.md

@@ -3,7 +3,7 @@
 
 This repository hosts the version of the code used for the [preprint](https://www.biorxiv.org/content/10.1101/2021.05.12.443427v1) *Multiscale light-sheet organoid imaging framework* (de Medeiros, Ortiz et al 2021).
 
-**LSTree** is a digital organoid and lineage tree extraction framework for light sheet movies. It combines pre-processing and analysis tools to extract multiple features from the 3D recorded data. Ultimately, the extracted features can be visualized onto both lineage trees and 3D segmentation meshes in a combined way via a web-based viewer.
+**LSTree** is a digital organoid and lineage tree extraction workflow for light-sheet movies. It combines pre-processing and analysis tools to extract multiple features from the 3D recorded data. Ultimately, the extracted features can be visualized onto both lineage trees and 3D segmentation meshes in a combined way via a web-based viewer.
 
 Below you will find instructions on how to install the environment to run LSTree as well as how to run it on the two example datasets.
 
@@ -88,6 +88,43 @@ pip install LSTree/
 # Usage
 The entire analysis pipeline is implemented as a [Luigi workflow](https://github.com/spotify/luigi) and majors steps can be run with the commands detailed below and on the following sections. Jupyter notebooks for interactive [visualization of the results](/webview/webview.ipynb) and [drawing 3D labels](/notebooks/3D_annotator.ipynb) are also provided.
 
+
+
+## Initial Requirements 
+
+### **MaMuT.xml**
+
+As the name hints, **LSTree** heavily depends on the existence of tracking data, in form of a MaMuT `.xml` file, so that all features of a light-sheet recording can be extracted. To fulfill this requirement lineage trees can be created via [Mastodon](https://github.com/mastodon-sc/mastodon), or [Elephant](https://elephant-track.github.io/#/v0.3/) or any other tracking algorithm as long as the output can be written in MaMuT `.xml` form.Ideally, to be able to extract all features, each movie folder should include a MaMuT (`mamut.xml`) lineage tree (see Lineage tree section below) along with an [experiment.json](/example/data/002-Budding/experiment.json) file containing information about acquisition settings which are used e.g. for rescaling, deconvolution and for showing the data with the right temporal spacing, among others:
+
+
+### **experiment.json**
+En [experiment.json](/example/data/002-Budding/experiment.json) file containing information about acquisition settings which are used e.g. for rescaling, deconvolution and for showing the data with the right temporal spacing, among others:
+
+```
+{
+    "mag": 25,                  # Magnification (for getting the right PSF file)
+    "time_interval": 0.1667,    # value in hours
+    "spacing": [
+        2,                      # Z
+        0.26,                   # Y
+        0.26                    # X
+    ],
+    "wavelengths": {            # in nanometers
+        "Channel0": 488,        
+        "Channel1": 561,
+        "Channel2": 638
+    }
+}
+```  
+### **dataset.xml**
+
+Before poerforming segmentation and feature extraction, LSTree requires a `dataset.xml` file that has the structure of the BigDataViewer `.xml`.
+- some information is gotten from here that is necessary for running the segmentation (spacing?`?)  
+
+LINK TO DATASET FILE IN EXAMPLE DATA
+more information linbk to bdv!!
+
+
 ## Folder structure
 A certain data structure is expected so that the workflow can run smoothly: it should ideally be organized with 2-level sub-folders for movie and channels respectively:
 
@@ -96,6 +133,7 @@ A certain data structure is expected so that the workflow can run smoothly: it s
 └── MOVIE_DIRECTORY
     ├── experiment.json
     ├── mamut.xml
+    ├── dataset.xml
     ├── nuclei_annot
     │   ├── FILENAME-T0017.tif
     │   ├── FILENAME-T0134.tif
@@ -124,36 +162,10 @@ A certain data structure is expected so that the workflow can run smoothly: it s
 
 Generated outputs will appear as new sub-folders (E.g. Channel0-Deconv, Channel1-Deconv, nuclei_segmentation, cell_segmentation, etc.).
 
----
-
-## Initial Requirements 
-
-### **MaMuT.xml**
-
-As the name hints, **LSTree** heavily depends on the existence of tracking data, in form of a MaMuT `.xml` file, so that all features of a light-sheet recording can be extracted. To fulfill this requirement lineage trees can be created via [Mastodon](https://github.com/mastodon-sc/mastodon), or [Elephant](https://elephant-track.github.io/#/v0.3/) or any other tracking algorithm as long as the output can be written in MaMuT `.xml` form.Ideally, to be able to extract all features, each movie folder should include a MaMuT (`mamut.xml`) lineage tree (see Lineage tree section below) along with an [experiment.json](/example/data/002-Budding/experiment.json) file containing information about acquisition settings which are used e.g. for rescaling, deconvolution and for showing the data with the right temporal spacing, among others:
+> :warning: **Folder structure organisation**: 
 
 
-### **experiment.json**
-En [experiment.json](/example/data/002-Budding/experiment.json) file containing information about acquisition settings which are used e.g. for rescaling, deconvolution and for showing the data with the right temporal spacing, among others:
-
-```
-{
-    "mag": 25,                  # Magnification (for getting the right PSF file)
-    "time_interval": 0.1667,    # value in hours
-    "spacing": [
-        2,                      # Z
-        0.26,                   # Y
-        0.26                    # X
-    ],
-    "wavelengths": {            # in nanometers
-        "Channel0": 488,        
-        "Channel1": 561,
-        "Channel2": 638
-    }
-}
-```  
 
----
 ## Configuration file
 General parameters for each tasks are configured through a global configuration file [config.cfg](config.cfg), which is initially set up to run the [example](example/README.md) datasets.
 
@@ -172,6 +184,10 @@ General parameters for each tasks are configured through a global configuration
 
 
 ---
+## Acknowledgements
+
+Big thanks to the [Liberali Lab](https://liberalilab.org/) for enabling the creation of these tools, the support and help with the initial development, discussions and future plans of LSTree. Also to Ko Sugawara for the nice collaboration regarding the utilization of `Elephant` and its integration within the `LSTree` umbrella. Thanks to Vladimir Ulman for helpful discussions regarding tracking in general, and Markus Rempfler forimportant initial support with RDCNet implementations. Also big thanks to Tim-Oliver for insightful discussions and help with debugging. 
 
+---
 ## Funding support
 This work was supported by EMBO (ALTF 571-2018 to G.M.), SNSF (POOP3_157531 to P.L.). This work received funding from the ERC under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 758617).

example/README.md

@@ -1,38 +1,8 @@
 # Introduction - LSTree example usage
 
-To get acquainted with LSTree, we provide two test datasets that can be used for running the workflow. For simplicity they have already been pre-processed (only cropping, to reduce memory requirements and processing time) and the example lineage trees have been created and exported as MaMuT .xml files. Below we will describe what these steps entail, as well as providing more in-depth information on the main tasks behind the workflow.
-
-## Quickstart
-
-### 1) Processing the example data
-
-As long as the data has been cropped and lineage tree created/curated and exported via MaMuT .xml ( which is the case for the example datasets ), the entire LSTree workflow can be activated via the command below (this takes around 10 minutes with the hardware configuration we used):
-
-```bash
-LUIGI_CONFIG_PATH=./config.cfg luigi --local-scheduler --module lstree ViewerTask
-```
-
-By default this command will run on the test dataset provided (002-Budding and 003-Enterocyst) using all models trained with intestinal organoid images. Therefore, the configuration file must be first adapted to the right input paths before using it on new user data.
-
-
-### 2) Visualization of the output
-
-After processing is finished all the outputs can be visualized with the [webviewer](../webview/README.md).
-The included web-based viewer allows visualizing a lineage tree with a linked view of the 3D cell/nuclei segmentation at a given timepoint. More information on how to use it, along with example notebook can be found [here](../webview/README.md).
-
-
-
-
----
-
-> :warning: **Important: How to rerun tasks**: If there are samples for which the output files already exist, then these are skipped. To rerun a specific task* all necessary intermediate and final outputs should be deleted. That also include training deep learning models, i.e. if a trained model exist, it is used without retraining. ( *Lineage tree prediction is currently the only task that can resume in case it is suddenly stopped. ) 
-
-
-
----
-
-
+To get acquainted with LSTree, we provide two test datasets that can be used for running the workflow. For simplicity they have already been pre-processed (only cropping, to reduce memory requirements and processing time) and the example lineage trees have been created and exported as MaMuT .xml files. Below we will describe what these steps entail, as well as providing more in-depth information on the main tasks behind the workflow.`
 
+Each new step of the workflow will generate output within a dedicated folder, fwith specific naming conventions. These naming conventions can be changed in the `config.cfg` file.
 
 
 
@@ -49,8 +19,9 @@ Raw images are first denoised with [Noise2Void](https://github.com/juglab/n2v) t
 ```bash
 LUIGI_CONFIG_PATH=./config.cfg luigi --local-scheduler --module lstree MultiDeconvolutionTask
 ```
-> **Important**: The outputs from this step include a folder with '-Deconv' as a suffix, and the other steps use the images inside as input for segmentation. If this is not needed, a - so far still hacky - way around it is to copy the raw images to an empty folder with the same name before running the config file.
+> **-Deconv folder requirement**: The outputs from this step include a folder with '-Deconv' as a suffix, and the other steps use the images inside as input for segmentation. If this is not needed, a - so far still hacky - way around it is to copy the raw images to an empty folder with the same name before running the config file.
 
+> **PSF estimation**: As expected during deconvolution, a point-spread-function (PSF) of the optics used for imaging is necessary. You can uye [Huygens PSF Distiller](https://svi.nl/Huygens-PSF-Distiller) or the [PSF extraction from python-microscopy](http://python-microscopy.org/doc/PSFExtraction.html).
 
 ## 3. Lineage tree
 LSTree works directly with MaMuT `.xml` files that contain tracking data made using [Mastodon](https://github.com/mastodon-sc/mastodon) Fiji plugin. Subsequently a deep learning model can be (re)trained to predict trees that require fewer manual corrections. Alternatively, one can also use the output from [Elephant](https://elephant-track.github.io/#/v0.3/) ( as MaMuT `.xml` ) to fine tune an existing model or just as final lineage tree output for further processing (segmentation and feature extraction steps). 
@@ -147,3 +118,36 @@ The segmentation pipeline requires manual annotations of nuclei and lumen/epithe
 
 
 
+## Quickstart
+
+### 1) Processing the example data
+
+For an easy first-time use, you can run everything in one go via a single command. This will trigger the entire workflow on two example datasets( [002-Budding](/example/data/002-Budding/) and [003-Enterocyst](/example/data/003-Enterocyst/) ) using all models trained with `intestinal organoid images` (this takes around 10 minutes with the hardware configuration we used).
+
+As long as the data has been cropped and lineage tree created/curated and exported via MaMuT .xml ( which is the case for the example datasets ), the entire LSTree workflow can be activated via the command below (this takes around 10 minutes with the [hardware configuration we used](LINK)):
+
+```bash
+LUIGI_CONFIG_PATH=./config.cfg luigi --local-scheduler --module lstree ViewerTask
+```
+
+It is also possible to run each step separately on the example data.
+If you start from scratch using a different dataset(s) the configuration file needs to be properly changed. More information can be found inside the [config.cfg](../config.cfg) itself as well as in the [configuration file parameters](../config_params.md) readme.
+### 2) Visualization of the output
+
+After processing is finished all the outputs can be visualized with the [webviewer](../webview/README.md).
+The included web-based viewer allows visualizing a lineage tree with a linked view of the 3D cell/nuclei segmentation at a given timepoint. More information on how to use it, along with example notebook can be found [here](../webview/README.md).
+
+
+
+
+---
+
+> :warning: **Important: How to rerun tasks**: If there are samples for which the output files already exist, then these are skipped. To rerun a specific task* all necessary intermediate and final outputs should be deleted. That also include training deep learning models, i.e. if a trained model exist, it is used without retraining. ( *Lineage tree prediction is currently the only task that can resume in case it is suddenly stopped. ) 
+
+
+
+---
+
+
+
+

notebooks/README.md

@@ -8,7 +8,7 @@ jupter notebook --port=XXXX
 The jupyter notebook should initialize and port forward itself to your localhost using the XXXX specified port. This way you just need to open a browser and type:
 
 ```bash
-loalhost:XXXX
+localhost:XXXX
 ```
 and the jupyter notebook should open. Then you can browser yourself to the /notebook folder where the utilities are located.
 

webview/README.md

@@ -8,7 +8,7 @@ The included web-based viewer allows visualizing a lineage tree with a linked vi
 ## Running via the notebook / port forwarding
 You can run the Webviewer via the jupyter [notebook](webview.ipynb). This is a good first way to look at the tree-meshes visualization, as it is set to look initially at the processed example datasets.
 
-Alternatively, you can also use port forwarding to serve trhe notebook for remote visualization.
+Alternatively, you can also use port forwarding to serve the notebook for remote visualization.
 
 The viewer is then accessible either as a jupyter notebook  or directly served to the browser by activating the lstree environment and running:
 
@@ -23,16 +23,18 @@ or on a remote machine:
 
 ```bash
 cd /PATH_TO_LSTREE_REPOSITORY/webview
-panel serve --static-dirs static_data="static_data" --port PORT --allow-websocket-origin=WORKSTATION:PORT webview.ipynb  --args --basedir PATH_TO_PROCESSED_DATA
+panel serve --static-dirs static_data="static_data" --port PORT --allow-websocket-origin=WORKSTATION_NAME:PORT webview.ipynb  --args --basedir PATH_TO_PROCESSED_DATA
 ```
 
-Example, considering that the data is located in /Data:
+Example, considering that the data is located in `/Data/LSTree/example/data` and you are connected via e.g. ssh to `workstation1`:
 
 ```bash
 cd /LSTree/webview
-panel serve --static-dirs static_data="static_data" --port 7007 --allow-websocket-origin=workstation1:7007 webview.ipynb  --args --basedir /Data/LSTree/example/data
+panel serve --static-dirs static_data="static_data" --port 5174 --allow-websocket-origin=workstation1:5174 webview.ipynb  --args --basedir /Data/LSTree/example/data
 ```
 
+Port forwarding has been extensively tested with `--port=5174`.
+
 Extracted features such as the nuclei volume can be viewed as color:  
 <img src="../docs/viewer_volume.png" width="1000"/><br>