Separate data (pydicom#227)

* remove data
* add missing deid.dicom files

Signed-off-by: vsoch <[email protected]>

.github/workflows/docs.yaml

@@ -24,6 +24,7 @@ jobs:
           git clone https://github.com/pydicom/deid
           cd deid
           pip install -e .
+          pip install deid-data
           cd ./docs
           pip install -r api_docs/docs-requirements.txt
           /bin/bash ./apidoc.sh

.github/workflows/main.yaml

@@ -11,7 +11,7 @@ jobs:
   formatting:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v1
+      - uses: actions/checkout@v3
 
       - name: Setup black environment
         run: conda create --quiet --name black black
@@ -26,18 +26,19 @@ jobs:
   testing:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v1
+      - uses: actions/checkout@v3
 
       - name: Setup conda environment
         run: | 
           conda create --quiet --name testing
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate testing
           pip install matplotlib
-          python setup.py install
+          pip install .
 
       - name: Test deid
         run: |
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate testing
+          pip install deid-data==0.0.17
           python -m unittest discover -s deid/tests/ -p '[t|T]est*.py'

.travis.yml

@@ -14,7 +14,7 @@ matrix:
     python: "2.7"
 
 install:
-  - pip install pydicom matplotlib
+  - pip install pydicom matplotlib deid-data
   - cd $TRAVIS_BUILD_DIR/
   - python setup.py sdist
   - python setup.py install

deid/data/__init__.py

@@ -8,13 +8,9 @@
    from deid.data import get_dataset
 """
 
-
 import os
 
-from deid.logger import bot
-from deid.utils import get_installdir
-
-data_base = os.path.dirname(os.path.abspath(__file__))
+data_base = os.path.abspath(os.path.dirname(__file__))
 
 
 def get_dataset(dataset=None):
@@ -25,18 +21,9 @@ def get_dataset(dataset=None):
     based on a user-provided label. In the future, we can add https endpoints
     to retrieve online datasets.
     """
-    data_base = get_installdir()
-    valid_datasets = {
-        "dicom-cookies": os.path.join(data_base, "data", "dicom-cookies"),
-        "animals": os.path.join(data_base, "data", "animals"),
-        "humans": os.path.join(data_base, "data", "humans"),
-        "ultrasounds": os.path.join(data_base, "data", "ultrasounds"),
-    }
-
-    if dataset is not None:
-        # In case the user gave an extension
-        dataset = os.path.splitext(dataset)[0].lower()
-        if dataset in valid_datasets:
-            return valid_datasets[dataset]
-
-    bot.info("Valid datasets include: %s" % (",".join(list(valid_datasets.keys()))))
+    try:
+        from deid_data import data
+    except:
+        raise ValueError("install deid data with `pip install deid-data`")
+
+    return data.get_dataset(dataset)

deid/tests/test_utils.py

@@ -95,22 +95,9 @@ def test_recursive_find(self):
         print("Testing recursive find.")
         from deid.utils import recursive_find
 
-        found = 0
-        expected = 11
-        for file in recursive_find(self.pwd, pattern="*.dcm"):
-            found += 1
-        print("Found %s files" % (found))
-        self.assertTrue(found == expected)
-
-    def test_recursive_find_as_list(self):
-        """test_recursive_find_as_list should detect 7 dicoms"""
-        print("Testing recursive find as lit.")
-        from deid.utils import recursive_find
-
-        expected = 11
-        files = list(recursive_find(self.pwd, pattern="*.dcm"))
-        found = len(files)
-        print("Found %s files" % (len(files)))
+        expected = 3
+        found = len(list(recursive_find(self.pwd, pattern="deid*")))
+        print("Found %s deid files" % (found))
         self.assertTrue(found == expected)
 
 

docs/_docs/examples/client.md

@@ -7,6 +7,15 @@ order: 3
 Here is a quick example of using the deid executable. For more information on this
 client, see the [user docs]({{ site.baseurl }}/user-docs/client/) page.
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
 <a id="deid-executable">
 ### Deid Executable
 The deid executable is installed automatically with the module. Just running `deid` we see:

docs/_docs/examples/func-replace.md

@@ -14,6 +14,15 @@ If you are interested in the functions provided by deid (and you don't want to w
 own function) see [this documentation](https://pydicom.github.io/deid/user-docs/recipe-funcs/).
 Let's get started!
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
 <a id="imports">
 ## Imports
 We first import the functions that we need

docs/_docs/examples/header-expanders.md

@@ -65,6 +65,15 @@ If you want to use the expanders in your code, that's easy too! Here
 are the same examples. Let's first start with reading in a dicom file, 
 such as one of the dicom-cookies examples provided with deid.
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
 <a id="imports">
 ## Imports
 We first import the functions that we need

docs/_docs/examples/recipe.md

@@ -20,6 +20,16 @@ interacting with it, and then using it to replace identifiers. If you want to
 jump in, then go straight to the [script](https://github.com/pydicom/deid/blob/master/examples/dicom/recipe/deid-dicom-example.py) 
 that describes this example.
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
+
 <a id="recipe-management">
 ## Recipe Management
 

docs/_docs/getting-started/dicom-get.md

@@ -4,6 +4,19 @@ category: Getting Started
 order: 5
 ---
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
+
+<a id="get-identifiers">
+## Get Identifiers
+
 A get request using the deid module will return a data structure with headers found in a particular dataset. 
 Let's walk through these steps. As we did in the [loading]({{ site.baseurl }}/getting-started/dicom-loading), 
 the first step was to load a dicom dataset:

docs/_docs/getting-started/dicom-loading.md

@@ -4,6 +4,20 @@ category: Getting Started
 order: 3
 ---
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
+
+<a id="loading">
+## Loading
+
+
 While they are different file organizations for dicom, we are going to take a simple 
 approach of assuming some top level directory with some number of files within 
 (yes, including subdirectories). For example, if you retrieved your data using a 

docs/_docs/getting-started/dicom-pixels.md

@@ -10,13 +10,25 @@ into the data. If you don't want the detalis, jump into our
 [example script](https://github.com/pydicom/deid/blob/master/examples/dicom/pixels/run-cleaner-client.py). 
 Here we will walk through how this cleaner was derived, and how it works.
 
+ - [Data](#data)
  - [Inspiration from CTP](#inspiration-from-ctp)
  - [Deid Implementation](#deid-implementation)
  - [Client](#client) to control the cleaning process
  - [Detect](#detect) areas in the image likely to need cleaning
  - [Clean and Save](#clean-and-save)
  - [Debugging](#debugging) and other important notes
 
+
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
+
 <a id="inspiration-from-ctp">
 ## Inspiration from CTP
 

docs/_docs/getting-started/dicom-put.md

@@ -4,6 +4,15 @@ category: Getting Started
 order: 7
 ---
 
+<a id="data">
+## Data
+
+To run these examples, you'll need to install external deid-data.
+
+```bash
+$ pip install deid-data
+```
+
 At this point, we have a bunch of dicom files, have written a recipe with 
 actions, and want to run those actions across the files. The easiest way
 to do this is with the `DicomParser`