Skip to content

Commit

Permalink
adding example and dockerignore
Browse files Browse the repository at this point in the history
  • Loading branch information
@vsoch
vsoch committed Sep 2, 2017
1 parent f1dda4c commit d71f973
Show file tree
Hide file tree
Showing 4 changed files with 103 additions and 97 deletions.
2 changes: 2 additions & 0 deletions .dockerignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
deid.egg-info
.build
147 changes: 82 additions & 65 deletions docs/getting-started.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -173,39 +173,63 @@ actions:
What we want to do is inspect:

```
deid inspect --help
usage: deid inspect [-h] folder [folder ...]
usage: deid inspect [-h] [--deid DEID] [--save] folder [folder ...]
positional arguments:
folder input folder or single image. If not provided, test data will be
used.
folder input folder or single image. If not provided, test data will
be used.
optional arguments:
-h, --help show this help message and exit
--deid DEID deid file with preferences, if not specified, default
used.
-h, --help show this help message and exit
--deid DEID deid file with preferences, if not specified, default used.
--save, -s save result to output tab separated file.
```

Let's run the command with test data (dicom cookies) and specify the deid in our examples folder:

```
deid inspect --deid examples/deid deid/data/dicom-cookies
SUMMARY ================================
Found 7 valid dicom files
FLAGGED image6.dcm in section dangerouscookie
LABEL: LABEL Criteria for Dangerous Cookie
CRITERIA: PatientSex contains M and OperatorsName notequals bold bread
FLAGGED image5.dcm in section dangerouscookie
LABEL: LABEL Criteria for Dangerous Cookie
CRITERIA: PatientSex contains M and OperatorsName notequals bold bread
CLEAN deid/data/dicom-cookies/image4.dcm
FLAGGED dangerouscookie
deid/data/dicom-cookies/image2.dcm
deid/data/dicom-cookies/image7.dcm
deid/data/dicom-cookies/image6.dcm
deid/data/dicom-cookies/image3.dcm
deid/data/dicom-cookies/image1.dcm
deid/data/dicom-cookies/image5.dcm
SUMMARY ================================
CLEAN 5 files
FLAGGED dangerouscookie 2 files
```
You will see an output, and then a summary of file lists for each of clean and flagged.

If you want to run the above and save the result to file:

```
deid inspect --deid examples/deid deid/data/dicom-cookies --save
...
SUMMARY ================================
CLEAN 5 files
FLAGGED dangerouscookie 2 files
Result written to pixel-flag-results-dicom-cookies-17-09-02.tsv
```

and the file looks like this - images with OperatorsName notequals "bold bread" and PatientSex "M" are flagged:

```
dicom_file pixels_flagged flag_list reason
deid/data/dicom-cookies/image4.dcm,CLEAN
deid/data/dicom-cookies/image2.dcm,CLEAN
deid/data/dicom-cookies/image7.dcm,CLEAN
deid/data/dicom-cookies/image3.dcm,CLEAN
deid/data/dicom-cookies/image1.dcm,CLEAN
deid/data/dicom-cookies/image1.dcm,FLAGGED dangerouscookie PatientSex contains M and OperatorsName notequals bold bread
deid/data/dicom-cookies/image1.dcm,FLAGGED dangerouscookie PatientSex contains M and OperatorsName notequals bold bread
```

### Within Python
First, let's load the example "dicom cookies" dataset. We will first run this example within python, and then using a command line client (not written yet).

Expand Down Expand Up @@ -252,7 +276,6 @@ FORMAT dicom
LABEL Criteria for Dangerous Cookie
contains PatientSex M
+ missing PatientBirthDate
+ notequals OperatorsName bold bread
coordinates 0,0,512,110
Expand All @@ -273,7 +296,7 @@ REPLACE SOPInstanceUID var:source_id

We won't be using the header section for this example, but for your FYI, this is the recipe for how we would want to replace information in the header, if we were cleaning the headers. Right now we are just filtering images to flag those that might have PHI. Let's very strictly walk through the logic that will be taken above:

1. If the header contains field PatientSex "M" (Male), and is missing PatientBirthDate and OperatorsName is not "bold bread," we flag. Otherwise, keep going.
1. If the header contains field PatientSex "M" (Male), and OperatorsName is not "bold bread," we flag. Otherwise, keep going.
2. If the header has field Rows 2048 and Columns 1536 we flag.

The flag that is done first (more specific) is the final decision. This means that you should have your known coordinates of PHI (eg, specific modality, manufacturer, etc) first, and followed by more general estimates of PHI. Likely a later group will create flags for more manual inspection.
Expand All @@ -285,61 +308,55 @@ from deid.dicom import has_burned_pixels
groups = has_burned_pixels(dicom_files=dicom_files, deid='examples/deid')
```

We immediately see that six are flagged for not having OperatorsName "bold bread"
We immediately see that two are flagged:

```
FLAGGED image2.dcm in section dangerouscookie
LABEL: criteria for dangerous cookie
CRITERIA: and OperatorsName notequals bold bread
FLAGGED image7.dcm in section dangerouscookie
LABEL: criteria for dangerous cookie
CRITERIA: and OperatorsName notequals bold bread
FLAGGED image6.dcm in section dangerouscookie
LABEL: criteria for dangerous cookie
CRITERIA: and OperatorsName notequals bold bread
FLAGGED image3.dcm in section dangerouscookie
LABEL: criteria for dangerous cookie
CRITERIA: and OperatorsName notequals bold bread
FLAGGED image1.dcm in section dangerouscookie
LABEL: criteria for dangerous cookie
CRITERIA: and OperatorsName notequals bold bread
LABEL: LABEL Criteria for Dangerous Cookie
CRITERIA: PatientSex contains M and OperatorsName notequals bold bread
FLAGGED image5.dcm in section dangerouscookie
LABEL: criteria for dangerous cookie
CRITERIA: and OperatorsName notequals bold bread
LABEL: LABEL Criteria for Dangerous Cookie
CRITERIA: PatientSex contains M and OperatorsName notequals bold bread
```

Is this accurate?

```
for dicom_file in dicom_files:
dicom = read_file(dicom_file)
print("%s:%s" %(os.path.basename(dicom_file),dicom.OperatorsName))
image4.dcm:bold bread
image2.dcm:lingering hill
image7.dcm:sweet brook
image6.dcm:green paper
image3.dcm:nameless voice
image1.dcm:fragrant pond
image5.dcm:curly darkness
print("%s:%s - %s" %(os.path.basename(dicom_file),
dicom.OperatorsName,
dicom.PatientSex))
image4.dcm:bold bread - M
image2.dcm:lingering hill - F
image7.dcm:sweet brook - F
image6.dcm:green paper - M <--- FLAGGED
image3.dcm:nameless voice - F
image1.dcm:fragrant pond - F
image5.dcm:curly darkness - M <--- FLAGGED
```

Seems to be! The data structure returned gives us programmatic access to the groups, including list of clean (top), list of flagged and flag list name (flagged) and given flagged, a lookup dictionary with reasons:

```
{
"clean":[
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image4.dcm",
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image2.dcm",
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image7.dcm",
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image3.dcm",
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image1.dcm"
],
"flagged":{
"dangerouscookie":[
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image6.dcm",
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image5.dcm"
]
},
"reason":{
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image5.dcm":" PatientSex contains M and OperatorsName notequals bold bread",
"/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image6.dcm":" PatientSex contains M and OperatorsName notequals bold bread"
}
}
```

Seems to be! The data structure returned gives us programmatic access to the groups:

```
{'clean': ['/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image4.dcm'],
'flagged': {'dangerouscookie': ['/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image2.dcm',
'/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image7.dcm',
'/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image6.dcm',
'/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image3.dcm',
'/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image1.dcm',
'/home/vanessa/Documents/Dropbox/Code/dicom/deid/deid/data/dicom-cookies/image5.dcm']}}
```

More examples (and client) coming soon, need to write tests, etc. first!
1 change: 0 additions & 1 deletion examples/deid/deid.dicom
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ FORMAT dicom

LABEL Criteria for Dangerous Cookie
contains PatientSex M
+ missing PatientBirthDate
+ notequals OperatorsName bold bread
coordinates 0,0,512,110

Expand Down
50 changes: 19 additions & 31 deletions examples/dicom/deid-dicom-inspect-pixels.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,49 +11,37 @@
get_files,
has_burned_pixels
)
from pydicom import read_file
from deid.data import get_dataset
from deid.logger import bot
import os
bot.level = 3

base = get_dataset('dicom-cookies')
dicom_files = get_files(base)

groups = has_burned_pixels(dicom_files=dicom_files, deid='examples/deid')

# FLAGGED image2.dcm in section dangerouscookie
# LABEL: criteria for dangerous cookie
# CRITERIA: and OperatorsName notequals bold bread

# FLAGGED image7.dcm in section dangerouscookie
# LABEL: criteria for dangerous cookie
# CRITERIA: and OperatorsName notequals bold bread

# Found 7 valid dicom files
# FLAGGED image6.dcm in section dangerouscookie
# LABEL: criteria for dangerous cookie
# CRITERIA: and OperatorsName notequals bold bread

# FLAGGED image3.dcm in section dangerouscookie
# LABEL: criteria for dangerous cookie
# CRITERIA: and OperatorsName notequals bold bread

# FLAGGED image1.dcm in section dangerouscookie
# LABEL: criteria for dangerous cookie
# CRITERIA: and OperatorsName notequals bold bread

# LABEL: LABEL Criteria for Dangerous Cookie
# CRITERIA: PatientSex contains M and OperatorsName notequals bold bread
# FLAGGED image5.dcm in section dangerouscookie
# LABEL: criteria for dangerous cookie
# CRITERIA: and OperatorsName notequals bold bread
# LABEL: LABEL Criteria for Dangerous Cookie
# CRITERIA: PatientSex contains M and OperatorsName notequals bold bread

# We see above six are flagged because the operator's name isn't bold bread.
# We see above two are flagged for OperatorsName not bold bread, and being male
# Is this accurate?
for dicom_file in dicom_files:
dicom = read_file(dicom_file)
print("%s:%s" %(os.path.basename(dicom_file),dicom.OperatorsName))

#image4.dcm:bold bread
#image2.dcm:lingering hill
#image7.dcm:sweet brook
#image6.dcm:green paper
#image3.dcm:nameless voice
#image1.dcm:fragrant pond
#image5.dcm:curly darkness
print("%s:%s - %s" %(os.path.basename(dicom_file),
dicom.OperatorsName,
dicom.PatientSex))

# image4.dcm:bold bread - M
# image2.dcm:lingering hill - F
# image7.dcm:sweet brook - F
# image6.dcm:green paper - M <--- FLAGGED
# image3.dcm:nameless voice - F
# image1.dcm:fragrant pond - F
# image5.dcm:curly darkness - M <--- FLAGGED

0 comments on commit d71f973

Please sign in to comment.