adding linting/formatting with pre-commit (pydicom#239)

* adding linting/formatting with pre-commit
Signed-off-by: vsoch <[email protected]>

.github/dev-requirements.txt

@@ -0,0 +1,4 @@
+pre-commit
+black
+isort
+flake8

.github/workflows/codespell.yaml

@@ -14,5 +14,5 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - uses: crate-ci/typos@592b36d23c62cb378f6097a292bc902ee73f93ef # version 1.0.4
-        with: 
+        with:
           files: ./deid ./docs/_docs ./docs/README.md ./docs/pages ./examples

.github/workflows/docs.yaml

@@ -47,4 +47,3 @@ jobs:
              git commit -a -m "Adding changed documentation files"
              git push origin gh-pages || echo "up to date"
           fi
-

.github/workflows/main.yaml

@@ -16,20 +16,20 @@ jobs:
       - name: Setup black environment
         run: conda create --quiet --name black black
 
-      - name: Check formatting
+      - name: Lint and format Python code
         run: |
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate black
-          pip install black --upgrade
-          black --check --verbose deid
+          pip install -r .github/dev-requirements.txt
+          pre-commit run --all-files
 
   testing:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
 
       - name: Setup conda environment
-        run: | 
+        run: |
           conda create --quiet --name testing
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate testing

.gitignore

@@ -42,4 +42,3 @@ Temporary Items
 
 # PyCharm
 .idea/
-

.pre-commit-config.yaml

@@ -0,0 +1,31 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.3.0
+    hooks:
+      - id: check-added-large-files
+      - id: check-case-conflict
+      - id: check-docstring-first
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: mixed-line-ending
+
+  - repo: local
+    hooks:
+      - id: black
+        name: black
+        language: python
+        types: [python]
+        entry: black
+
+      - id: isort
+        name: isort
+        args: [--filter-files]
+        language: python
+        types: [python]
+        entry: isort
+
+      - id: flake8
+        name: flake8
+        language: python
+        types: [python]
+        entry: flake8

CHANGELOG.md

@@ -15,32 +15,33 @@ Referenced versions in headers are tagged on Github, in parentheses are for pypi
 
 ## [vxx](https://github.com/pydicom/deid/tree/master) (master)
 
+- pre-commit for linting and formatting (0.3.1)
 - Add `ctpcoordinates` and `ctpkeepcoordinates` to handle different formats (0.3.0)
-- Minimum Python required is 3.7, numpy 1.20
+  - Minimum Python required is 3.7, numpy 1.20
 - Remove unecessary typing - adds bugs (0.2.37)
 - Provide data as an external package (0.2.36)
 - Restore expand_sequences to get_identifiers (0.2.35)
 - Add function to clean datasets without `DicomCleaner` [#223](https://github.com/pydicom/deid/pull/223) (0.2.34)
 - add select:vr:XX field expander to select elements by VR (0.2.33)
-- rename group:XXXX field expander to select:group:XXXX
+  - rename group:XXXX field expander to select:group:XXXX
 - add group:XXXX field expander to select all elements with a specified DICOM tag group (0.2.32)
 - custom class example for using dicom.Dataset, not requiring on client init [#211](https://github.com/pydicom/deid/pull/211) (0.2.31)
 - adding support for deid provided functions [#207](https://github.com/pydicom/deid/issues/207) (0.2.3)
 - update CTP deid.dicom up until [this commit](https://github.com/johnperry/CTP/commit/345b05b157c046532e8791a63ababbf6d0dba59b) (0.2.29)
 - various LGTM alert fixes [#186](https://github.com/pydicom/deid/pull/186) (0.0.28)
-- `REPLACE/JITTER` actions have now higher priority than `REMOVE`, allowing to whitelist fields from `REMOVE ALL/Field` [#197](https://github.com/pydicom/deid/issues/197)
-- `ADD/KEEP` actions have now higher priority than `REMOVE`, allowing to whitelist fields from `REMOVE ALL/Field` [#197](https://github.com/pydicom/deid/issues/197)
-- updated pydicom dependency from 2.1.1 to 2.2.2 [#194](https://github.com/pydicom/deid/issues/194)
+  - `REPLACE/JITTER` actions have now higher priority than `REMOVE`, allowing to whitelist fields from `REMOVE ALL/Field` [#197](https://github.com/pydicom/deid/issues/197)
+  - `ADD/KEEP` actions have now higher priority than `REMOVE`, allowing to whitelist fields from `REMOVE ALL/Field` [#197](https://github.com/pydicom/deid/issues/197)
+  - updated pydicom dependency from 2.1.1 to 2.2.2 [#194](https://github.com/pydicom/deid/issues/194)
 - bug fix for exception when attempting to jitter DA/DT which cannot be jittered (space) [#189] (<https://github.com/pydicom/deid/issues/189>) (0.2.27)
 - adding support to manipulate file meta [#183](https://github.com/pydicom/deid/issues/183) (0.2.26)
 - updated pydicom dependency from 1.3.0 to 2.1.1 [#171](https://github.com/pydicom/deid/issues/171) (0.2.25)
-- bug fix for multivalued fields in %values lists [#174](https://github.com/pydicom/deid/issues/174)
-- allowing other VR types for jitter [#175](https://github.com/pydicom/deid/issues/175)
-- ensuring that an add/replace of an existing value is also updated in fields [#173](https://github.com/pydicom/deid/issues/173)
+  - bug fix for multivalued fields in %values lists [#174](https://github.com/pydicom/deid/issues/174)
+  - allowing other VR types for jitter [#175](https://github.com/pydicom/deid/issues/175)
+  - ensuring that an add/replace of an existing value is also updated in fields [#173](https://github.com/pydicom/deid/issues/173)
 - change to correct issue with deidentifying RGB images [#165](https://github.com/pydicom/deid/issues/165) (0.2.24)
 - removing verbosity of debug logger (0.2.23)
 - changing iteration technique through fields to properly add nested uids [#153](https://github.com/pydicom/deid/issues/153) (0.2.22)
-- change to return results from detect when recipe does not contain filters [#155](https://github.com/pydicom/deid/issues/155)
+  - change to return results from detect when recipe does not contain filters [#155](https://github.com/pydicom/deid/issues/155)
 - fix to correct bug in detect [#142](https://github.com/pydicom/deid/issues/142)  (0.2.21)
 - fixes to detect and clean to better represent keep/coordinates (0.2.20)
 - modify default VR for added tags [#146](https://github.com/pydicom/deid/issues/146), bug with private tags in %fields section [#147](https://github.com/pydicom/deid/issues/147) (0.2.19)
@@ -78,14 +79,14 @@ Referenced versions in headers are tagged on Github, in parentheses are for pypi
 - need to clean up temporary directory (mkdtemp), issue #68 (0.1.18)
 - fixing issue #65, save for compressed data (0.1.17)
 - matplotlib must be less than or equal to 2.1.2 for install (0.1.16)
-- fixing bug with clean coordinate flipping rectangle
+  - fixing bug with clean coordinate flipping rectangle
 - Fixing bug with saving self.cleaned (0.1.15)
 - Allowing for datasets to be passed in functions (not necessary for files) (0.1.14)
 - index should be full path in header.py (0.1.13)
 - pydicom bumped to install latest (1.0.2) (0.1.12)
 - ensuring that ids for images are full paths (0.1.11)
-- addition of the DeidRecipe class to better interact with and combine deid recipe files.
-- the get_files function now returns a generator instead of a list.
+  - addition of the DeidRecipe class to better interact with and combine deid recipe files.
+  - the get_files function now returns a generator instead of a list.
 
 ## [0.1.1](https://pypi.python.org/packages/28/26/ee80e7f1c3f65fae1c901497bb2388701158f0c96e0d633ab301abeaa478/deid-0.1.1.tar.gz#md5=39df7efb03e5d3b63308016742062a43) (0.1.1)
 

CONTRIBUTING.md

@@ -24,8 +24,8 @@ This code is licensed under the MIT [LICENSE](LICENSE).
 4. The project's default copyright and header have been included in any new
    source files.
 5. All (major) changes to deid must be documented in
-   [docs](docs). If your PR changes a core functionality, please 
-   include clear description of the changes in your PR so that the docs 
+   [docs](docs). If your PR changes a core functionality, please
+   include clear description of the changes in your PR so that the docs
    can be updated, or better, submit another PR to update the docs directly.
 6. If necessary, update the [README](README.md), and the [CHANGELOG](CHANGELOG.md).
 7. The pull request will be reviewed by others, and the final merge must be
@@ -97,7 +97,7 @@ an incident. Further details of specific enforcement policies may be posted
 separately.
 
 Project maintainers, contributors and users who do not follow or enforce the
-Code of Conduct in good faith may face temporary or permanent repercussions 
+Code of Conduct in good faith may face temporary or permanent repercussions
 with their involvement in the project as determined by the project's leader(s).
 
 ## Attribution

deid/config/utils.py

@@ -135,7 +135,7 @@ def load_deid(path=None):
             parts = line.split(" ")
             if len(parts) > 1:
                 section_name = " ".join(parts[1:])
-            section = re.sub("[%]|(\s+)", "", parts[0]).lower()
+            section = re.sub("[%]|(\s+)", "", parts[0]).lower()  # noqa
             if section not in sections:
                 bot.exit("%s is not a valid section." % section)
 
@@ -225,7 +225,7 @@ def parse_format(line):
     ==========
     line: the line that starts with format.
     """
-    fmt = re.sub("FORMAT|(\s+)", "", line).lower()
+    fmt = re.sub("FORMAT|(\s+)", "", line).lower()  # noqa
     if fmt not in formats:
         bot.exit("%s is not a valid format." % fmt)
     bot.debug("FORMAT set to %s" % fmt)
@@ -386,8 +386,8 @@ def parse_member(members, operator=None):
         member = members.pop(0).strip()
 
         # Find the first || or +
-        match_or = re.search("\|\|", member)
-        match_and = re.search("\+", member)
+        match_or = re.search("\|\|", member)  # noqa
+        match_and = re.search("\+", member)  # noqa
 
         if match_or is not None:
             operator = "||"

deid/data/__init__.py

@@ -2,12 +2,6 @@
 __copyright__ = "Copyright 2016-2022, Vanessa Sochat"
 __license__ = "MIT"
 
-"""
-Simple loading functions for datasets
-
-   from deid.data import get_dataset
-"""
-
 import os
 
 data_base = os.path.abspath(os.path.dirname(__file__))
@@ -23,7 +17,7 @@ def get_dataset(dataset=None):
     """
     try:
         from deid_data import data
-    except:
+    except ImportError:
         raise ValueError("install deid data with `pip install deid-data`")
 
     return data.get_dataset(dataset)

deid/data/deid.dicom

@@ -213,7 +213,7 @@ LABEL Logiq US LOGIQE  # (AMBR)
 LABEL Philips IU22 # (CTP)
   contains Modality US
   + contains Manufacturer Philips
-  + equals Rows 480 
+  + equals Rows 480
   + contains ManufacturerModelName iU22
   ctpcoordinates 0,0,640,47
 
@@ -285,7 +285,7 @@ LABEL EPIQ_7G (CTP)
   ctpcoordinates 0,0,800,59
 
 LABEL Z_ONE # (CTP)
-  contains Modality US 
+  contains Modality US
   + contains Manufacturer Zonare
   + equals Rows 600
   + contains ManufacturerModelName Z_ONE
@@ -316,7 +316,7 @@ LABEL ATL HDI4000 # (FFUR)
 
 LABEL Siemens SC2000 # (CTP)
   contains Modality US
-  + contains Manufacturer Siemens 
+  + contains Manufacturer Siemens
   + equals Rows 768
   + equals Columns 1024
   + contains ManufacturerModelName SC2000
@@ -325,7 +325,7 @@ LABEL Siemens SC2000 # (CTP)
 
 LABEL Siemens Antares # (CTP)
   contains Modality US
-  + contains Manufacturer Siemens 
+  + contains Manufacturer Siemens
   + equals Rows 768
   + equals Columns 1024
   + contains ManufacturerModelName Antares

deid/data/deid.dicom.xray.chest

@@ -2,7 +2,7 @@ FORMAT dicom
 
 %filter whitelist
 
-LABEL Matt Lungren CHEST 
+LABEL Matt Lungren CHEST
   contains Modality CR|DX|PR
   || contains StudyDescription DX|PR
   + contains StudyDescription CHEST || contains BodyPartExamined CHEST

deid/dicom/__init__.py

@@ -1,4 +1,4 @@
 from .fields import extract_sequence
 from .header import get_identifiers, remove_private_identifiers, replace_identifiers
-from .pixels import DicomCleaner, has_burned_pixels, clean_pixel_data
+from .pixels import DicomCleaner, clean_pixel_data, has_burned_pixels
 from .utils import get_files

deid/dicom/config.json

@@ -1,6 +1,6 @@
 {
-   "get": { 
-         
+   "get": {
+
          "skip": ["PixelData",
                   "RedPaletteColorLookupTableData",
                   "GreenPaletteColorLookupTableData",
@@ -20,7 +20,7 @@
 
    "put":{
 
-          "actions":[ 
+          "actions":[
               {"action":"KEEP","field":"PixelData"}
       ]
    }

deid/dicom/groups.py

@@ -2,10 +2,6 @@
 __copyright__ = "Copyright 2016-2022, Vanessa Sochat"
 __license__ = "MIT"
 
-"""
-groups: functions to derive groups of fields or values
-"""
-
 
 from pydicom.multival import MultiValue
 

deid/dicom/header.py

@@ -2,10 +2,6 @@
 __copyright__ = "Copyright 2016-2022, Vanessa Sochat"
 __license__ = "MIT"
 
-"""
-header.py: functions to extract identifiers from dicom headers
-"""
-
 
 import os
 

deid/dicom/pixels/clean.py

@@ -2,9 +2,6 @@
 __copyright__ = "Copyright 2016-2022, Vanessa Sochat"
 __license__ = "MIT"
 
-"""
-clean.py: functions for pixel scrubbing
-"""
 
 import math
 import os
@@ -26,7 +23,7 @@
 
 matplotlib.use("pdf")
 
-from matplotlib import pyplot as plt
+from matplotlib import pyplot as plt  # noqa
 
 bot.level = 3
 

deid/dicom/pixels/detect.py

@@ -3,13 +3,9 @@
 __license__ = "MIT"
 
 
-"""
-detect.py: functions for pixel scrubbing
-"""
+from typing import List, Optional, Union
 
-from typing import Union, List, Optional
-
-from pydicom import read_file, FileDataset
+from pydicom import FileDataset, read_file
 from pydicom.sequence import Sequence
 
 from deid.config import DeidRecipe

deid/dicom/tags.py

@@ -109,7 +109,7 @@ def remove_sequences(dicom):
     dicom: the loaded dicom to remove sequences
     """
     for elem in dicom.iterall():
-        if isinstance(elem.value, Sequence) and dicom.get(elem.tag) != None:
+        if isinstance(elem.value, Sequence) and dicom.get(elem.tag) is not None:
             del dicom[elem.tag]
     return dicom
 

deid/dicom/utils.py

@@ -11,8 +11,8 @@
 
 from deid.logger import bot
 from deid.utils import recursive_find
-from .validate import validate_dicoms
 
+from .validate import validate_dicoms
 
 ################################################################################
 # Functions for Dicom files

deid/logger/message.py

@@ -2,9 +2,6 @@
 __copyright__ = "Copyright 2016-2022, Vanessa Sochat"
 __license__ = "MIT"
 
-"""
-logger/message.py: Python logger base
-"""
 
 import os
 import sys

deid/main/__init__.py

@@ -77,7 +77,7 @@ def get_parser():
         dest="command",
     )
 
-    version = subparsers.add_parser(
+    subparsers.add_parser(
         "version", help="print version and exit"  # pylint: disable=unused-variable
     )
 

deid/tests/Xtest_dicom_header.py

@@ -4,9 +4,6 @@
 __copyright__ = "Copyright 2016-2022, Vanessa Sochat"
 __license__ = "MIT"
 
-"""
-Test dicom header
-"""
 
 import os
 import shutil

deid/tests/resources/filter_multiple_first_filter_match.dicom

@@ -13,4 +13,4 @@ LABEL - To be tested with Cat.dcm.  Intended to NOT flag the image.
   + contains Manufacturer Agfa
 
 %header
-ADD PatientIdentityRemoved No 
+ADD PatientIdentityRemoved No

deid/tests/resources/filter_multiple_rule_innerop_false.dicom

@@ -7,4 +7,4 @@ LABEL - To be tested with Cat.dcm.  Intended to NOT flag the image.
   + contains Manufacturer Agfa
 
 %header
-ADD PatientIdentityRemoved No 
+ADD PatientIdentityRemoved No