0.0.16 release.

Type and line length fixes. Added examples along with api to render to string. Support nested directory structures when rendering a directory. flake and mypy changes. Update github action to use poetry run. Added filter tests. Add logo, command line usage, key features, current limitations, benchmark and other items to legitmize the readme. Support flatten, cleanup empty dirs and update examples.
chaseastewart · Jan 11, 2024 · 902ae01 · 902ae01
1 parent c2043a5
commit 902ae01
Show file tree

Hide file tree

Showing 17 changed files with 1,293 additions and 208 deletions.
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
@@ -10,7 +10,7 @@ on:
     branches: [ "main" ]
 
 jobs:
-  linting:
+  linting-check:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -22,19 +22,48 @@ jobs:
       uses: actions/setup-python@v3
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        python -m pip install flake8
-    - name: Lint with flake8
+    - name: Install Poetry
+      uses: snok/install-poetry@v1
+      with:
+        virtualenvs-create: true
+        virtualenvs-in-project: true
+        installer-parallel: true
+    - name: Install project
+      run: poetry install --no-interaction
+    - name: flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+        poetry run flake8 fhir_converter --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 90 chars wide
+        poetry run flake8 fhir_converter --count --ignore=E203,E721 --exit-zero --max-complexity=10 --max-line-length=90 --statistics
+
+  type-check:
+    needs: linting-check
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.12"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install Poetry
+      uses: snok/install-poetry@v1
+      with:
+        virtualenvs-create: true
+        virtualenvs-in-project: true
+        installer-parallel: true
+    - name: Install project
+      run: poetry install --no-interaction
+    - name: mypy
+      run: |
+        poetry run mypy fhir_converter
 
   test:
-    needs: linting
+    needs: type-check
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -58,5 +87,4 @@ jobs:
       run: poetry install --no-interaction
     - name: Run tests
       run: |
-        poetry run python --version
-        poetry run python -m unittest discover
+        poetry run pytest
diff --git a/.gitignore b/.gitignore
@@ -27,4 +27,11 @@ __pycache__/
 *.py[cod]
 
 # mypy
-.mypy_cache/
+.mypy_cache/
+
+#pytest
+.coverage*
+coverage.xml
+junit/
+htmlcov/
+.pytest_cache
diff --git a/README.md b/README.md
@@ -1,14 +1,54 @@
-# python-fhir-converter
+<p align="center">
+  <img src="./logo.png" width="360" />
+</p>
+<p align="center">
+    <em>Python FHIR converter, fastish, most nuts and bolts included, ready for production</em>
+</p>
+<p align="center">
+<a href="https://github.com/jg-rp/liquid/blob/main/LICENSE" target="_blank">
+  <img src="https://img.shields.io/pypi/l/python-liquid.svg?style=flat-square" alt="License">
+</a>
+<a href="https://pypi.org/project/python-fhir-converter/" target="_blank">
+  <img src="https://img.shields.io/pypi/v/python-fhir-converter.svg?style=flat-square" alt="PyPi - Version">
+</a>
+<a href="https://pypi.org/project/python-fhir-converter" target="_blank">
+  <img src="https://img.shields.io/pypi/pyversions/python-fhir-converter.svg?style=flat-square" alt="Python versions">
+</a>
+<a href="https://black.readthedocs.io/en/stable/index.html" target="_blank">
+  <img src="https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square" />
+</a>
+</p>
+
+---
+
+Provides a python native version of [FHIR-Converter](https://github.com/microsoft/FHIR-Converter) written in C#.
+
+The key features are:
+
+* **Fastish**: Leverages Cython where possible 
+* **Move fast**: Designed to be extensibile. Use the thin rendering API or leverage the builtin parts
+* **Easy**: Designed to be easy to use, extend and deploy.
+* **Robust**: Get production-ready code.
+
+Limitations:
+* **Only CDA->FHIR** is currently builtin. Additional work is needed to implement the filters, etc to support FHIR->FHIR and HL7v2->FHIR and back.
+* **Python-liquid requires** a comma between parameters to filters. This does not appear to be a restriction with DotLiquid. As a result templates brought to this environment may need commas added.
+
+Built on the back of:
+
+* [FHIR-Converter](https://github.com/microsoft/FHIR-Converter)
+* [python-liquid](https://github.com/jg-rp/liquid)
 
-Provides a python native version of the [reference implementation](https://github.com/microsoft/FHIR-Converter) written in C#. The goal is to generate 1:1 output for the same [templates](https://github.com/microsoft/FHIR-Converter/tree/main/data/Templates). Built on the back of [python-liquid](https://github.com/jg-rp/liquid)
 
 **Table of Contents**
 
 - [Install](#install)
 - [Links](#links)
-- [Example](#example)
+- [Basic Usage](#basic-usage)
+- [Command line interface](#command-line-interface)
 - [Related Projects](#related-projects)
 
+
 ## Install
 
 Install Python FHIR Converter using [Pipenv](https://pipenv.pypa.io/en/latest/):
@@ -29,16 +69,103 @@ $ pip install python-fhir-converter
 - Source Code: https://github.com/chaseastewart/fhir-converter
 - Issue Tracker: https://github.com/chaseastewart/fhir-converter/issues
 
-## Example
+
+## Basic Usage
+Source can be found [here](./scripts/examples.py)
 
 ```python
-from fhir_converter.processors import CcdaProcessor
+from functools import partial
+from pathlib import Path
+
+from fhir_converter.loaders import get_file_system_loader
+from fhir_converter.renderers import (
+    CcdaRenderer,
+    get_environment,
+    render_files_to_dir,
+    render_to_dir,
+)
+from fhir_converter.utils import mkdir_if_not_exists
+
+templates_dir, sample_data_dir, data_out_dir = (
+    Path("data/templates/ccda"),
+    Path("data/sample/ccda"),
+    Path("data/out"),
+)
+
+from_file = sample_data_dir.joinpath("CCD.ccda")
+mkdir_if_not_exists(data_out_dir)
+
+# Render the file to string using the rendering defaults indenting the output
+with open(from_file) as xml_in:
+    # indent is provided, any other kwargs supported by dump may be provided
+    print(CcdaRenderer().render_fhir_string("CCD", xml_in, indent=1))
 
-processor = CcdaProcessor.from_template_dir(TEMPLATE_DIR)
-with open("data/sample/ccda/Discharge_Summary.ccda") as ccda_file:
-    print(processor.convert(template_name="DischargeSummary", xml_input=ccda_file))
+# Create a renderer that will load the user defined templates into the rendering env
+renderer = CcdaRenderer(
+    get_environment(loader=get_file_system_loader(search_path=templates_dir))
+)
+
+# Render the file to the output directory using the default CCD template
+render_to_dir(
+    render=partial(renderer.render_fhir, "CCD"),
+    from_file=from_file,
+    to_dir=data_out_dir,
+)
+
+# Render all of the sample files to the output directory using the user defined
+# pampi (problems, allergies, meds, procedures, immunizations) template
+render_files_to_dir(
+    render=partial(renderer.render_fhir, "pampi"),
+    from_dir=sample_data_dir,
+    to_dir=data_out_dir,
+    path_filter=lambda p: p.suffix in (".ccda", ".xml"),
+)
+```
+
+## Command line interface
+
+The package comes with a CLI interface that can be invoked either by the script name
+``fhir_converter_cli`` or as python module ``python -m fhir_converter``. The CLI allows you to transform a single file or an entire directory.
+
+```bash
+fhir_converter_cli  --from-file  ./data/sample/ccda/CCD.ccda --to-dir ./data/out --template-name CCD
+---------------------------------------------------------------
+RENDER SUCCESS
+---------------------------------------------------------------
+Total time: 0.14s
+Finished at: 2024-01-11 10:49:44.182033
+Final Memory: 32M
+---------------------------------------------------------------
+
+fhir-converter % fhir_converter_cli --template-dir ./data/templates/ccda --from-dir ./data/sample/ccda --to-dir ./data/out --template-name pampi
+---------------------------------------------------------------
+RENDER SUCCESS
+---------------------------------------------------------------
+Total time: 0.32s
+Finished at: 2024-01-11 10:49:44.182033
+Final Memory: 37M
+---------------------------------------------------------------
+```
+
+## Benchmark
+
+You can run the [benchmark](./scripts/benchmark.py) from the root of the source tree. Test rig is a 14-inch, 2021 Macbook Pro with the binned M1 PRO not in low power mode.
+```text
+   Ordered by: cumulative time
+
+   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
+        3    0.000    0.000   16.998    5.666 ../scripts/benchmark.py:75(render_samples)
+       22    0.003    0.000   16.997    0.773 ../fhir-converter/fhir_converter/renderers.py:187(render_files_to_dir)
+      484    0.002    0.000   16.968    0.035 ../fhir-converter/fhir_converter/renderers.py:220(render_to_dir)
+      484    0.010    0.000   16.842    0.035 ../fhir-converter/fhir_converter/renderers.py:93(render_fhir)
+      484    0.003    0.000   14.674    0.030 ../fhir-converter/fhir_converter/renderers.py:117(render_to_fhir)
 ```
+The test fixture profiles the converter using a single thread. The samples are rendered using all of the builtin templates along with the handful of user defined templates. The percall time is relative to the rendering template being used, the number of files being rendered (there is some warm up) and the size of the files to be rendered. In a 60 minute period in similar conditions a little over 100K CDA documents could be rendered into FHIR bundles. Note: including the original CDA document in the bundle as a DocumentReference adds noticable overhead to the render. Omitting this via a user defined template is recommended if this is not required for your usecase.
+
 
 ## Related Projects
 
-- [python-liquid](https://github.com/jg-rp/liquid) Python engine for the Liquid template language.
+- [FHIR-Converter](https://github.com/microsoft/FHIR-Converter)
+- [python-liquid](https://github.com/jg-rp/liquid)
+- [pyjson5](https://github.com/Kijewski/pyjson5)
+- [xmltodict](https://github.com/martinblech/xmltodict)
diff --git a/data/sample/nested/bad_data/empty.ccda b/data/sample/nested/bad_data/empty.ccda
diff --git a/data/sample/nested/bad_data/empty.xml b/data/sample/nested/bad_data/empty.xml