Merge pull request #31 from eurodecision/dev

v0.0.2
eurodecision · Oct 17, 2022 · 6acea99 · 6acea99
2 parents c94ef2d + 2d9eff9
commit 6acea99
Show file tree

Hide file tree

Showing 115 changed files with 1,622 additions and 5,375 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -1,2 +1,2 @@
 [run]
-omit=pdcleaner/tests/*
+omit=tests/*
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -31,10 +31,30 @@ jobs:
         # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 
+    - name: Install python package
+      run: |
+        python setup.py develop
+
     - name: Test with pytest
       run: |
         pytest
 
+    - name: Get Coverage for badge
+      run: |
+        SUMMARY="$(pytest --cov --cov-report term-missing | tail -3 | head -1)"
+        TOKENS=($SUMMARY)
+        echo "COVERAGE=$(echo ${TOKENS[-1]})" >> $GITHUB_ENV
+
+    - name: Create the Badge
+      uses: schneegans/[email protected]
+      with:
+        auth: ${{ secrets.GIST_TOKEN }}
+        gistID: 74e11571ef9b1a732e44fe9fbbdc7f0b
+        filename: pdcleaner_coverage.json
+        label: Test Coverage
+        message: ${{ env.COVERAGE }}
+        color: green
+
     - name: Build python package
       run: |
         python setup.py sdist bdist_wheel

diff --git a/.gitignore b/.gitignore
@@ -2,6 +2,7 @@
 .vscode/
 .ipynb_checkpoints/
 .idea/
+.pytest_cache/
 
 #compiled sources
 *.py[ocd]

diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
@@ -0,0 +1,128 @@
+
+# Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[email protected].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq]. Translations are available at
+[https://www.contributor-covenant.org/translations].
+
diff --git a/README.md b/README.md
@@ -0,0 +1,88 @@
+# Pandas-cleaner
+
+<img src=https://img.shields.io/pypi/v/pandas-cleaner.svg target=https://pypi.python.org/pypi/pandas-cleaner> <img src=https://img.shields.io/pypi/l/pandas-cleaner.svg target=https://pypi.python.org/pypi/pandas-cleaner> <img src="https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/ed-ialifinaritra/74e11571ef9b1a732e44fe9fbbdc7f0b/raw/pdcleaner_coverage.json">
+
+Pandas-cleaner is a Python package, built on top of pandas, that provides methods detect, analyze and clean errors in datasets with different types of data (numerical, categorical, text, datetimes...).
+
+![](https://github.com/eurodecision/pandas-cleaner/blob/master/docs/source/pandas-cleaner.png)
+
+## Features
+
+Pandas-cleaner offers functionnalities to automatically :
+
+:arrow_right: **detect** different kind of potential errors in datasets such as outliers, inconsistencies, typos, wrong-typed ..., given predefined rules or statistiscal estimations,  via an easy-to-use API extending pandas,
+
+:arrow_right: **analyze** these errors, via reports and plots, to check the validity of the set and/or decide if any correction is needed,
+
+:arrow_right: **clean** the datasets, either by dropping the lines with errors, emptying, correcting or replacing bad values,
+
+:arrow_right: **reapply** the same rules to any other incoming fresh data.
+
+## Usage
+
+Import the package
+
+```python
+import pandas as pd
+import pdcleaner
+```
+
+Create an example data series
+
+```python
+series = pd.Series([1, 5, -6, 100, 10])
+```
+
+Detect the errors in the series with a given method (such as `bounded`, `iqr`, `zscore` and many more depending the type of data...)
+
+```python
+detector = series.cleaner.detect('bounded', lower=0, upper=10)
+```
+
+Inspect the result:
+
+```python
+detector.report()
+```
+```none
+                                 Detection report                               
+==============================================================================
+Method:                      bounded      Nb samples:                        5
+Date:                January 24,2022      Nb errors:                         2
+Time:                       16:06:08      Nb rows with NaN:                  0
+------------------------------------------------------------------------------
+lower                              0      upper                             10
+inclusive                       both      sided                           both
+==============================================================================
+```
+
+Check the potential errors that have been detected
+
+```python
+detector.detected()
+```
+```
+ 2     -6
+ 3    100
+ dtype: int64
+```
+
+Clean the detected errors from the series using the chosen method among `drop`, `to_na`, `clip`
+, `replace`...
+
+```python
+series.cleaner.clean("drop", detector, inplace=True)
+   series
+```
+```
+ 0      1
+ 1      5
+ 4     10
+ dtype: int64
+```
+
+## Contributing to pandas-cleaner
+
+All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
+
+Issues and bugs can be reported at https://github.com/eurodecision/pandas-cleaner/issues
diff --git a/RELEASE.md b/RELEASE.md
@@ -1,4 +1,34 @@
-## Version 0.0.1
+### What's new in 0.0.2 (October 17, 2022)
+
++ **New detection methods**
+    + `'value'` to detect cells different from a given value
+    + ``'custom'`` to apply an user-defined python callable to detect the errors
+    + ``'spaces'`` to detect extra spaces before or after strings
+    + ``'duplicated'`` to detect duplicated records (whole row or subset of columns)
+    + ``'date_range'`` to detect dates outside a given time range
++ **New cleaning method**
+    + ``'cast'`` associated to the detector ``castable`` to cast strings into floats, integers, dates or NaNs
++ **Enhancements**
+    + The detectors have two new useful methods `.detected()` and `.valid()` that return valid/unvalid rows
+    + Gaussian detectors (``iqr``, ``zscore``, ``modzscore``) now have a systematic normality test and an option to transform the distribution
+      (Box-Cox, Yeo-Johnson) before applying the detector
+    + Reports are more detailed when using a multivariate qualitative/ quantitative detector
+    + Detector ``castable`` now applies to booleans
+    + Detector ``values`` is now called ``enum``
+    + Detector ``keycollision`` 
+        + is now called ``alternatives`` 
+        + is much faster
+        + now offers the choice to keep either most frequent representation or the first encountered
+    + Plot method for ``freq`` and ``cat`` detectors now have options ``nfirst`` and ``nlast``
+    + API for detector ``length`` has been simplified
++ **Bug fixes**
+    + Detector ``pattern`` now works with compiled regex
+    + Cleaning methods applied to DataFrames have been fixed
+    + Booleans are better displayed in reports
+    + Fixed bug with ``association`` detector plot method
++  **Code**
+    + Detector classes now have much simplier names
+    + ``tests`` folder has been reorganized  to mimic ``src`` structure
 
 ### What's new in 0.0.1 (February 25, 2022)
 

diff --git a/README.rst → _README.rst b/README.rst → _README.rst
@@ -6,6 +6,14 @@
 pandas-cleaner
 ==============
 
+.. image:: https://img.shields.io/pypi/v/pandas-cleaner.svg
+        :target: https://pypi.python.org/pypi/pandas-cleaner
+
+.. image:: https://img.shields.io/pypi/l/pandas-cleaner.svg
+        :target: https://pypi.python.org/pypi/pandas-cleaner
+
+.. image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/ed-ialifinaritra/74e11571ef9b1a732e44fe9fbbdc7f0b/raw/pdcleaner_coverage.json
+
 
 What is it ?
 ------------
@@ -70,7 +78,7 @@ Check the potential errors that have been detected
 
 .. code-block:: python
 
-   detector.detected
+   detector.detected()
 
 .. parsed-literal::
 
@@ -93,21 +101,6 @@ Clean the detected errors from the series using the chosen method among `drop`,
     4     10
     dtype: int64
 
-Documentation
--------------
-
-The documentation is still a **work in progress**. 
-
-* Clone the project
-
-* Build the documentation using :
-
-.. code-block:: bash
-
-    cd docs
-    make html
-
-* Open `docs/build/html/index.html` in your browser
 
 Contributing to pandas-cleaner
 ------------------------------

diff --git a/docs/source/_static/favicon.png b/docs/source/_static/favicon.png
diff --git a/docs/source/_static/logo_ed_star.png b/docs/source/_static/logo_ed_star.png
diff --git a/docs/source/_static/pandas-cleaner-dark.png b/docs/source/_static/pandas-cleaner-dark.png
diff --git a/docs/source/_static/pandas-cleaner.png b/docs/source/_static/pandas-cleaner.png
diff --git a/docs/source/_static/plot_alternatives_1.png b/docs/source/_static/plot_alternatives_1.png
diff --git a/docs/source/_static/plot_alternatives_nfirst_2.png b/docs/source/_static/plot_alternatives_nfirst_2.png
diff --git a/docs/source/_static/plot_alternatives_nlast_1.png b/docs/source/_static/plot_alternatives_nlast_1.png
diff --git a/docs/source/_static/plot_association.png b/docs/source/_static/plot_association.png
diff --git a/docs/source/_static/plot_freq.png b/docs/source/_static/plot_freq.png
diff --git a/docs/source/_static/plot_freq_nfirst_nlast.png b/docs/source/_static/plot_freq_nfirst_nlast.png
diff --git a/docs/source/_static/plot_keycollision_1.png b/docs/source/_static/plot_keycollision_1.png
diff --git a/docs/source/_static/plot_keycollision_nfirst_2.png b/docs/source/_static/plot_keycollision_nfirst_2.png
diff --git a/docs/source/_static/plot_keycollision_nlast_1.png b/docs/source/_static/plot_keycollision_nlast_1.png
diff --git a/docs/source/_static/plot_numseries copy.png b/docs/source/_static/plot_numseries copy.png
diff --git a/docs/source/_static/plot_numseries.png b/docs/source/_static/plot_numseries.png
diff --git a/docs/source/_templates/logo.html b/docs/source/_templates/logo.html
@@ -1,7 +1,7 @@
 <div class="sidebar-end-items">
-<a href="https://www.eurodecision.eu/" target="_blank">
+<a href="https://www.eurodecision.com/" target="_blank">
 
-<img src="https://www.eurodecision.com/wp-content/themes/eurodecision/img/logo-eurod.jpg" width=250>
+<img src="https://www.eurodecision.com/wp-content/themes/eurodecision/img/logo-eurod.jpg" width=125>
 </a>
 </div>