Mic-716/Coerce empty strings to np.nan for PO Box addresses #354
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
albrja
requested review from
hussain-jafari,
mattkappel,
ramittal,
rmudambi and
stevebachmeier
as code owners
hussain-jafari
approved these changes
Nov 30, 2023
Base automatically changed from
mic-4671/user-warnings
to
release-candidate/v1.0
December 1, 2023 00:33
|
Can you confirm/check that the columns you've explicitly called out here are the only ones that contain empty strings, at least for a few shards? |
albrja
force-pushed
the
mic-4716/po-box-street-details-nans
branch
from
674d245 to
009f95d
Compare
src/pseudopeople/interface.py
Outdated
| ) -> pd.DataFrame: | ||
| # Coerce dtypes prior to noising to catch issues early as well as | ||
| # get most columns away from dtype 'category' and into 'object' (strings) | ||
| for col in dataset.columns: | ||
| if cleanse_int_cols and col.name in INT_COLUMNS: | ||
| data[col.name] = cleanse_integer_columns(data[col.name]) | ||
| # Coerce emtpy strings to NaNs for mailing address columns that have PO boxes |
stevebachmeier
approved these changes
Dec 1, 2023
rmudambi
pushed a commit
that referenced
this pull request
Jan 19, 2024
Mic-716/Coerce empty strings to np.nan for PO box addresses Coerces null value from empty string to np.nan for addresses with a PO box. - *Category*: Bugfix - *JIRA issue*: [MIC-4716](https://jira.ihme.washington.edu/browse/MIC-4717) -Coerces null value from empty string to np.nan for addresses with a PO box. This issue was present for full USA dataset. Note the new full USA data did not have this bug so I suspect the recent pandas 2.1 update may have fixed this issue. Testing Tested on two different full USA datasets and both will output correct null values for mailing address columns.
albrja
added a commit
that referenced
this pull request
Feb 12, 2024
Mic-716/Coerce empty strings to np.nan for PO box addresses Coerces null value from empty string to np.nan for addresses with a PO box. - *Category*: Bugfix - *JIRA issue*: [MIC-4716](https://jira.ihme.washington.edu/browse/MIC-4717) -Coerces null value from empty string to np.nan for addresses with a PO box. This issue was present for full USA dataset. Note the new full USA data did not have this bug so I suspect the recent pandas 2.1 update may have fixed this issue. Testing Tested on two different full USA datasets and both will output correct null values for mailing address columns.
albrja
added a commit
that referenced
this pull request
Feb 12, 2024
Mic-716/Coerce empty strings to np.nan for PO box addresses Coerces null value from empty string to np.nan for addresses with a PO box. - *Category*: Bugfix - *JIRA issue*: [MIC-4716](https://jira.ihme.washington.edu/browse/MIC-4717) -Coerces null value from empty string to np.nan for addresses with a PO box. This issue was present for full USA dataset. Note the new full USA data did not have this bug so I suspect the recent pandas 2.1 update may have fixed this issue. Testing Tested on two different full USA datasets and both will output correct null values for mailing address columns.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mic-716/Coerce empty strings to np.nan for PO box addresses
Coerces null value from empty string to np.nan for addresses with a PO box.
-Coerces null value from empty string to np.nan for addresses with a PO box. This issue was present for full USA dataset. Note the new full USA data did not have this bug so I suspect the recent pandas 2.1 update may have fixed this issue.
Testing
Tested on two different full USA datasets and both will output correct null values for mailing address columns.