Bug/issue324 #297
Merged
Bug/issue324 #297
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The way it was done resulted in a subtle bug where sometimes the target levels where not assigned consistently on different nodes. This is now fixed: The user inputs the levels he wishes to assign to positive and negative outcomes. This also eliminates the need to use filters to handle multilevel variables. Only two levels are always filter from within the algorithm now (method: keep_levels).
Also, correctly escape negative_level and positive_level params.
The Logistic Regression algorithm now accepts input from user for positive and negative levels, instead of assign them automatically. Thus the test inputs must include those params instead of a random filter for two values.
Problem: -------- In some cases all levels of a categorical variable are not present in every node. This led to exceptions since the corresponding design matrices schemata were not aligned across local nodes. Solution: --------- Add method for aligning categorical variables across local nodes. Checks all categorical columns in design matrices produced by patsy and if any level is missing in some local node it adds the corresponding column (all zeros since the level was missing originally).
ThanKarab
approved these changes
Jan 5, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bug fix (issue324):
Problem:
The initialization procedure in LogisticRegression sometimes assigns y values (0 and 1) to different levels of the
target variable across local nodes. The code which assigns y levels in [0, 1] is
y = self.data.variables.iloc[:, 1]which is not guarantied to lead to the same values, hence the bug.
Solution:
The user inputs the levels he wishes to assign to
positive and negative outcomes. This also eliminates the need to use
filters to handle multilevel variables. Only two levels are always
filter from within the algorithm now (method: keep_levels).
Bug fix:
Problem:
In some cases all levels of a categorical variable are not present in
every node. This led to exceptions since the corresponding design
matrices schemata were not aligned across local nodes.
Solution:
Add method for aligning categorical variables across local nodes. Checks
all categorical columns in design matrices produced by patsy and if any
level is missing in some local node it adds the corresponding column
(all zeros since the level was missing originally).
Caveat:
Adding non-existing levels to a local nodes, without affecting the record count is only possible in
design matrices with dummy coding, i.e. in
variablesandcovariablesmatrices. This is done simplyby adding all-zero columns for the missing levels. Warning: in the
fullmatrix which in not dummy codedthe missing levels are still missing!
Enhancement: Add more nodes to LogisticRegression and CalibrationBelt tests.