Support of Nan

[lionelkusch]

There will be a need to test how to handle Nan in the methods and to add tests associated with it.

[jpaillard]

For LOCO and PermutationImportance, I think that the handling of NaNs values can be left to the sklearn pipeline (using a model that supports them natively or including an imputation step). However, for CPI, I think we need further work inside the library as the main predictive model should support NaNs but also the estimator of the conditional distribution, $p(X^j | X^{-j})$.

Not sure, but this might also apply to knockoffs?

[lionelkusch]

Do you know if Lasso and RandomForest can handle natively NaNs values?
I ask the question because dcrt are based on this estimator.

[jpaillard]

RandomForest yes, Lasso no

[bthirion]

For LOCO and PermutationImportance, I think that the handling of NaNs values can be left to the sklearn pipeline (using a model that supports them natively or including an imputation step). However, for CPI, I think we need further work inside the library as the main predictive model should support NaNs but also the estimator of the conditional distribution, p ( X j | X − j ) .

Not sure, but this might also apply to knockoffs?

But in most cases, the end estimator will be an sklearn one. If we can defer the handling of NaN to it, it would be ideal.

[lionelkusch]

The problem is not for the estimator but for the method itself.
In the case of CPI, the imputation model needs to deal with NaN which is not trivial and for knockoff, the estimation of the distribution requires supporting NaN.
Moreover, in dcrt0, we use Lasso to distillate the variable of interest and which doesn't support NaN.
In consequence, we need to decide a general politic of dealing with NaN (not supported or partially supported or fully supported).

[bthirion]

My best suggestion is to take a look at https://skrub-data.org
which is a sklearn extension for this type of data.
In a first instance, I think its is acceptable not to support NaNs. Why not defer to the user to perform cleaning on the data prior to importance analysis ?
i.e. the user would implement one of the following strategies:

• ignore features with NaN values
• ignore samples with NaN values
• handle NaNs as a special value (makes sense for categorical data only)
• impute NaN values

[lionelkusch]

It was my suggestion to @jpaillard but he told me that NaN contains information by itself for the estimation and they are important to keep it.

[bthirion]

Only when you work with categorical variables AFAIK

[jpaillard]

I would argue for the two last options: for categorical data, consider NaNs as a special value, and for continuous impute NaN values.

Why not defer to the user to perform cleaning on the data prior to importance analysis?

The user might want to use a main predictive model/pipeline that handles NaN values (i.e., RandomForestRegressor) but a conditional estimator, $\nu_j(X^{-j}) = p(X^j | X^{-j})$ that does not (i.e. RidgeCV).
While I agree that for the main predictive model, the strategy for handling NaNs should be the user's responsibility, I think it makes sense to provide a default & customizable strategy for handling NaNs in the conditional sampling step.

[bthirion]

But then, the problem should rather be addressed from the beginning, before using hidimstat, no ?

[jpaillard]

I was specifically thinking about the case where the predictive model is HistGradientBoosting for which NaN values are handled even for continuous variables.

This estimator has native support for missing values (NaNs). During training, the tree grower learns at each split point whether samples with missing values should go to the left or right child, based on the potential gain.

This may be too specific, and we should leave the management of NaNs to the user.