Feature Request: Crossed random effects for multilevel data

[jmgirard]

In tidymodels/TMwR#288, I suggested adding the feature of having {tidyposterior} detect multilevel data (e.g., from rsample::group_vfold_cv()) and add random effects for the grouping/cluster variable to the model comparison analysis. This would usually be a cross-classified model with random effects (e.g., intercepts) for both resample and group. Accounting for this extra level of dependency in the data is important to properly estimating uncertainty.

I will work on posting some example code below (but am running low on time today).

[topepo]

Can you provide an example? Without our resampling tools, I'm not sure how the resampling statistics will have a hierarchical structure to them.

There is a formula argument where you can pass something custom.

[jmgirard]

The two main examples would be longitudinal and clustered data. If you have multiple observations per person and are trying to make observation-level predictions, then your estimates of resampling statistic uncertainty will be biased to the extent that predictive performance is more similar within persons (e.g., some people are "easier" to predict than others). Similarly, if you have multiple students from the same classroom and are trying to make student-level predictions, then your estimates of resampling statistic uncertainty will be biased to the extent that predictive performance is more similar within classrooms (e.g., some classrooms are "easier" to predict than others).