Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a toggle to runMultiplePlp to skip diagnosis #392

Open
egillax opened this issue Apr 26, 2023 · 3 comments
Open

Add a toggle to runMultiplePlp to skip diagnosis #392

egillax opened this issue Apr 26, 2023 · 3 comments

Comments

@egillax
Copy link
Collaborator

egillax commented Apr 26, 2023

I would prefer to be able to skip the diagnosis which is run when developing models using runMultiplePlp. It's taking a non-trivial amount of time and I've never looked at the output, although maybe I would for a clinical study. There are also certain inefficiencies how it's done. For example if you have 2 predictions problems and 3 classifiers. It will run the diagnosis 6 times (2x3). But as far as I can see you only need to run it once per prediction problem.

I'd like to have a toggle to turn it off. And would prefer actually the default for it to be off.

What do you think @jreps. I've also discussed this with @lhjohn and I believe he agrees.

@jreps
Copy link
Collaborator

jreps commented Apr 26, 2023

I was thinking yesterday we can make it more optimized (e.g., if the model design is the same except for the model then they can share diagnostics currently but that may change if we add classifier specific diagnostics) but I think it should be there by default as it can help identify issues with the prediction design and all HADES analyses have diagnostics to help identify issues.

@egillax
Copy link
Collaborator Author

egillax commented Apr 26, 2023

What issues will it identify?

@egillax
Copy link
Collaborator Author

egillax commented Apr 26, 2023

If the goal is to identify issues in your design. Then I think it's in the wrong place currently. Currently while it runs before runPlp it doesn't affect it in any way. I think a more reasonable workflow would be:

  1. diagnosePlp and look at results
  2. adjust design if problematic
  3. develop model

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants