Simplify adding static parameters and metric to MLFlow

[AHarouni]

Describe the solution you'd like
I would like to have an way to add static variable as well as running variables to the mlflow handle.
This was mainly for monai label training as in Project-MONAI/MONAILabel#1383 (comment)
the ask was to fix it in monai core since monai label simply uses the handle

[wyli]

I think you should first make it part of engine.state then fetch it here

MONAI/monai/handlers/mlflow_handler.py

Line 202 in e18097d

attrs = {attr: getattr(engine.state, attr, None) for attr in self.default_tracking_params}

@Nic-Ma @binliunls please help confirm the usage (converting this into discussion for now)

[AHarouni]

@SachidanandAlle and Yucheng Tang can you confirm that this works for you. I need a simple way to add config parameters to mlflow tracking. @wyli can you follow up with @SachidanandAlle
If what you propose is something they can encapsulate on their end and expose a simple function in monai label to overwrite. Then it would work.

[wyli]

Sure, I'll discuss this with Sachi and Nic..

[Nic-Ma]

Hi @AHarouni ,

If you run a bundle, you can simplify add --tracking "mlflow" to enable MLFlow tracking and it will also record the config.
For more details, please refer to the tutorial:
https://github.com/Project-MONAI/tutorials/blob/main/experiment_management/bundle_integrate_mlflow.ipynb

Thanks.

[AHarouni]

Hi @Nic-Ma
In my mind bundle is for the last mile when I already have a good architecture/ parameters, and would like to export it to label or deploy.

In any case I am not using bundles yet for this project. I am training using monai label and need to be able to easily export static parameters in my experiment as num_samples for my crop transform, label cropping ratio, did I load a pretrained model, if so which one, etc
As it stands without this feature using mlflow is missing lots of information so I can NOT use it.

My work around is to manually add save my parameters to a json file for each experiment.

[Nic-Ma]

Hi @AHarouni ,

I understand your case now. I don't see any easy way to track the freestyle parameters if not using bundle, maybe you can submit a PR if you have any enhancement ideas for the MLFlowHandler.

Thanks in advance.

[binliunls]

For static variables, the MLFlowHandler takes a dict parameter experiment_param as input and will record the content inside it, when the engine starts.

For running variables, the MLFlowHandler by default logs the metrics every epoch and loss every iteration. The loss is transformed from the engine.state.output by the output_transform, which is a parameter of the MLFlowHandler and can be changed by users. If the running variables are part of metrics or loss, they will be recorded by the MLFlowHandler automatically.

If users want a more customized way to log running variables, a user defined epoch_logger or iteration_logger could also be passed to the MLFlowHandler to replace the default ones.