-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
System Information (please complete the following information):
- OS & Version: Windows 11 [Version 10.0.22621.675]
- ML.NET Version: ML.NT v2.0.0-preview.22525.3
- .NET Version: .NET 6.0.9
Describe the bug
I created a custom class that implements the Microsoft.ML.AutoML.IMonitor to monitor my training. It's ReportRunningTrial method has the following simple implementation:
_logger.LogInformation($"{"Running".PadLeft(9)} Trial #{setting.TrialId.ToString().PadLeft(4)} - Pipeline: {_pipeline.ToString(setting.Parameter).PadRight(70)}");
The other, ReportBestTrial, ReportCompletedTrial and ReportFailTrial methods look very similar.
When I run my training it looks like that the ReportRunningTrial method's pipeline object wasn't updated, so it doesn't show the pipeline of the current trial.
To Reproduce
Steps to reproduce the behavior:
- Create a custom class that implements IMonitor
- Create an ML.NET experiment, and use that monitor class, like this:
var experiment = mlContext.Auto().CreateExperiment();
experiment
.SetPipeline(pipeline)
.SetTrainingTimeInSeconds(trainingTimeInSeconds)
.SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
.SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
.SetMonitor(monitor)
.SetMaximumMemoryUsageInMegaByte(20 * 1024);
- Make sure that you allow more than one type of (regression) algorithms
- Run the experiment
- Check the output: you will see that the
settings.TrialIdproperty values look good, but the_pipeline.ToString(setting.Parameter)returns an invalid value in theReportRunningTrialmethod.
Expected behavior
The _pipeline.ToString(setting.Parameter) should return the just-started trial's pipeline.
Screenshots, Code, Sample Projects

Additional context
You can see the problem on the screenshot:
Running Trial 0's pipeline looks like it was FastTreeRegression, but in reality it's FastForestRegression.
Running Trial 1's pipeline looks like it was FastForestRegression, but it's FastTreeRegression.
Running Trial 2's pipeline looks like it was FastForestRegression, but it's LightGbmRegression.
etc.
Edit: After testing a little more I'm not sure anymore which methods of the IMonitor interface report the pipeline wrong, it is possible that ReportRunningTrial is the correct one and the others are incorrect.