System.OperationCanceledException when calling experimentResult.BestRun.Estimator.Fit

[superichmann]

System Information (please complete the following information):

• OS & Version: Microsoft Windows 10 Pro
• ML.NET Version: ML.NET 2.0.1
• AutoML version: AutoML 0.20.1
• .NET Version: .NET 6.0

Describe the bug
As outlined in this tutorial, a model can be refitted to include the latest data used in the validation phase of the AutoML experiment.

When calling this method, as in my case:
private static ITransformer RefitBestPipeline(ExperimentResult<RegressionMetrics> experimentResult,IDataView final) { RunDetail<RegressionMetrics> bestRun = experimentResult.BestRun; return bestRun.Estimator.Fit(final); }
the return line causes the exception.

To Reproduce

1. Create a regression automl experiment in C#.
2. run the experiment on train validation data split
3. After experiment finished, call the RefitBestPipeline function with the best model and the entire IDataView (train+validation)
4. The error should occur on the return line

Expected behavior
The function should return the new ITransformer

Screenshots, Code, Sample Projects

System.OperationCanceledException
  HResult=0x8013153B
  Message=Operation was canceled.
  Source=Microsoft.ML.Core
  StackTrace:
   at Microsoft.ML.Runtime.Contracts.CheckAlive(IHostEnvironment env)
   at Microsoft.ML.Trainers.FastTree.DataConverter.MemImpl.InitializeBins(Int32 maxBins, IParallelTraining parallelTraining)
   at Microsoft.ML.Trainers.FastTree.DataConverter.MemImpl..ctor(RoleMappedData data, IHost host, Int32 maxBins, Single maxLabel, Boolean noFlocks, Int32 minDocsPerLeaf, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeatureIndices, Boolean categoricalSplit)
   at Microsoft.ML.Trainers.FastTree.DataConverter.Create(RoleMappedData data, IHost host, Int32 maxBins, Single maxLabel, Boolean diskTranspose, Boolean noFlocks, Int32 minDocsPerLeaf, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeatureIndices, Boolean categoricalSplit)
   at Microsoft.ML.Trainers.FastTree.ExamplesToFastTreeBins.FindBinsAndReturnDataset(RoleMappedData data, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeaturIndices, Boolean categoricalSplit)
   at Microsoft.ML.Trainers.FastTree.FastTreeTrainerBase`3.ConvertData(RoleMappedData trainData)
   at Microsoft.ML.Trainers.FastTree.FastForestRegressionTrainer.TrainModelCore(TrainContext context)
   at Microsoft.ML.Trainers.TrainerEstimatorBase`2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
   at Microsoft.ML.Trainers.TrainerEstimatorBase`2.Fit(IDataView input)
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
   at mlnet.AutoMLHelper.RefitBestPipeline(ExperimentResult`1 experimentResult, IDataView final)

Additional context
Note: I have read all of the other opened issues on this exception type and none of the proposed workarounds worked.
If relevant, In my case, the mlContext used to load the data is different then the one performing the experiment.
Perhaps the provided code example is too old now? (4 years) and irrelevant anymore?

[LittleLittleCloud]

You need to create a new MLContext for retraining purpose. As the original context has been canceled and can't be used to train a model.

[superichmann]

Hi again ;) I have read your comments in other issues but did not understand exactly how to do it.

The function is not called from mlContext but from bestRun.Estimator.Fit.

How do I enforce bestRun.Estimator.Fit to run from a new mlContext?

[LittleLittleCloud]

Ha,,, I see what you mean. It's possible that the refit experience is broken, I'll see what I can do here.

[jeffhandley]

I just ran into this bug using 2.0.1. @michaelgsharp / @ericstj, are we planning to publish a 2.0.2 with the fix from #6572?

[LittleLittleCloud]

The fix should already be released in 3.0.0-preview and after. Maybe try update to that version and see if you still hit this error? Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Jeff Handley ***@***.***> Sent: Saturday, June 10, 2023 6:05:54 PM To: dotnet/machinelearning ***@***.***> Cc: XiaoYun Zhang ***@***.***>; State change ***@***.***> Subject: Re: [dotnet/machinelearning] System.OperationCanceledException when calling experimentResult.BestRun.Estimator.Fit (Issue #6565) I just ran into this bug using 2.0.1<https://www.nuget.org/packages/Microsoft.ML/2.0.1>. @michaelgsharp<https://github.com/michaelgsharp> / @ericstj<https://github.com/ericstj>, are we planning to publish a 2.0.2 with the fix from #6572<#6572>? — Reply to this email directly, view it on GitHub<#6565 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEAYLOQ27SS7LIOG42Z32LTXKUKXFANCNFSM6AAAAAAUYW2VB4>. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[jeffhandley]

I did not encounter that same issue with the 3.0.0-preview.23266.6 package, but I encountered a different issue with model testing that I can't spend time troubleshooting right now. I'm thus blocked trying to adopt a version > 1.7.1. For a blocking issue on retraining, we should consider servicing 2.0.x with the fix rather than requiring users to adopt a new (preview) major version . For the time being, I'm rolling back to 1.7.1 to be unblocked.

My repro is when running the https://github.com/dotnet/issue-labeler (main branch) CreateMikLabelModel app with that repo set to use 2.0.x.