"Input string was not in a correct format" while model training

[nartalex]

System Information:

• Model Builder Version: 16.1.0.2026302
• Visual Studio Version: Enterprise 16.7.0 Preview 2.0

Describe the bug
Exception while training model:

Input string was not in a correct format.
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment`3.d__21.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 147

To Reproduce
Steps to reproduce the behavior:

1. Download dataset from here
2. Select Value prediction in Step 1 of Model Builder
3. Select Label - AvgTemperature, Features - 7 of 7 columns selected
4. Start training and wait for iteration 67 (it took ~25 min on my i5-9600)
5. See error

Expected behavior
Successful model training

Screenshots

Output screenshot in case you need to know exact trainers:

[beccamc]

@nartalex Thanks for reporting this problem! I've tried to reproduce and my training session succeeded. Can you try to update your Model Builder version 16.1.0.2027905?

[nartalex]

Hi @beccamc, I've tried the exact same dataset with new version of Model Builder and here what happened:
Steps were the same and failed iteration is 67 again

System Information:

Model Builder Version: 16.1.0.2027905
Visual Studio Version: Enterprise 16.7.0 Preview 3.1

Describe the bug
Exception while training model:

Input string was not in a correct format.
at System.Number.ParseSingle(String value, NumberStyles options, NumberFormatInfo numfmt)
at Microsoft.ML.AutoML.SweeperProbabilityUtils.ParameterSetAsFloatArray(IValueGenerator[] sweepParams, ParameterSet ps, Boolean expandCategoricals)
at Microsoft.ML.AutoML.SmacSweeper.FitModel(IEnumerable1 previousRuns) at Microsoft.ML.AutoML.SmacSweeper.ProposeSweeps(Int32 maxSweeps, IEnumerable1 previousRuns)
at Microsoft.ML.AutoML.PipelineSuggester.SampleHyperparameters(MLContext context, SuggestedTrainer trainer, IEnumerable1 history, Boolean isMaximizingMetric) at Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IEnumerable1 trainerWhitelist) at Microsoft.ML.AutoML.Experiment2.Execute()
at Microsoft.ML.AutoML.ExperimentBase2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator1 preFeaturizer, IProgress1 progressHandler, IRunner1 runner)
at Microsoft.ML.AutoML.ExperimentBase2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator1 preFeaturizer, IProgress1 progressHandler) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment3.<>c__DisplayClass21_0.b__5() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:line 81
at System.Threading.Tasks.Task1.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment3.d__21.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:line 108
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 147

[nikitayev]

Me too:

в System.Number.ParseSingle(String value, NumberStyles options, NumberFormatInfo numfmt)
в Microsoft.ML.AutoML.SweeperProbabilityUtils.ParameterSetAsFloatArray(IValueGenerator[] sweepParams, ParameterSet ps, Boolean expandCategoricals)
в Microsoft.ML.AutoML.SmacSweeper.FitModel(IEnumerable1 previousRuns) в Microsoft.ML.AutoML.SmacSweeper.ProposeSweeps(Int32 maxSweeps, IEnumerable1 previousRuns)
в Microsoft.ML.AutoML.PipelineSuggester.SampleHyperparameters(MLContext context, SuggestedTrainer trainer, IEnumerable1 history, Boolean isMaximizingMetric) в Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IEnumerable1 trainerAllowList) в Microsoft.ML.AutoML.Experiment2.Execute()
в Microsoft.ML.AutoML.ExperimentBase2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator1 preFeaturizer, IProgress1 progressHandler, IRunner1 runner)
в Microsoft.ML.AutoML.ExperimentBase2.ExecuteCrossValSummary(IDataView[] trainDatasets, ColumnInformation columnInfo, IDataView[] validationDatasets, IEstimator1 preFeaturizer, IProgress1 progressHandler) в Microsoft.ML.AutoML.ExperimentBase2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator1 preFeaturizer, IProgress1 progressHandler)
в Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment3.<>c__DisplayClass21_0.<ExecuteAsync>b__5() в /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:строка 80 в System.Threading.Tasks.Task1.InnerInvoke()
в System.Threading.Tasks.Task.Execute()
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment`3.d__21.MoveNext() в //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:строка 112
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Microsoft.ML.ModelBuilder.AutoMLEngine.d__26.MoveNext() в //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:строка 134

Formula:
X*Y=Z

100 rows of teacjing set

[beccamc]

@nikitayev Sorry you've run into this problem. What version of Model Builder are you on?

[bug-1358]

I have the same problem like a guys above:

Input string was not in a correct format

in System.Number.ParseSingle(String value, NumberStyles options, NumberFormatInfo numfmt)
   in Microsoft.ML.AutoML.SweeperProbabilityUtils.ParameterSetAsFloatArray(IValueGenerator[] sweepParams, ParameterSet ps, Boolean expandCategoricals)
   in Microsoft.ML.AutoML.SmacSweeper.FitModel(IEnumerable`1 previousRuns)
   in Microsoft.ML.AutoML.SmacSweeper.ProposeSweeps(Int32 maxSweeps, IEnumerable`1 previousRuns)
   in Microsoft.ML.AutoML.PipelineSuggester.SampleHyperparameters(MLContext context, SuggestedTrainer trainer, IEnumerable`1 history, Boolean isMaximizingMetric)
   in Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable`1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IEnumerable`1 trainerAllowList)
   in Microsoft.ML.AutoML.Experiment`2.Execute()
   in Microsoft.ML.AutoML.ExperimentBase`2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator`1 preFeaturizer, IProgress`1 progressHandler, IRunner`1 runner)
   in Microsoft.ML.AutoML.ExperimentBase`2.ExecuteCrossValSummary(IDataView[] trainDatasets, ColumnInformation columnInfo, IDataView[] validationDatasets, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
   in Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
   in Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment`3.<>c__DisplayClass21_0.<ExecuteAsync>b__5() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:line 80
   in System.Threading.Tasks.Task`1.InnerInvoke()
   in System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment`3.<ExecuteAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AutoMLExperiment.cs:line 107
--- End of stack trace from previous location where exception was thrown ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__26.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 150

It's occurred about a 69th step, usually. You can find a sample of my training data in a below listing (the last column is the value to predict)

0.000000000;1.000000000;0.941176471;0.696078431;0.647058824;0.147058824;0.205882353;0.800000000;0.542857143;0.823529412;0.598039216;0.558823529;0.117647059;0.323529412;0.600000000;0.585714286;0.000000000
0.030303030;0.000000000;0.000000000;0.000000000;0.000000000;0.000000000;0.029411765;0.000000000;0.028571429;0.058823529;0.000000000;0.000000000;0.000000000;0.029411765;0.000000000;0.028571429;1.000000000
0.060606061;1.000000000;0.588235294;0.029411765;0.029411765;0.000000000;0.029411765;0.030000000;0.057142857;0.235294118;0.009803922;0.000000000;0.029411765;0.029411765;0.000000000;0.014285714;1.000000000

I've updated a ModelBuilder recently, my current version is 16.3.0.2056001.

[i-holo]

still no fixes.
faced with the same problem on clean VS 2019 and extension version 16.3.0.2056001

[Isakskj]

I have the same problem on my computer with windows language set to Norwegian (which uses comma as decimal separator). However, the problem does not occur on a computer with English as windows language.

[justinormont]

@beccamc: Is Model Builder referencing the current version of the Microsoft.ML.AutoML NuGet?

This internalization bug should have been fixed in dotnet/machinelearning#5163, which is released in the 0.17.4 NuGet (release notes).

[beccamc]

Hi all. We've recently discovered a bug in our installer, meaning that customers are on an old version of Model Builder even though it says it's updated.

Does your UI look like this? This is an old version of Model Builder.

The new UI (16.3) looks like this...

If your version says 16.3, but your UI is the old version you need to completely uninstall Model Builder and reinstall. Instructions to uninstall are here.

[vzhuqin]

Not repro this issue on Main branch:
https://privategallery.blob.core.windows.net/gallery/refs/heads/main/atom.xml

• Windows 10 Enterprise, Version 20H2
• ML.Net Model Builder (Preview): 16.5.21.2122301
• Microsoft Visual Studio Enterprise 2019: 16.9.4
• .Net: 5.0.202

Steps:
1). Download the dataset from: https://www.kaggle.com/sudalairajkumar/daily-temperature-of-major-cities
2). Create new C# console app with .Net 5.0;
3). Add model builder by right click on the project;
4). Click "Value prediction" on Scenario page to navigate to Environment page;
5). Select Local (CPU) on Environment page;
6). Select File and click "Browse..." to input the downloaded dataset, set "AvgTemperature" to "Column to predict (Label)" on Data page;
7). Click "Start training" with default 1800s train time on Train page and wait Training complete;
Screenshot: