Array dimensions exceeded supported range

[tcaivano]

System Information (please complete the following information):

• Model Builder Version (available in Manage Extensions dialog): 17.16.0.2326802
• Visual Studio Version: Community 2022 v 4.8.04084

Describe the bug

• On which step of the process did you run into an issue: About 1 hour into model training, Model Builder displays the following exception:
  2023-05-21 13:50:16.8911 DEBUG Array dimensions exceeded supported range. at System.Collections.Generic.List1.set_Capacity(Int32 value)
  at System.Collections.Generic.List1.EnsureCapacity(Int32 min) at System.Collections.Generic.List1.Add(T item)
  at Microsoft.ML.Trainers.FastTree.DataConverter.ValuesList.Add(Int32 index, Double value)
  at Microsoft.ML.Trainers.FastTree.DataConverter.MemImpl.MakeBoundariesAndCheckLabels(Int64& missingInstances, Int64& totalInstances)
  at Microsoft.ML.Trainers.FastTree.DataConverter.MemImpl..ctor(RoleMappedData data, IHost host, Double[][] binUpperBounds, Single maxLabel, Boolean dummy, Boolean noFlocks, PredictionKind kind, Int32[] categoricalFeatureIndices, Boolean categoricalSplit)
  at Microsoft.ML.Trainers.FastTree.DataConverter.Create(RoleMappedData data, IHost host, Int32 maxBins, Single maxLabel, Boolean diskTranspose, Boolean noFlocks, Int32 minDocsPerLeaf, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeatureIndices, Boolean categoricalSplit)
  at Microsoft.ML.Trainers.FastTree.ExamplesToFastTreeBins.FindBinsAndReturnDataset(RoleMappedData data, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeaturIndices, Boolean categoricalSplit)
  at Microsoft.ML.Trainers.FastTree.FastTreeTrainerBase3.ConvertData(RoleMappedData trainData) at Microsoft.ML.Trainers.FastTree.FastTreeBinaryTrainer.TrainModelCore(TrainContext context) at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
  at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings) at Microsoft.ML.AutoML.SweepablePipelineRunner.RunAsync(TrialSettings settings, CancellationToken ct) at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__24.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.BinaryClassificationExperiment.<ExecuteAsync>d__14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/BinaryClassificationExperiment.cs:line 148 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 169 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
• Clear description of the problem: I am attempting to train a model using a large dataset of 468,079,637 rows with 8 columns each using Model Builder. Five of the columns are string, which do not exceed 255 characters. The remaining columns are boolean (0, 1) values. The total dataset size is 26.3 GBs. I have configured the training to be on a boolean column, and the dataset is unbalanced such that a majority of the items are of class 0. One column is an index and is ignored in Data -> Advanced data options. I started training for 7200 seconds, during which time there is relatively low memory usage in my system (maximum 31.7 GBs). After about 50 minutes of training, memory usage spikes to use all remaining memory, and the above exception is displayed.

To Reproduce
Steps to reproduce the behavior:
N/A

Expected behavior
There are no Array dimensions exceptions.

Additional context
This is using the Data classification scenario. In Advanced training options, all trainers are enabled.

[tcaivano]

I noticed today that the Data Preview only detects 43,176,428 rows:

[zewditu]

@tcaivano thanks for reporting the issue related with big data size, we will take look into it.

[LittleLittleCloud]

@tcaivano
What's the column type of your col1 && col2, looks like they should be categorical.

And looks like you are hitting OOM error in fast tree trainer. This is because fast tree uses in-memory transposer by default. The AutoML already support disk transposer but we didn't expose it in model builder for now.

As a work-around, you can disable fast tree, fast forest and only enable lgbm, which based on our experience is the trainer that works best on huge dataset. In the meantime, we will push a fix to model builder to enable in-disk transposer for fast tree.

[tcaivano]

@LittleLittleCloud Yes, you're right - my mistake, both col1 and col2 should both be categorical, not strings.
I'll try that out soon - I'm currently running some additional tests that are taking up my cycles but I should have some bandwidth within the new few hours.

[tcaivano]

I'm getting some issues with LGBM too.
2023-05-23 15:50:43.0820 DEBUG Array dimensions exceeded supported range. at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.EnsureCapacity(Int32 min) at System.Collections.Generic.List1.Add(T item)
at Microsoft.ML.Trainers.LightGbm.LightGbmTrainerBase4.GetMetainfo(IChannel ch, Factory factory, Int32& numRow, Single[]& labels, Single[]& weights, Int32[]& groups) at Microsoft.ML.Trainers.LightGbm.LightGbmTrainerBase4.LoadTrainingData(IChannel ch, RoleMappedData trainData, CategoricalMetaData& catMetaData)
at Microsoft.ML.Trainers.LightGbm.LightGbmTrainerBase4.TrainModelCore(TrainContext context) at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings) at Microsoft.ML.AutoML.SweepablePipelineRunner.RunAsync(TrialSettings settings, CancellationToken ct) at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__24.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.BinaryClassificationExperiment.<ExecuteAsync>d__14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/BinaryClassificationExperiment.cs:line 148 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 169 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)

[LittleLittleCloud]

Hmmm OK, can you try sdca/lbfgs then, they both support streaming training so might work?

[tcaivano]

No luck with LBFGS:
2023-05-23 17:51:59.9082 DEBUG Array dimensions exceeded supported range. at System.Array.Resize[T](T[]& array, Int32 newSize) at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized) at Microsoft.ML.Trainers.LbfgsTrainerBase3.TrainCore(IChannel ch, RoleMappedData data)
at Microsoft.ML.Trainers.LbfgsTrainerBase3.TrainModelCore(TrainContext context) at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings) at Microsoft.ML.AutoML.SweepablePipelineRunner.RunAsync(TrialSettings settings, CancellationToken ct) at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__24.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.BinaryClassificationExperiment.<ExecuteAsync>d__14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/BinaryClassificationExperiment.cs:line 148 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 169 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
No error with sdca, but I've yet to get anything with a greater than 0 F1 score

[LittleLittleCloud]

Action item

• add disk-transpose fast-tree trainer in advanced training option

[luisquintanilla]

PR in ML.NET complete. Model Builder PR pending. To be released alongside advanced training (tuner) options.