AutoML - Cannot multi-thread predictions

[SpeedyCraftah]

System information

• OS version/distro: Windows 10 Pro
• .NET Version (eg., dotnet --info): .NET Core 3.1

Issue

• What did you do? I compiled a model with the visual studio model builder then attempted to predict multiple labels with multi-threading.
• What happened? An exception gets thrown: An exception of type 'System.IndexOutOfRangeException' occurred in Microsoft.ML.Transforms.dll but was not handled in user code - Index was outside the bounds of the array.
• What did you expect? Not throw an error and run the predictions in parallel.

Source code / logs

using Ml_net_sentimentML.Model;
using System;
using System.Threading.Tasks;

namespace myMLApp
{
    class Program
    {
        public static void Main()
        {
            MainAsync().GetAwaiter().GetResult();
        }

        async static Task MainAsync()
        {
            string[] toPredict = { "This is a great!", "This is bad", "It doesn't work" };
            Task<string>[] resultTasks = new Task<string>[toPredict.Length];

            for (int i = 0; i < toPredict.Length; i++)
            {
                string review = toPredict[i];

                resultTasks[i] = Task.Run(() => PredictAndParse(review));
            }

            string[] results = await Task.WhenAll(resultTasks);

            for (int i = 0; i < results.Length; i++)
            {
                Console.WriteLine(results[i] + "\n");
            }
        }

        static string PredictAndParse(string review)
        {
            ModelInput input = new ModelInput()
            {
                Review = review
            };

            ModelOutput prediction = ConsumeModel.Predict(input);

            string sentiment = prediction.Prediction == "1" ? "Positive" : "Negative";

            return $"Text: {input.Review}\nSentiment: {sentiment}\nScore: {prediction.Score[0]} | {prediction.Score[1]}";
        }
    }
}

Full exception:

System.IndexOutOfRangeException
  HResult=0x80131508
  Message=Index was outside the bounds of the array.
  Source=Microsoft.ML.Transforms
  StackTrace:
   at Microsoft.ML.Transforms.Text.TokenizingByCharactersTransformer.Mapper.<>c__DisplayClass15_0.<MakeGetterOne>b__0(VBuffer`1& dst)
   at Microsoft.ML.Data.RowCursorUtils.<>c__DisplayClass11_0`2.<GetVecGetterAsCore>b__0(VBuffer`1& dst)
   at Microsoft.ML.Transforms.Text.NgramExtractingTransformer.Mapper.<>c__DisplayClass11_0.<MakeGetter>b__2(VBuffer`1& dst)
   at Microsoft.ML.Transforms.LpNormNormalizingTransformer.Mapper.<>c__DisplayClass8_0.<MakeGetter>b__5(VBuffer`1& dst)
   at Microsoft.ML.Data.ColumnConcatenatingTransformer.Mapper.BoundColumn.<>c__DisplayClass20_0`1.<MakeGetter>b__0(VBuffer`1& dst)
   at Microsoft.ML.Transforms.NormalizeTransform.AffineColumnFunction.Sng.ImplVec.<>c__DisplayClass5_0.<GetGetter>b__0(VBuffer`1& dst)
   at Microsoft.ML.Data.SchemaBindablePredictorWrapperBase.<>c__DisplayClass18_0`2.<GetValueGetter>b__0(TDst& dst)
   at Microsoft.ML.Data.PredictedLabelScorerBase.EnsureCachedPosition[TScore](Int64& cachedPosition, TScore& score, DataViewRow boundRow, ValueGetter`1 scoreGetter)
   at Microsoft.ML.Data.MulticlassClassificationScorer.<>c__DisplayClass16_0.<GetPredictedLabelGetter>b__0(UInt32& dst)
   at Microsoft.ML.Transforms.KeyToValueMappingTransformer.Mapper.KeyToValueMap`2.<>c__DisplayClass8_0.<GetMappingGetter>b__0(TValue& dst)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.<>c__DisplayClass9_0`2.<CreateConvertingActionSetter>b__0(TRow row)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.FillValues(TRow row)
   at Microsoft.ML.Data.TypedCursorable`1.RowImplementation.FillValues(TRow row)
   at Microsoft.ML.PredictionEngineBase`2.FillValues(TDst prediction)
   at Microsoft.ML.PredictionEngine`2.Predict(TSrc example, TDst& prediction)
   at Microsoft.ML.PredictionEngineBase`2.Predict(TSrc example)
   at Ml_net_sentimentML.Model.ConsumeModel.Predict(ModelInput input) in C:\Users\USER\source\repos\ml.net sentiment\ml.net sentimentML.Model\ConsumeModel.cs:line 20
   at myMLApp.Program.PredictAndParse(String review) in C:\Users\USER\source\repos\ml.net sentiment\ml.net sentiment\Program.cs:line 48
   at myMLApp.Program.<>c__DisplayClass1_0.<MainAsync>b__0() in C:\Users\USER\source\repos\ml.net sentiment\ml.net sentiment\Program.cs:line 26
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)```

I'm quite new to C# so I wouldn't be surprised if this was an issue on my end.

Note

When not attempting to multi-thread while predicting, it works fine (iterating over the toPredict array and calling ConsumeModel.Predict on all of them separately), no exceptions get thrown other than the fact that it's slow since it runs inconcurrently.

[michaelgsharp]

Hey @SpeedyCraftah, The PredictionEngine itself is not thread safe. This article talks a little more about it and talks about setting up a PredictionEnginePool in ASP.NET. It's not your exact same case, but it should hopefully give you more insight.

I don't think our documentation itself says that PredictionEngine is not thread safe, so I will add that in so its more clear.

Does that help? I'm going to close this ticket (I'll reference this issue when I create the PR for documentation updates) for now since thread safety is a known issue, but if you have more questions please feel free to re-open this and ask them!

[SpeedyCraftah]

It turns out I didn't need it after all since predictions are <1ms! The initial loading of the DDLs and model conflicted with my time measurements which resulted in +2 seconds.