Error on prediction with LSTM Model in ONNX Format in ML.Net

[ladislav-dolezal]

System information

• Windows 10
• .NET Core 3.1, Microsoft.ML 1.5.0, Microsoft.ML.OnnxRuntime 1.3.0, Microsoft.ML.OnnxTransformer 1.5.0

Issue

I would like to predict Values with my LSTM Model in ONNX Format. I am struggling on data input for my ONNX Model in my ML.NET project. When I run my code, I get an Exception: System.NullReferenceException: 'Object reference not set to an instance of an object.' on prediction. The input data is just dummy. I am trying to find out, how to input data in form (None,3,7) as expected from LSTM Model. LSTM Model ist attached. Can someone help me, what I am doing wrong?
Why I get exception or how to deliver data for my ONNX Model.

using Microsoft.ML;
using Microsoft.ML.Data;
using System;
using System.Data.SqlClient;

namespace InQu.ML.Test.ONNX.FehlerAbschätzungLSTM
{
    class Program
    {
        static void Main(string[] args)
        {
            var mlContext = new MLContext();
            
            var x = new InputData[] { 
                        new InputData() { X = new float[,] { { 1, 2, 3, 4, 5, 6, 7 }, { 1, 2, 3, 4, 5, 6, 7 }, { 1, 2, 3, 4, 5, 6, 7 } } }                       
                    };

            IDataView dataView = mlContext.Data.LoadFromEnumerable<InputData>(x);

            var pipeline = mlContext.Transforms.ApplyOnnxModel(modelFile: @".\scikit-learn\AbschatzungFehlerLSTM.onnx", 
                            inputColumnNames: new[] { "input_layer",  }, 
                            outputColumnNames: new[] { "dense_3" });

            var model = pipeline.Fit(dataView);
            
            var predEngine = mlContext.Model.CreatePredictionEngine<InputData, OutputData>(model);

            OutputData prediction = new OutputData() { y = new float[0] };
            predEngine.Predict(x[0], ref prediction);
        }
    }

    public class InputData
    {
        [ColumnName("input_layer")]
        [VectorType(3,7)]
        public float[,] X { get; set; }        
    }

    public class OutputData
    {
        [ColumnName("dense_3")]
        [VectorType(1)]
        public float[] y { get; set; }
    }
}

[antoniovs1029]

Hi, @ladodc . I believe this is a duplicate of dotnet/machinelearning-samples#828 So I'll close that issue, to continue the discussion here.

Can you please share the full stack trace of your exception? Also, if you're able to share your .onnx model, it would be very useful so that we can take a closer look!

At first look, I think the problem is that you're not correctly setting the input for your onnx model. Unfortunately, there's no general advice I could provide on how you should do this, since it will depend on the input expected by you .onnx model.

I'd think the problem is that you need to reshape your input float[,] array into an array of the correct shape expected by onnx (you mention, (None, 3, 7)). The problem is that ML.NET doesn't currently have any transformer to do tensor reshaping, and it's necessary for users to actually implement their own reshape logic and use it inside a CustomMappingTransformer (see issue #765 ).

In particular when working with Onnx, I gave this advice to a user that was having a similar problem like you (although in their case the problem was somewhat more complicated, as they were dealing with input images, and also needing to reshape that input). So you can check that advice to get an idea of how to use CustomMappingTransformer for your use case:
#4795 (comment)

Note: I'm not sure on how reshape should be done if the onnx model expects a dimension of size None such as in your case. I don't know if simply reshaping it to (1x3x7) would do, or maybe even (3x7). I'd need your .onnx model to actually experiment myself until I figure out what the correct logic should be for your CustomMappingTransformer to work as expected.

[ladislav-dolezal]

Hi Antonio,

thanks for your response. I will try to follow your recommendation. Meanwhile I can give you my ONNX Model to experiment with this.

AbschatzungFehlerLSTM.zip

[antoniovs1029]

Hi, @ladodc .

So, after looking closer into this, it turns out that the (None, 3, 7) input you have, will actually be pretty much treated as (1,3,7) by the OnnxTransformer. The first dimension is considered the batch dimension, so it can be ignored.

Also, after playing around I realized that there's no need for the CustomMappingTransformer that I suggested in a previous comment, because you're actually already setting the input dimensions in the input class to match the ones expected by the OnnxTransformer. The real problem is that multidimensional input arrays aren't allowed in ML.NET, i.e. this is causing the exception:

    public class InputData
    {
        [ColumnName("input_layer")]
        [VectorType(3,7)]
        public float[,] X { get; set; }  // Using 3x7 arrays      
    }

It should be:

    public class InputData
    {
        [ColumnName("input_layer")]
        [VectorType(3,7)] // This could also be (1,3,7), it seems it doesn't matter
        public float[] X { get; set; }  // Using 1-D arrays of size 21      
    }

...

new InputData() { X = new float[] { 1, 2, 3, 4, 5, 6, 7 , 1, 2, 3, 4, 5, ... 

The info of the shape is already set by the VectorType(3,7) attribute, and ML.NET does require the users to input 1-D arrays. The exception you got is caused by this, and I'll try to open a PR soon so that a more understandable exception is thrown.

After making the change above, I was able to run your code without problems, and got a predictions array. Still, the prediction was "0"... I tried modifying the values of the input vector, and I always got a prediction = "0". I did debug this, and it seems everything is working fine in the OnnxTransformer, so I'm not sure: Is this expected? Is your model supposed to return 0 with the data you provided?. I'm not yet sure what your model is supposed to do (the sigmoid at the end that returns 1 float, makes me think this is a binary classification problem), so please let us know if this expected behavior. If it's not expected, then please let us know what would be the expected behavior, and if possible any code you used to create the model, so that I can further investigate if this is a problem in ML.NET, in OnnxRuntime, or in your model. Thanks! 😄

[antoniovs1029]

I've opened a PR updating the docs of VectorTypeAttribute(dims) so it's clearer that it's only meant to be used for 1-D arrays. And, as mentioned above, I was able to run your sample after flattening the input array. And so I'll close this issue now.

Please, feel free to reopen the issue if you find that the behavior of running the Onnx model isn't as expected, and provide the info requested on my last comment if this is the case. Thanks. 😄