Inquiry on Data Type Compatibility and Specification in Ingest API

[mrchypark]

Dear Team,

Firstly, I would like to express my respect for your hard work and dedication to our project. I am reaching out to discuss a matter I encountered with the /api/v1/ingest endpoint. It appears that during the initial stream setup, a column with a float64 datatype is created. However, when data like the integer '1' is ingested into this float64 column, a 400 error with the message "Invalid Request: Could not process this event due to mismatch in datatype" is returned. Interestingly, inputting the value '1.1' processes correctly.

To better understand this, I manually inserted data and verified the schema via the schema API, confirming that the 'in' field is indeed 'Float64'. This has led me to wonder if this is the intended behavior, especially since I am not fully versed in strict typing systems.

My primary inquiry is whether there is a method to explicitly specify types when ingesting event data or metadata. This information would greatly aid in aligning our data ingestion processes with the system’s requirements. Alternatively, considering and documenting support for forced typecasting in certain special cases would be very helpful.

Thank you for your time and assistance in this matter.

[nitisht]

Thank you for the kind works @mrchypark . We’ll get back with some ideas for this.

[nitisht]

from @gyoza

Execution Error: Query Execution failed due to error in datafusion: Arrow error: Invalid argument error: column types must match schema types, expected List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found Boolean at column index 0

[Eshanatnight]

Hi @mrchypark
I apologize for the late reply.
currently during the Ingestion process, the schema of the log is enforced.
say if the original schema was

{
	"this_is_a_i64": 128,
	"this_is_a_f64":255.99 
}

and then a json blob of say,

{
	"this_is_a_i64": 127.99,
	"this_is_a_f64":256 
}

While parsing the incoming log, the schema of the incoming log is matched with the schema of the previous records. Hence the error

[mrchypark]

@Eshanatnight Thank you for your response. So, what would be a considerate course of action now?

[Eshanatnight]

@mrchypark If possible can you give me till the end of the week to discuss with the team and come up with some ideas?

Till then I can suggest that keeping the logs schemafull.

And Thanks again for your continued interest.

[nitisht]

Hi @mrchypark we discussed this in details. I think it will be much cleaner here for the application (sending the logs) to ensure it always sends proper data (vs dynamically changing types). Even if we store the log events as sent by the application, it will be difficult for users to make sense of this data. Take for example HTTP status column - 200, 500 etc.

If this is stored as numbers, you can query things like select * from logstream where status > 500 AND status < 600. But if the application changes the type to string, e.g. "200", such queries will not work.

I also researched a little on how this is being handled in other systems in this ecosystem. It is not a common practice to allow dynamically changing field types. General recommendation is to add a new field with the new type (with a different name).

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings

[nitisht]

Happy to discuss further with you regarding the specific application in this case and how we can fix it.

[mrchypark]

There are certain behaviors that I expect to be automatically handled in programming. Specifically, rather than expecting 200 and "200" to be treated the same, I anticipate that 20.0 and 20 would be considered equivalent. More precisely, I am referring to cases where a float can accept an integer input, such as entering 20 and having it stored as 20.0, or inputting 0 and having it stored as 0.0.

Languages may not strictly enforce this distinction, or may automatically convert 0.0 to 0 for reasons such as compression. I believe that if this could be addressed by parseable, it would maintain simplicity in usage across different tools.

For example, the R language provides Numeric and Integer types, with numeric encompassing both integers and floats, and integer strictly behaving as an integer. I think making floats behave similarly to numeric in R would aid in ease of use.

Thank you for considering this issue.

[mrchypark]

First and foremost, I'd like to extend my heartfelt thanks to everyone for their hard work and dedication. The development challenges we face are numerous, and your effort in overcoming them does not go unnoticed.

I wanted to bring up an important consideration that has emerged from our current work with JSON data handling, specifically in the context of receiving numerical inputs via HTTP POST requests. As outlined in the JSON specification (RFC 8259, Section 6, available at https://www.rfc-editor.org/rfc/rfc8259.html#section-6), there is an inherent challenge in distinguishing between float and integer data types when parsing JSON bodies. This is particularly problematic when our schema is set to expect float values, yet the JSON body presents a number that is technically an integer, making it difficult to treat this data as a float without further conversion.

This distinction, or rather the lack thereof, can potentially complicate our data processing, as it requires additional logic to ensure numerical values are interpreted and stored accurately according to their intended use. I believe addressing this nuanced aspect of data ingestion could significantly enhance the robustness and flexibility of our system.

Please consider this as we continue to develop and refine our data handling capabilities. Any thoughts on potential solutions or workarounds would be greatly appreciated. Again, thank you for your incredible dedication and for navigating these complexities with such skill and professionalism.

[nitisht]

Thanks for the detailed report @mrchypark . I believe we can add a add / define schema API that would let users define the schema for their data. But this API should only be called after creation of stream and before sending any event to the stream.

Do you think you can call that API with relevant schema and then proceed with event ingestion?

[mrchypark]

Is it possible to enforce a schema for data being sent, either by including it with the data or specifying only optional fields? If it's mandatory to send this schema every time before sending data, it might lead to a cumbersome design.

I am collecting data from IoT sensors using Benthos and Vector. If I have to send the schema every time before sending data, it would make these tools unusable for me.

[nitisht]

Schema addition will be a one time step - just after stream creation and before sending any event. After that server will enforce that schema on all events sent to that log stream.

Will this work for you?

[mrchypark]

Yes. It 'll works for me. It would be perfect if type cast is documented.