System.Text.Json: (De)serialization support for quoted numbers

[NickCraver]

Original proposal by @NickCraver (click to view)

Apologies if this issue exists already...I hunted and couldn't find one.

I've been trying to switch a lot of usages over to System.Text.Json from Newtonsoft and Jil, but a recurring theme is handling external APIs and their not-quite-right returns of JSON.

Unfortunately, an example I'm seeing over and over is:

{ "field":"12345" }

They're numbers, but as strings. int, double, decimal, whatever. It happens all the time. And since it's someone's API, it's unlikely we'll get the world to fix this.

Note: this is something Newtonsoft handles by default, which is why I don't think most people realize it's an issue.

Is there any chance we can let System.Text.Json handle number types being quoted when deserializing? Something a user opts-into via JsonSerializerOptions?

Small repro example:

void Main()
{
	JsonSerializer.Parse<Foo>(@"{""Bar"":""1234""}");
}
public class Foo
{
	public int Bar { get; set; }
}

cc @ahsonkhan @steveharter

---

Edited by @layomia:

Users want to be able to deserialize JSON strings into number properties. Scenarios include deserializing "NaN", "Infinity" and "-Infinity" (#31024).

We should add an option to enable this scenario, and possibly allow serializing numbers as strings.

These semantics allow better interop with various API endpoints accross the web.

API Proposal

namespace System.Text.Json
{
    public partial sealed class JsonSerializerOptions
    {
        public JsonNumberHandling NumberHandling { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    [Flags]
    public enum JsonNumberHandling : byte
    {
        /// <summary>
        /// No specified number handling behavior. Numbers can only be read from <see cref="JsonTokenType.Number"/> and will only be written as JSON numbers (without quotes).
        /// </summary>
        None = 0x0,
        /// <summary>
        /// Numbers can be read from <see cref="JsonTokenType.String"/>. Does not prevent numbers from being read from <see cref="JsonTokenType.Number"/>.
        /// </summary>
        AllowReadingFromString = 0x1,
        /// <summary>
        /// Numbers will be written as JSON strings (with quotes), not as JSON numbers.
        /// </summary>
        WriteAsString = 0x2,
        /// Floating point constants represented as <see cref="JsonTokenType.String"/> tokens
        /// such as "NaN", "Infinity", "-Infinity", can be read when reading, and such CLR values
        /// such as <see cref="float.NaN"/>, <see cref="double.PositiveInfinity"/>, <see cref="float.NegativeInfinity"/> will be written as their corresponding JSON string representations.
        AllowNamedFloatingPointLiterals = 0x4
    }

    public partial sealed class JsonNumberHandlingAttribute : JsonAttribute
    {
        public JsonNumberHandling Handling { get; }
    
        public JsonNumberHandlingAttribute(JsonNumberHandling handling)
        {
            Handling = handling;
        }
    }
}

Usage

Allow reading numbers from strings; write numbers as strings.

public class ClassWithInts
{
    public int NumberOne { get; set; }
    public int NumberTwo { get; set; }
}

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString | JsonNumberHandling.WriteAsString 
};

string json = @"{""Number1"":1,""Number2"":""2""}";

ClassWithInts @class = JsonSerializer.Deserializer<ClassWithInts>(json, options);
Console.WriteLine(@class.NumberOne); // 1
Console.WriteLine(@class.NumberTwo); // 2

json = JsonSerializer.Serialize(@class, options);
Console.WriteLine(json); // @"{""Number1"":""1"",""Number2"":""2""}";

Allow reading numbers from floating point constants; write floating point constants

Given a class:

public class ClassWithNumbers
{
    public int IntNumber { get; set; }
    public float FloatNumber { get; set; }
}

Without the new option, reading floating-point-constant representations fails:

string json = @"{""IntNumber"":1,""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json);

// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Single. Path: $.FloatNumber | LineNumber: 0 | BytePositionInLine: 34.

Writing also fails:

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj);

// Unhandled exception. System.ArgumentException: .NET number values such as positive and negative infinity cannot be written as valid JSON.

With the new option, reading floating-point-constant representations works:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":1,""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // NaN

Writing floating-point-constant representations also works:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":-1,"FloatNumber":"NaN"}

Allow reading numbers from string; support reading and writing floating point constants

Reading:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":""1"",""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // NaN

Writing:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":-1,"FloatNumber":"NaN"}

Write numbers as strings; support reading and writing floating point constants

Reading:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":""1""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // 0

string json = @"{""IntNumber"":""1"",""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 16.

Writing:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":"-1","FloatNumber":"NaN"}

Read semantics are the same as the serializer's default (no quotes)

For example, integers cannot have decimal places:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1.0""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 17.

Similarly, there can't be any leading or trailing trivia:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1a""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 15.

Leading or trailing whitespace is also not allowed:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1 ""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 15.

Note that formats such as the currency and percentage formats are not allowed. Commas (,), the percent symbol (%), and underscores (_) are not permitted in input strings.

Notes

Newtonsoft compat

• Newtonsoft.Json allows implicit conversions from strings to numbers on deserialization, i.e, no custom logic or setting is required. We will require that users explictly opt in to this behavior.
• Newtonsoft.Json does not provide a built-in way to serialize numbers as strings. Custom logic, e.g. a custom converter needs to provided for this behavior. This proposal includes an easy way to specify this behavior.

Read/write semantics

• The number reading and writing behavior of this feature is exactly the same as if no quotes were specified, except that surrounding quotes can be allowed on deserialization, and can be written on serialization.
  • Reading/writing is not culture aware (i.e InvariantCulture). A custom converter has to be specified for culture-aware handling.
  • The "G17" standard numeric format is used when reading and writing double representations.
  • The "G9" standard format is used when reading and writing float representations.
  • Percentage and currency formats are not supported.
• Numbers represented as JsonTokenType.String tokens will be unescaped if needed when deserializing. This is in keeping with other string-based parsing for types like DateTime and Guid.
• Only the literal "NaN", "Infinity", and "-Infinty" values are allowed. This is in keeping with the most common variations on the web. The matching is case-sensitive. We can be more permissive in the future, based on user feedback.

[NickCraver]

From Twitter: a lot of people correctly point out that numbers are intentionally quoted in APIs so they're not assumed to be floats and lose precision. It's an excellent point, and even more reason to support deserializing them I think.

[bartonjs]

numbers are intentionally quoted in APIs so they're not assumed to be floats and lose precision

To me that says that the user should still deserialize them as strings and only turn them into numbers when they're ready to accept the loss of precision.

[manigandham]

@bartonjs

That's the intention. The data types for the class properties are what we want the final result to be.

If we have to use string-based properties first and then parse ourselves then there's not much utility for this serialization library in terms of mapping to types at all, especially for APIs.

[steveharter]

With support for a custom double and\or decimal converter you can do this yourself. Perhaps there will be future support for a set of "Json.NET-compat" converters at some point but that is not available now.

Below is an skeleton of such a custom converter. Also see https://github.com/dotnet/corefx/issues/38713 which has a custom object converter that converts to bool if token type is True\False.

public class DoubleConverterWithStringSupport : JsonConverter<double>
{
    public override double Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            return ...;
        }

        // Default behavior; will throw if TokenType != Number
        return reader.GetDouble();
    }

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        // Write as number or string? Could also use [JsonConverter] to specify a different converter for certain properties.
    }
}

[steveharter]

Moving to Future. cc @rynowak

[NickCraver]

I'm trying to unwrap all that (thanks for the workaround!) - is that saying we'd never consider adding it to the base serializer and it'd only ever live as an extension point? I'm finding the lack of this to be a very common pain point, so I'm wondering what the bar is for inclusion if so.

[bartonjs]

@NickCraver I think it's just "there's a whole lot of policy here, and we would need to completely spec it out, and we're trying to be feature complete 30 days ago... so this isn't required for v1 (.NET Core 3.0)"

• Is there an option to say if deserializing should be strict vs coercive? (Personally, I hate coercive APIs, so I would always want it in "this data better conform to schema" mode)
• If numbers are deserialized through either JSON Number or JSON String, how are they written?
  • Do we need per property toggles other than the converters?
• Is only invariant format supported? Or also CurrentCulture? Or CurrentUICulture?
• How should the deserialization handle it if the property is typed as int but the value is the string "3.0"? Is that 3, or an exception? (Currently (IIRC) if it's a JSON number that says 3.0 and the field is int it throws)
• Should it ignore trailing spaces if they ended up in the string? Leading spaces?
• Should it support the percentage format? The currency format?

It's a simple-sounding request, but there are a lot of details and many of them are hard to change once the feature first ships. So the .NET Core 3.0 answer is "deserialization doesn't coerce numbers out of JSON strings; but the converters feature lets you effectively opt-in on a property-by-property basis".

[NickCraver]

Totally understand, I was mostly curious about where the bar is and if moving to this was a good from a future standpoint (or we shouldn't try to move certain code sections to it). That list helps a lot - I agree this can't be just tossed in.

Thanks for the quick reply!

[manigandham]

The converter option mentioned by @steveharter works great now in the .NET Core 3.0 preview-7 release. Confirmed server-side and in Blazor.

public class LongToStringConverter : JsonConverter<long>
{
    public override long Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            ReadOnlySpan<byte> span = reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan;
            if (Utf8Parser.TryParse(span, out long number, out int bytesConsumed) && span.Length == bytesConsumed)
                return number;

            if (Int64.TryParse(reader.GetString(), out number))
                return number;
        }

        return reader.GetInt64();
    }

    public override void Write(Utf8JsonWriter writer, long value, JsonSerializerOptions options)
    {
        writer.WriteStringValue(value.ToString());
    }
}

services.AddControllersWithViews()
   .AddJsonOptions(options =>
   {
        options.JsonSerializerOptions.Converters.Add(new LongToStringSupport());
   });

This was developed in the other thread: https://github.com/dotnet/corefx/issues/36563#issuecomment-519788518

[manigandham]

@AlexPaskhin

It's not every type... this case is about Javascript/JSON not supporting large numeric types so it needs a string conversion, and the reasoning by the team on why this isn't built-in makes sense. The converter code is right there so you copy/paste if you need it, but I guess they can add some default converters for long and double in the package.

What other problems are you having? You don't have to use System.Text.Json if you need more flexibility, Json.NET is still available and easily enabled.

[oleg-varlamov]

We have a similar situation: we decided to move our project to .net core 3.0. and use the built-in System.Text.Json. We had to rewrite a small part of the functionality, but in general, the build was successful.
However, in runtime we began to receive json parsing errors(not only from other services, but also from our own). We had to do a commit revert and return to Newtonsoft.Json.
Of course, it is clear that it is necessary to correctly transfer all numeric values, without quotes. But our partners are unlikely to want to redo the already approved integration format.

[stephentoub]

MS Project Manager has wasted million dollars to develop a technical debt. As a share holder I will send complain about him to the MS board. It should help him to listen.

You are welcome to send your feedback to whomever you please, of course. I'll look forward to hearing their response to you.

Please also keep in mind that trying to threaten folks here is against our code of conduct. Folks here are actively listening to feedback and engaging and trying to make a better library for everyone, but conduct such as yours will only serve for them to start ignoring you.

Thank you.

cc: @terrajobst

[terrajobst]

@AlexPaskhin

Designs are often trade-off decisions, which means making the product better for someone might make it worse for someone else. Implying that this means we don't care or are not good at our job is missing the point.

We appreciate passion and dissenting views, especially when they help us finding bugs and holes in the design. However, we don't appreciate non-constructive feedback, such as your last responses. Implying we don't do our job or directly threatening us isn't acceptable behavior and against our Code of Conduct. Please keep this in mind when engaging with us or we're forced to block you because it's distracting us from delivering quality by engaging in constructive discourse with our community, which is indeed our job.

[AlexPaskhin]

@terrajobst
I’ve removed all comments in these thread as it is useless, you like/happy with things that already done.
I wish you luck with design trade-off. I wish that your version of “JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate.”
I wish that System.Text.JSON would have a wide success with other teams, with your JSON/CSON (CSharp Object Notation) design trade off.

[steveharter]

For 5.0 we can consider an option to support "looser" deserialization modes depending on how common these scenarios are, and whether we can agree on common semantics.

However I believe all such modes can be supported today by using custom converters and by extending JsonConverterAttribute so that specific properties can be targeted (instead of all properties). For some cases it may be that there is no common consensus for the semantics so leaving it as-is and requiring the use of custom converters and custom attributes is the best solution. Perhaps doc and examples for common scenarios works too, or the community creates extensions for common scenarios.

A 5.0 feature could be to add a options.DeserializeQuotedNumbers == true. Then we need to determine if serializing as numbers is always desired, otherwise we'd need another option like SerializeNumbersAsQuoted (perhaps a flags enum would be easier to extend instead of separate option properties). There are subtleties here like do we account for culture settings? We'd also need a new attribute so that specific properties can be targeted instead of all through the global option.

Similarly, there has been feedback on what primitive types to deserialize when the property is a System.Object. Today that is JsonElement. The requests have been to deserialize true\false as System.Bool, numbers as System.Double and strings as System.String instead of JsonElement.

Adding additional option(s) for this is also possible, however there are subtleties: precision can be lost (e.g. System.Double has less precision than System.Decimal -- what is the intent?) and when the JSON values belong to an object being deserialized into System.Object, the deserializer would still need to deserialize the object and all of its properties as JsonElement since it doesn't know what type to create and deserialize into (without getting into polymorphic deserialization, which is a separate feature).