Allow custom converters for dictionary keys

[NN---]

Edited by @layomia.

Original post by @NN--- (click to view)

Description

JsonConverterAttribute is considered when used in a collection or as Dictionary value.
However, it is not considered when it is used as Dictionary key.

using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;

public class OptionConverter : JsonConverter<Option>
{
    public override Option? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        throw new NotImplementedException();
    }

    public override void Write(Utf8JsonWriter writer, Option value, JsonSerializerOptions options)
    {
        writer.WriteStringValue(value.Value);
    }
}

[JsonConverter(typeof(OptionConverter))]
public class Option
{
    public string Value { get; set; }
}

public class P
{
    static void Main()
    {
        var x = new Option {Value = "abc"};
        Console.WriteLine(JsonSerializer.Serialize(x)); // OK

        var y = new Option[] { new Option { Value = "abc" } };
        Console.WriteLine(JsonSerializer.Serialize(y)); // OK

        var z = new List<Option> { new Option { Value = "abc" } };
        Console.WriteLine(JsonSerializer.Serialize(z)); // OK

        var u = new Dictionary<string, Option> { { "abc", new Option { Value = "abc" } } };
        Console.WriteLine(JsonSerializer.Serialize(u)); // OK

        var d = new Dictionary<Option, string> {{new Option{Value = "abc"}, ""}};
        Console.WriteLine(JsonSerializer.Serialize(d)); // TKey has unsupported type
    }
}

Please support JsonConvertAttribute on Dictionary keys.

Configuration

.NET 5.0.1

Regression?

No.

Other information

---

Today custom converters provided for types that appear in input graphs as dictionary keys (mostly primitives like string, numeric types, guids etc) cannot be used to handle dictionary keys. This manifests as NotSupportedException being thrown by the serializer when a custom converter is used for these primitive types & and the types are serialized as dictionary keys. We should provide API to override the NSE-throwing behavior and allow custom converters to handle dictionary keys.

In .NET 5, the (de)serialization of dictionary keys was handled entirely by the serializer even when custom converters were provided for the dictionary key types. This behavior was broken earlier in the .NET 6 timeline as a known side effect of internal STJ infrastructure changes. Now, custom converters are also invoked for dictionary keys, but the mechanism to support them is internal and defaults to throwing NotSupportedException for all converters that are not internal in STJ & are supported as dictionary keys.

API Proposal

This involves exposing the following internal virtual methods in the following form:

public abstract class JsonConverter<T> : JsonConverter
{
  public virtual bool HandleNull => throw null;

  public abstract T? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options);
  
  public abstract void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options);

+ protected virtual T? ReadFromPropertyName(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => throw null;

+ protected virtual void WriteToPropertyName(Utf8JsonWriter writer, T value, JsonSerializerOptions options) { }
}

Notes

• We cannot have null values as dictionary keys, so the HandleNull property is not consulted when invoking the Read/WriteToPropertyName methods. For the same reason, wrt to nullability, we have a signature of T ReadFromPropertyName(...) and not T? ReadFromPropertyName.... Similarly we [DisallowNull] for the input T in the corresponding write method.
• New methods are introduced instead of reusing the existing Read and Write methods because the new functionality assumes that we are reading and writing strictly JSON property names, where the token type of the written value is JsonTokeType.PropertyName. To avoid user-confusion about whether to check the current reader.TokenType when reading, and whether to write values using writer.WritePropertyName, we introduce new methods where we can document the expected usage patterns.
• The introduction of these APIs means that all serializable types can be serialized and deserialized as dictionary keys, provided that users provide implementations of the new methods. This includes complex types like POCOs and collections. This feature gives us functionality that was present in Newtonsoft.Json (which is based on System.ComponentModel.TypeConverter infrastructure).
• The signatures of the new methods are based on the pre-existing Read/Write methods, e.g. passing JsonSerializerOptions instances & the type to convert on deserialization.

API usage

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace Test
{
    class Program
    {
        internal sealed class CustomStringConverter : JsonConverter<string>
        {
            public override string Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
            {	
	        Debug.Assert(reader.TokenType == JsonTokenType.String);
                return reader.GetString();
	    }

            public override void Write(Utf8JsonWriter writer, string value, JsonSerializerOptions options)
                => writer.WriteStringValue(value);
			
            public override string ReadFromPropertyName(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
            {
                Debug.Assert(reader.TokenType == JsonTokenType.PropertyName);
                return reader.GetString();
            }
			
            protected override void WriteToPropertyName(Utf8JsonWriter writer, string value, JsonSerializerOptions options)
                => writer.WritePropertyName(value);
        }

        static void Main()
        {
            JsonSerializerOptions  options = new() { Converters = { new CustomStringConverter() } };
            Dictionary<string, string> value = new() { ["key"] = "value" };

            string json = JsonSerializer.Serialize(value, options);
            Console.WriteLine(json); // {"key":"value"}
        }
    }
}

[layomia]

The larger feature here is allowing custom converters for dictionary keys. This was discussed as part of the design for allowing non-string dictionary keys in .NET 5 - #32676. The most likely route to achieving this is exposing these internal methods on JsonConverter<T>:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs

Lines 542 to 546 in 3116c4d

	internal virtual T ReadWithQuotes(ref Utf8JsonReader reader)
	=> throw new InvalidOperationException();
	internal virtual void WriteWithQuotes(Utf8JsonWriter writer, [DisallowNull] T value, JsonSerializerOptions options, ref WriteStack state)
	=> throw new InvalidOperationException();

cc @jozkee

[layomia]

Moving this to future for now, but I'm ready to react if anyone needs this as a result of #50074.