Skip to content

Commit 073269c

Browse files
authored
Update custom serialization docs (#4580)
This commit updates the custom serialization docs that discuss using JsonNetSerializer. 1. Remove the implication that the internal serializer is based on Json.NET. 2. Include recommendation to hook up JsonNetSerializer when using Newtonsoft.Json.Linq types like JObject
1 parent f08b6b7 commit 073269c

File tree

3 files changed

+76
-59
lines changed

3 files changed

+76
-59
lines changed

docs/client-concepts/high-level/serialization/custom-serialization.asciidoc

Lines changed: 33 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -15,32 +15,38 @@ please modify the original csharp file found at the link and submit the PR with
1515
[[custom-serialization]]
1616
=== Custom Serialization
1717

18-
NEST 6.x ships with a _shaded_ Json.NET dependency, meaning that all of Json.NET's types are
19-
internalized and IL merged into the NEST assembly, and their namespace has been changed
20-
from `Newtonsoft.Json` to `Nest.Json`.
21-
22-
Why would we do this, you may ask? Well, NEST has always isolated its dependency on Json.NET as best as it could,
23-
but this meant that we had to mandate how certain things were in the client. For instance,
24-
NEST heavily relied on the fact that the `IContractResolver` used by the configured serializer was
25-
an instance of `ElasticContractResolver`, so if you wanted to deserialize your `_source` or `_fields`
26-
using your own resolver, you were out of luck. In addition, continued improvements to NEST's serialization pipeline
27-
was stymied by this dependency and as client authors, we wanted to unhinder ourselves from this in order to explore the myriad
28-
of exciting developments happening in the .NET ecosystem around performance with the likes of `Span<T>`,
29-
`ArrayPool<T>` and a more compact UTF-8 string representation.
30-
31-
So what did we do in 6.x and how does it affect you?
32-
33-
The `NEST` nuget package from 6.0.0 onwards will use the internal Json.NET serializer and will in effect, behave the same
34-
as it did in previous releases. If you previously relied on custom Json.NET serialization and configuration with custom
35-
`JsonSerializerSettings` and `JsonConverter` types for example, things have changed a bit. The following section explains
36-
how to continue working with these with NEST 6.x.
18+
After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to
19+
announce that the next stage of serialization improvements have been completed in 7.0.
20+
21+
Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast
22+
serializer that works directly with UTF-8 binary.
23+
24+
With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have
25+
proven too onerous to carry forward at this stage.
26+
27+
* JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization
28+
routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons.
29+
We are considering options for exposing indented JSON for development and debugging purposes.
30+
31+
* NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from
32+
that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
33+
34+
* Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees.
35+
Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.
36+
37+
* Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[],
38+
but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.
39+
40+
* Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer
41+
in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a
42+
case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the
43+
C# POCO property name.
3744

3845
[float]
3946
==== Injecting a new serializer
4047

41-
Starting with NEST 6.x you can inject a serializer that is isolated to only be called
42-
for the (de)serialization of `_source`, `_fields`, or wherever a user provided value is expected
43-
to be written and returned.
48+
You can inject a serializer that is isolated to only be called for the (de)serialization of `_source`, `_fields`, or
49+
wherever a user provided value is expected to be written and returned.
4450

4551
Within NEST, we refer to this serializer as the `SourceSerializer`.
4652

@@ -109,8 +115,11 @@ back to NEST's built-in serializer.
109115

110116
We ship a separate {nuget}/NEST.JsonNetSerializer[NEST.JsonNetSerializer] package that helps in composing a custom `SourceSerializer`
111117
using `Json.NET`, that is smart enough to delegate the serialization of known NEST types back to the built-in
112-
`RequestResponseSerializer`. This package is also useful if you want to control how your documents and values are stored
113-
and retrieved from Elasticsearch using `Json.NET`, without interfering with the way NEST uses `Json.NET` internally.
118+
`RequestResponseSerializer`. This package is also useful if
119+
120+
. You want to control how your documents and values are stored and retrieved from Elasticsearch using `Json.NET`
121+
122+
. You want to use `Newtonsoft.Json.Linq` types such as `JObject` within your documents
114123

115124
The easiest way to hook this custom source serializer up is as follows
116125

docs/query-dsl.asciidoc

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -49,13 +49,13 @@ NEST exposes all of the full text queries available in Elasticsearch
4949

5050
* <<intervals-usage,Intervals Usage>>
5151

52-
* <<match-bool-prefix-usage,Match Bool Prefix Usage>>
52+
* <<match-usage,Match Usage>>
5353

54-
* <<match-phrase-prefix-usage,Match Phrase Prefix Usage>>
54+
* <<match-bool-prefix-usage,Match Bool Prefix Usage>>
5555

5656
* <<match-phrase-usage,Match Phrase Usage>>
5757

58-
* <<match-usage,Match Usage>>
58+
* <<match-phrase-prefix-usage,Match Phrase Prefix Usage>>
5959

6060
* <<multi-match-usage,Multi Match Usage>>
6161

@@ -71,13 +71,13 @@ include::query-dsl/full-text/common-terms/common-terms-usage.asciidoc[]
7171

7272
include::query-dsl/full-text/intervals/intervals-usage.asciidoc[]
7373

74-
include::query-dsl/full-text/match-bool-prefix/match-bool-prefix-usage.asciidoc[]
74+
include::query-dsl/full-text/match/match-usage.asciidoc[]
7575

76-
include::query-dsl/full-text/match-phrase-prefix/match-phrase-prefix-usage.asciidoc[]
76+
include::query-dsl/full-text/match-bool-prefix/match-bool-prefix-usage.asciidoc[]
7777

7878
include::query-dsl/full-text/match-phrase/match-phrase-usage.asciidoc[]
7979

80-
include::query-dsl/full-text/match/match-usage.asciidoc[]
80+
include::query-dsl/full-text/match-phrase-prefix/match-phrase-prefix-usage.asciidoc[]
8181

8282
include::query-dsl/full-text/multi-match/multi-match-usage.asciidoc[]
8383

@@ -122,14 +122,14 @@ NEST exposes all of the term queries available in Elasticsearch
122122

123123
* <<term-query-usage,Term Query Usage>>
124124

125-
* <<terms-set-query-usage,Terms Set Query Usage>>
126-
127125
* <<terms-list-query-usage,Terms List Query Usage>>
128126

129127
* <<terms-lookup-query-usage,Terms Lookup Query Usage>>
130128

131129
* <<terms-query-usage,Terms Query Usage>>
132130

131+
* <<terms-set-query-usage,Terms Set Query Usage>>
132+
133133
* <<wildcard-query-usage,Wildcard Query Usage>>
134134

135135
See the Elasticsearch documentation on {ref_current}/term-level-queries.html[Term level queries] for more details.
@@ -160,14 +160,14 @@ include::query-dsl/term-level/regexp/regexp-query-usage.asciidoc[]
160160

161161
include::query-dsl/term-level/term/term-query-usage.asciidoc[]
162162

163-
include::query-dsl/term-level/terms-set/terms-set-query-usage.asciidoc[]
164-
165163
include::query-dsl/term-level/terms/terms-list-query-usage.asciidoc[]
166164

167165
include::query-dsl/term-level/terms/terms-lookup-query-usage.asciidoc[]
168166

169167
include::query-dsl/term-level/terms/terms-query-usage.asciidoc[]
170168

169+
include::query-dsl/term-level/terms-set/terms-set-query-usage.asciidoc[]
170+
171171
include::query-dsl/term-level/wildcard/wildcard-query-usage.asciidoc[]
172172

173173
[[compound-queries]]
@@ -283,10 +283,10 @@ Specialized types of queries that do not fit into other groups
283283

284284
* <<rank-feature-query-usage,Rank Feature Query Usage>>
285285

286-
* <<script-score-query-usage,Script Score Query Usage>>
287-
288286
* <<script-query-usage,Script Query Usage>>
289287

288+
* <<script-score-query-usage,Script Score Query Usage>>
289+
290290
* <<shape-query-usage,Shape Query Usage>>
291291

292292
See the Elasticsearch documentation on {ref_current}/specialized-queries.html[Specialized queries] for more details.
@@ -305,10 +305,10 @@ include::query-dsl/specialized/pinned/pinned-query-usage.asciidoc[]
305305

306306
include::query-dsl/specialized/rank-feature/rank-feature-query-usage.asciidoc[]
307307

308-
include::query-dsl/specialized/script-score/script-score-query-usage.asciidoc[]
309-
310308
include::query-dsl/specialized/script/script-query-usage.asciidoc[]
311309

310+
include::query-dsl/specialized/script-score/script-score-query-usage.asciidoc[]
311+
312312
include::query-dsl/specialized/shape/shape-query-usage.asciidoc[]
313313

314314
[[span-queries]]

tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs

Lines changed: 29 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -21,34 +21,40 @@ namespace Tests.ClientConcepts.HighLevel.Serialization
2121
/**[[custom-serialization]]
2222
* === Custom Serialization
2323
*
24-
* NEST 6.x ships with a _shaded_ Json.NET dependency, meaning that all of Json.NET's types are
25-
* internalized and IL merged into the NEST assembly, and their namespace has been changed
26-
* from `Newtonsoft.Json` to `Nest.Json`.
24+
* After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to
25+
* announce that the next stage of serialization improvements have been completed in 7.0.
2726
*
28-
* Why would we do this, you may ask? Well, NEST has always isolated its dependency on Json.NET as best as it could,
29-
* but this meant that we had to mandate how certain things were in the client. For instance,
30-
* NEST heavily relied on the fact that the `IContractResolver` used by the configured serializer was
31-
* an instance of `ElasticContractResolver`, so if you wanted to deserialize your `_source` or `_fields`
32-
* using your own resolver, you were out of luck. In addition, continued improvements to NEST's serialization pipeline
33-
* was stymied by this dependency and as client authors, we wanted to unhinder ourselves from this in order to explore the myriad
34-
* of exciting developments happening in the .NET ecosystem around performance with the likes of `Span<T>`,
35-
* `ArrayPool<T>` and a more compact UTF-8 string representation.
27+
* Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast
28+
* serializer that works directly with UTF-8 binary.
3629
*
37-
* So what did we do in 6.x and how does it affect you?
30+
* With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have
31+
* proven too onerous to carry forward at this stage.
3832
*
39-
* The `NEST` nuget package from 6.0.0 onwards will use the internal Json.NET serializer and will in effect, behave the same
40-
* as it did in previous releases. If you previously relied on custom Json.NET serialization and configuration with custom
41-
* `JsonSerializerSettings` and `JsonConverter` types for example, things have changed a bit. The following section explains
42-
* how to continue working with these with NEST 6.x.
33+
* - JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization
34+
* routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons.
35+
* We are considering options for exposing indented JSON for development and debugging purposes.
36+
*
37+
* - NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from
38+
* that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
39+
*
40+
* - Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees.
41+
* Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.
42+
*
43+
* - Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[],
44+
* but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.
45+
*
46+
* - Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer
47+
* in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a
48+
* case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the
49+
* C# POCO property name.
4350
*/
4451
public class GettingStarted
4552
{
4653
/**[float]
4754
* ==== Injecting a new serializer
4855
*
49-
* Starting with NEST 6.x you can inject a serializer that is isolated to only be called
50-
* for the (de)serialization of `_source`, `_fields`, or wherever a user provided value is expected
51-
* to be written and returned.
56+
* You can inject a serializer that is isolated to only be called for the (de)serialization of `_source`, `_fields`, or
57+
* wherever a user provided value is expected to be written and returned.
5258
*
5359
* Within NEST, we refer to this serializer as the `SourceSerializer`.
5460
*
@@ -114,8 +120,10 @@ public class MyPercolationDocument
114120
*
115121
* We ship a separate {nuget}/NEST.JsonNetSerializer[NEST.JsonNetSerializer] package that helps in composing a custom `SourceSerializer`
116122
* using `Json.NET`, that is smart enough to delegate the serialization of known NEST types back to the built-in
117-
* `RequestResponseSerializer`. This package is also useful if you want to control how your documents and values are stored
118-
* and retrieved from Elasticsearch using `Json.NET`, without interfering with the way NEST uses `Json.NET` internally.
123+
* `RequestResponseSerializer`. This package is also useful if
124+
*
125+
* . You want to control how your documents and values are stored and retrieved from Elasticsearch using `Json.NET`
126+
* . You want to use `Newtonsoft.Json.Linq` types such as `JObject` within your documents
119127
*
120128
* The easiest way to hook this custom source serializer up is as follows
121129
*/

0 commit comments

Comments
 (0)