Skip to content

Commit 8ef5aec

Browse files
MaximMikhisorMaximMikhisor
authored
Merge pull request #1 from neuecc/master
Merging
2 parents 7a4a9ef + 64ee45c commit 8ef5aec

File tree

64 files changed

+1228
-549
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+1228
-549
lines changed

‎Directory.Build.props‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,10 @@
2828
<AdditionalFiles Include="$(MSBuildThisFileDirectory)stylecop.json" Visible="false" />
2929
</ItemGroup>
3030
<ItemGroup>
31-
<PackageReference Include="Microsoft.NETFramework.ReferenceAssemblies" Version="1.0.0" PrivateAssets="All" />
32-
<PackageReference Include="Nerdbank.GitVersioning" Version="3.2.7-beta" PrivateAssets="all" />
33-
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="16.5.124-alpha" PrivateAssets="All" />
34-
<PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0" PrivateAssets="All" />
31+
<PackageReference Include="Microsoft.NETFramework.ReferenceAssemblies" Version="1.0.2" PrivateAssets="All" />
32+
<PackageReference Include="Nerdbank.GitVersioning" Version="3.4.244" PrivateAssets="all" />
33+
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="17.0.64" PrivateAssets="All" />
34+
<PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.1.1" PrivateAssets="All" />
3535
<PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.261" PrivateAssets="all" />
3636
</ItemGroup>
3737

‎README.md‎

Lines changed: 73 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ MessagePack has a compact binary size and a full set of general purpose expressi
3434
- [Security](#security)
3535
- [Performance](#performance)
3636
- [Deserialization Performance for different options](#deserialization-performance-for-different-options)
37+
- [String interning](#string-interning)
3738
- [LZ4 Compression](#lz4-compression)
3839
- [Attributions](#attributions)
3940
- [Comparison with protobuf, JSON, ZeroFormatter](#comparison-with-protobuf-json-zeroformatter)
@@ -699,7 +700,7 @@ Benchmarks comparing MessagePack For C# to other serializers were run on `Window
699700
* Avoid string key decoding for lookup maps (string key and use automata based name lookup with inlined IL code generation, see: [AutomataDictionary](https://github.com/neuecc/MessagePack-CSharp/blob/bcedbce3fd98cb294210d6b4a22bdc4c75ccd916/src/MessagePack/Internal/AutomataDictionary.cs)
700701
* To encode string keys, use pre-generated member name bytes and fixed sized byte array copies in IL, see: [UnsafeMemory.cs](https://github.com/neuecc/MessagePack-CSharp/blob/f17ddc5d107d3a2f66f60398b214ef87919ff892/src/MessagePack/Internal/UnsafeMemory.cs)
701702

702-
Before creating this library, I implemented a fast fast serializer with [ZeroFormatter#Performance](https://github.com/neuecc/ZeroFormatter#performance). This is a further evolved implementation. MessagePack for C# is always fast and optimized for all types (primitive, small struct, large object, any collections).
703+
Before creating this library, I implemented a fast serializer with [ZeroFormatter#Performance](https://github.com/neuecc/ZeroFormatter#performance). This is a further evolved implementation. MessagePack for C# is always fast and optimized for all types (primitive, small struct, large object, any collections).
703704

704705
### <a name="deserialize-performance"></a>Deserialization Performance for different options
705706

@@ -758,6 +759,49 @@ Extra note, this is serialization benchmark result.
758759

759760
Of course, `IntKey` is fastest but `StringKey` also performs reasonably well.
760761

762+
### <a name="string-interning"></a>String interning
763+
764+
The msgpack format does not provide for reusing strings in the data stream.
765+
This naturally leads the deserializer to create a new `string` object for every string encountered,
766+
even if it is equal to another string previously encountered.
767+
768+
When deserializing data that may contain the same strings repeatedly it can be worthwhile
769+
to have the deserializer take a little extra time to check whether it has seen a given string before
770+
and reuse it if it has.
771+
772+
To enable string interning on *all* string values, use a resolver that specifies `StringInterningFormatter`
773+
before any of the standard ones, like this:
774+
775+
```cs
776+
var options = MessagePackSerializerOptions.Standard.WithResolver(
777+
CompositeResolver.Create(
778+
new IMessagePackFormatter[] { new StringInterningFormatter() },
779+
new IFormatterResolver[] { StandardResolver.Instance }));
780+
781+
MessagePackSerializer.Deserialize<ClassOfStrings>(data, options);
782+
```
783+
784+
If you know which fields of a particular type are likely to contain duplicate strings,
785+
you can apply the string interning formatter to just those fields so the deserializer only pays
786+
for the interned string check where it matters most.
787+
Note that this technique requires a `[MessagePackObject]` or `[DataContract]` class.
788+
789+
```cs
790+
[MessagePackObject]
791+
public class ClassOfStrings
792+
{
793+
[Key(0)]
794+
[MessagePackFormatter(typeof(StringInterningFormatter))]
795+
public string InternedString { get; set; }
796+
797+
[Key(1)]
798+
public string OrdinaryString { get; set; }
799+
}
800+
```
801+
802+
If you are writing your own formatter for some type that contains strings,
803+
you can call on the `StringInterningFormatter` directly from your formatter as well for the strings.
804+
761805
## LZ4 Compression
762806

763807
MessagePack is a fast and *compact* format but it is not compression. [LZ4](https://github.com/lz4/lz4) is an extremely fast compression algorithm, and using it MessagePack for C# can achieve extremely fast performance as well as extremely compact binary sizes!
@@ -1452,24 +1496,36 @@ var resolver = MessagePack.Resolvers.CompositeResolver.Create(
14521496

14531497
## Reserved Extension Types
14541498

1455-
MessagePack for C# already used some MessagePack extension type codes, be careful to use same ext code.
1499+
MessagePack for C# already used some MessagePack extension type codes, be careful to avoid using the same ext code for other purposes.
1500+
1501+
Range | Reserved for
1502+
--|--
1503+
\[-128, -1\] | Reserved by the msgpack spec for predefined types
1504+
\[30, 120) | Reserved for this library's use to support common types in .NET
1505+
1506+
This leaves the following ranges for your use:
1507+
1508+
- \[0, 30)
1509+
- \[120, 127]
1510+
1511+
Within the *reserved* ranges, this library defines or implements extensions that use these type codes:
14561512

14571513
| Code | Type | Use by |
1458-
| --- | --- | --- |
1459-
| -1 | DateTime | MessagePack-spec reserved for timestamp |
1460-
| 30 | Vector2[] | for Unity, UnsafeBlitFormatter |
1461-
| 31 | Vector3[] | for Unity, UnsafeBlitFormatter |
1462-
| 32 | Vector4[] | for Unity, UnsafeBlitFormatter |
1463-
| 33 | Quaternion[] | for Unity, UnsafeBlitFormatter |
1464-
| 34 | Color[] | for Unity, UnsafeBlitFormatter |
1465-
| 35 | Bounds[] | for Unity, UnsafeBlitFormatter |
1466-
| 36 | Rect[] | for Unity, UnsafeBlitFormatter |
1467-
| 37 | Int[] | for Unity, UnsafeBlitFormatter |
1468-
| 38 | Float[] | for Unity, UnsafeBlitFormatter |
1469-
| 39 | Double[] | for Unity, UnsafeBlitFormatter |
1470-
| 98 | All | MessagePackCompression.Lz4BlockArray |
1471-
| 99 | All | MessagePackCompression.Lz4Block |
1472-
| 100 | object | TypelessFormatter |
1514+
| ---- | ---- | --- |
1515+
| -1 | DateTime | MessagePack-spec reserved for timestamp |
1516+
| 30 | Vector2[] | for Unity, UnsafeBlitFormatter |
1517+
| 31 | Vector3[] | for Unity, UnsafeBlitFormatter |
1518+
| 32 | Vector4[] | for Unity, UnsafeBlitFormatter |
1519+
| 33 | Quaternion[] | for Unity, UnsafeBlitFormatter |
1520+
| 34 | Color[] | for Unity, UnsafeBlitFormatter |
1521+
| 35 | Bounds[] | for Unity, UnsafeBlitFormatter |
1522+
| 36 | Rect[] | for Unity, UnsafeBlitFormatter |
1523+
| 37 | Int[] | for Unity, UnsafeBlitFormatter |
1524+
| 38 | Float[] | for Unity, UnsafeBlitFormatter |
1525+
| 39 | Double[] | for Unity, UnsafeBlitFormatter |
1526+
| 98 | All | MessagePackCompression.Lz4BlockArray |
1527+
| 99 | All | MessagePackCompression.Lz4Block |
1528+
| 100 | object | TypelessFormatter |
14731529

14741530
## Unity support
14751531

‎azure-pipelines.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ stages:
1919
jobs:
2020
- job: Windows
2121
pool:
22-
vmImage: windows-2019
22+
vmImage: windows-2022
2323
steps:
2424
- checkout: self
2525
clean: true
@@ -44,7 +44,7 @@ stages:
4444

4545
- job: Linux
4646
pool:
47-
vmImage: Ubuntu 18.04
47+
vmImage: ubuntu-20.04
4848
steps:
4949
- checkout: self
5050
clean: true
@@ -53,7 +53,7 @@ stages:
5353

5454
- job: macOS
5555
pool:
56-
vmImage: macOS-10.15
56+
vmImage: macOS-11
5757
steps:
5858
- checkout: self
5959
clean: true
@@ -64,7 +64,7 @@ stages:
6464
# It also helps exercise mpc so bugs don't go unnoticed.
6565
- job: codegen_diff
6666
pool:
67-
vmImage: ubuntu-latest
67+
vmImage: ubuntu-20.04
6868
steps:
6969
- checkout: self
7070
clean: true
@@ -84,7 +84,7 @@ stages:
8484
jobs:
8585
- job: push
8686
pool:
87-
vmImage: ubuntu-latest
87+
vmImage: ubuntu-20.04
8888
steps:
8989
- download: current
9090
artifact: nuget

‎azure-pipelines/build_nonWindows.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,12 @@ steps:
1414
testRunTitle: netcoreapp3.1-$(Agent.JobName)
1515

1616
- task: DotNetCoreCLI@2
17-
displayName: Run MessagePack.Tests (net5.0)
17+
displayName: Run MessagePack.Tests (net6.0)
1818
inputs:
1919
command: test
2020
projects: tests/MessagePack.Tests/MessagePack.Tests.csproj
21-
arguments: --no-build -c $(BuildConfiguration) -f net5.0 -v n --settings "$(Build.Repository.LocalPath)/azure-pipelines/$(Agent.OS).runsettings"
22-
testRunTitle: net5.0-$(Agent.JobName)
21+
arguments: --no-build -c $(BuildConfiguration) -f net6.0 -v n --settings "$(Build.Repository.LocalPath)/azure-pipelines/$(Agent.OS).runsettings"
22+
testRunTitle: net6.0-$(Agent.JobName)
2323

2424
- bash: mono ~/.nuget/packages/xunit.runner.console/2.4.1/tools/net472/xunit.console.exe bin/MessagePack.Tests/$(BuildConfiguration)/net472/MessagePack.Tests.dll -html $(BUILD.ARTIFACTSTAGINGDIRECTORY)/build_logs/mono_testrun.html
2525
displayName: Run MessagePack.Tests (mono)

‎benchmark/SerializerBenchmark/SerializerBenchmark.csproj‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
<PackageReference Include="SpanJson" Version="3.0.1" />
2121
<PackageReference Include="System.Text.Json" Version="4.6.0" />
2222
<PackageReference Include="Utf8Json" Version="1.3.7" />
23-
<PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
23+
<PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
2424
<PackageReference Include="Newtonsoft.Json.Bson" Version="1.0.2" />
2525
</ItemGroup>
2626

‎global.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"sdk": {
3-
"version": "5.0.401",
3+
"version": "6.0.300",
44
"rollForward": "patch",
55
"allowPrerelease": false
66
}

‎sandbox/DynamicCodeDumper/DynamicCodeDumper.csproj‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -143,8 +143,7 @@
143143
</Compile>
144144
</ItemGroup>
145145
<ItemGroup>
146-
<PackageReference Include="Nerdbank.Streams" Version="2.4.48" />
147-
<PackageReference Include="System.Memory" Version="4.5.3" />
148-
<PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="4.5.2" />
146+
<PackageReference Include="Nerdbank.Streams" Version="2.8.57" />
147+
<PackageReference Include="Microsoft.NET.StringTools" Version="1.0.0" />
149148
</ItemGroup>
150149
</Project>

‎sandbox/PerfBenchmarkDotNet/PerfBenchmarkDotNet.csproj‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<Project Sdk="Microsoft.NET.Sdk">
22
<PropertyGroup>
33
<OutputType>Exe</OutputType>
4-
<TargetFrameworks>net472;net5.0</TargetFrameworks>
4+
<TargetFrameworks>net472;net6.0</TargetFrameworks>
55
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
66
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
77
</PropertyGroup>
@@ -19,7 +19,7 @@
1919
<PackageReference Include="Jil" version="2.15.4" />
2020
<PackageReference Include="MessagePack" version="1.4.3" />
2121
<PackageReference Include="MsgPack.Cli" version="0.9.0-rc1" />
22-
<PackageReference Include="Nerdbank.Streams" Version="2.4.48" />
22+
<PackageReference Include="Nerdbank.Streams" Version="2.8.57" />
2323
<PackageReference Include="Newtonsoft.Json" version="10.0.3" />
2424
<PackageReference Include="protobuf-net" version="2.3.2" />
2525
<PackageReference Include="RandomFixtureKit" Version="1.0.0" />

‎sandbox/PerfNetFramework/PerfNetFramework.csproj‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
<Project Sdk="Microsoft.NET.Sdk">
22
<PropertyGroup>
33
<OutputType>Exe</OutputType>
4-
<TargetFrameworks>net472;net5.0</TargetFrameworks>
4+
<TargetFrameworks>net472;net6.0</TargetFrameworks>
55
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
66
</PropertyGroup>
77
<ItemGroup>
88
<PackageReference Include="MsgPack.Cli" version="0.9.0-beta2" />
9-
<PackageReference Include="Nerdbank.Streams" Version="2.4.48" />
9+
<PackageReference Include="Nerdbank.Streams" Version="2.8.57" />
1010
<PackageReference Include="Newtonsoft.Json" version="10.0.3" />
1111
<PackageReference Include="protobuf-net" version="2.1.0" />
1212
<PackageReference Include="ZeroFormatter" version="1.6.4" />

0 commit comments

Comments
 (0)