Unix: add mode to inline Socket continuations #34945

tmds · 2020-04-14T13:28:29Z

On Unix, socket continuations are dispatched to the ThreadPool
from an event thread. This avoids continuations blocking the
event handling.

This adds an option to disable that dispatch. Continuations for
socket operations will be executed on the event thread directly.
This removes the overhead of context switching to the ThreadPool.

Currently this is implemented as an application level switch for
benchmarking, and experimentation. It may be made controllable
at the Socket level.

To avoid the event threads being a bottleneck, ProcessorCount
event threads are created. We lower MinHandlesForAdditionalEngine
to spread Sockets evenly over these event threads.

ghost · 2020-04-14T13:28:35Z

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

tmds · 2020-04-14T13:30:19Z

cc @stephentoub @davidfowl @adamsitnik @antonfirsov @karelz

src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive.cs

tmds · 2020-04-14T14:11:07Z

I'm still running tests with the flag set, some tests are taking long. They may have a problem with the inline continuations. I'll give them some more time.

stephentoub · 2020-04-14T14:12:44Z

At a minimum I expect you'll have problems with this one 😉
#1973

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

stephentoub · 2020-04-14T17:14:07Z

(FYI, I tried this out on the various ASP.NET perf machines. Without changes at the ASP.NET layer, just substituting this System.Net.Sockets.dll, it improves RPS on the asp-perf machines by ~20%, and similarly for the asp-citrine machines if they're limited to half their cores... but if they're not, then there's no RPS improvement, and actually a small 1%-ish regression.)

davidfowl · 2020-04-14T17:17:12Z

if they're limited to half their core

Limited how? epoll threads? IOQueues?

stephentoub · 2020-04-14T17:17:51Z

Limited how? epoll threads? IOQueues?

--cpu-set 0-13

davidfowl · 2020-04-14T17:19:11Z

--cpu-set 0-13

It would be good to understand which knobs that affect in .NET itself and try tweaking those manually to see where that comes from. Do yo have counters from that run?

stephentoub · 2020-04-14T17:39:40Z

Do yo have counters from that run?

I don't; --collect-counters isn't working for some reason.

scalablecory · 2020-04-14T19:12:16Z

Do we know how this will affect scalability under high concurrency? Are we essentially removing the thread pool and running the entire server under I/O threads with this unless someone explicitly escapes via e.g. Task.Run?

Do we understand perf here with bad async code that blocks?

stephentoub · 2020-04-14T19:13:58Z

Do we understand perf here with bad async code that blocks?

Abysmal. It is not safe. It will not be on by default.

Are we essentially removing the thread pool and running the entire server under I/O threads with this unless someone explicitly escapes via e.g. Task.Run?

At the sockets level, yes.

davidfowl · 2020-04-14T19:25:51Z

Abysmal. It is not safe. It will not be on by default.

s/safe/scalable for non IO based workloads

stephentoub · 2020-04-14T19:29:07Z

s/safe/scalable for non IO based workloads

These epoll threads are only involved if you're doing async networking operations, and if you are and you block waiting on something else that requires that same epoll thread (e.g. await ReceiveAsync(); ReceiveAsync().Wait();, you'll deadlock.

davidfowl · 2020-04-14T19:30:53Z

These epoll threads are only involved if you're doing async networking operations, and if you are and you block waiting on something else that requires that same epoll thread (e.g. await ReceiveAsync(); ReceiveAsync().Wait();, you'll deadlock.

Ah yes, everything is now winforms 😄. It's a good thing we protect the socket in ASP.NET Core 😄

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

tmds · 2020-04-15T13:50:49Z

I've hard coded InlineSocketContinuations to true to make CI give me a list of tests that fail.

tmds · 2020-04-15T19:13:14Z

CI lists failures in:

System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http2_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer
System.Net.Mail.Tests.SmtpClientTest.TestMailDeliveryAsync
System.Net.Http.Functional.Tests
System.Net.Mail.Functional.Tests
System.Net.Requests.Tests
System.Net.Sockets.Tests

Should I look at making all of these pass when inlining continuations? The changes involve ensuring no blocking occurs on the continuations.
Or maybe I can limit to System.Net.Sockets.Tests?

stephentoub · 2020-04-15T21:00:58Z

Should I look at making all of these pass when inlining continuations? The changes involve ensuring no blocking occurs on the continuations. Or maybe I can limit to System.Net.Sockets.Tests?

I'm actually a little surprised it's that few. 😉

If you have time, I think it'd be worth understanding why they're failing, and if there's obvious product or test blocking issues we could fix, but I expect the majority answer will simply be that sync-over-async is fundamentally problematic in the face of this option.

However, I'd prioritize System.Net.Sockets.Tests; I'd like us to be able to run that suite with the flag set, which means we need to understand those failures and either fix them where appropriate or disable the test conditionally on the env var being set.

And it'd be good to add a test (outer loop is fine) that uses RemoteInvoke to run some tests with the environment variable enabled just to help validate that the basic things work as expected. We do that with some ETW tests:

runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/LoggingTest.cs

Lines 35 to 67 in 5395b14

    
           [OuterLoop] 
        
           [Fact] 
        
           public void EventSource_EventsRaisedAsExpected() 
        
           { 
        
               RemoteExecutor.Invoke(() => 
        
               { 
        
                   using (var listener = new TestEventListener("Microsoft-System-Net-Sockets", EventLevel.Verbose)) 
        
                   { 
        
                       var events = new ConcurrentQueue<EventWrittenEventArgs>(); 
        
                       listener.RunWithCallback(events.Enqueue, () => 
        
                       { 
        
                           // Invoke several tests to execute code paths while tracing is enabled 
        
                           new SendReceiveSync(null).SendRecv_Stream_TCP(IPAddress.Loopback, false).GetAwaiter(); 
        
                           new SendReceiveSync(null).SendRecv_Stream_TCP(IPAddress.Loopback, true).GetAwaiter(); 
        
                           new SendReceiveTask(null).SendRecv_Stream_TCP(IPAddress.Loopback, false).GetAwaiter(); 
        
                           new SendReceiveTask(null).SendRecv_Stream_TCP(IPAddress.Loopback, true).GetAwaiter(); 
        
                           new SendReceiveEap(null).SendRecv_Stream_TCP(IPAddress.Loopback, false).GetAwaiter(); 
        
                           new SendReceiveEap(null).SendRecv_Stream_TCP(IPAddress.Loopback, true).GetAwaiter(); 
        
                           new SendReceiveApm(null).SendRecv_Stream_TCP(IPAddress.Loopback, false).GetAwaiter(); 
        
                           new SendReceiveApm(null).SendRecv_Stream_TCP(IPAddress.Loopback, true).GetAwaiter(); 
        
                           new NetworkStreamTest().CopyToAsync_AllDataCopied(4096).GetAwaiter().GetResult(); 
        
                           new NetworkStreamTest().Timeout_ValidData_Roundtrips().GetAwaiter().GetResult(); 
        
                       }); 
        
                       Assert.DoesNotContain(events, ev => ev.EventId == 0); // errors from the EventSource itself 
        
                       Assert.InRange(events.Count, 1, int.MaxValue); 
        
                   } 
        
               }).Dispose(); 
        
           }

tmds · 2020-04-16T15:07:02Z

However, I'd prioritize System.Net.Sockets.Tests; I'd like us to be able to run that suite with the flag set, which means we need to understand those failures and either fix them where appropriate or disable the test conditionally on the env var being set.

To make System.Net.Sockets.Tests pass with inline socket continuations, the mentioned BlockingAsyncContinuations_OperationsStillCompleteSuccessfull must be skipped. Other tests pass.

The cases in other test projects had blocking (like Wait(), Result calls).
If there is an API to control this at Socket(-operation) level, which defaults to not inline, these tests will pass without needing changes.

And it'd be good to add a test (outer loop is fine) that uses RemoteInvoke to run some tests with the environment variable enabled just to help validate that the basic things work as expected.

I'll look into adding a test like this.

davidfowl · 2020-04-16T16:22:11Z

src/libraries/System.Net.Sockets/tests/FunctionalTests/InlineContinuations.cs

20 mins? 5?

20 isn't that high:

/home/tmds/repos/runtime/artifacts/bin/testhost/netcoreapp5.0-Linux-Debug-x64/dotnet exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -nologo -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -method System.Net.Sockets.Tests.InlineContinuations.InlineSocketContinuations Discovering: System.Net.Sockets.Tests (method display = ClassAndMethod, method display options = None) Discovered: System.Net.Sockets.Tests (found 1 of 1197 test case) Starting: System.Net.Sockets.Tests (parallel test collections = on, max threads = 16) System.Net.Sockets.Tests: [Long Running Test] 'System.Net.Sockets.Tests.InlineContinuations.InlineSocketContinuations', Elapsed: 00:03:57 System.Net.Sockets.Tests: [Long Running Test] 'System.Net.Sockets.Tests.InlineContinuations.InlineSocketContinuations', Elapsed: 00:05:58 Finished: System.Net.Sockets.Tests === TEST EXECUTION SUMMARY === System.Net.Sockets.Tests Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 415.588s

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

On Unix, socket continuations are dispatched to the ThreadPool from an event thread. This avoids continuations blocking the event handling. This adds an option to disable that dispatch. Continuations for socket operations will be executed on the event thread directly. This removes the overhead of context switching to the ThreadPool. Currently this is implemented as an application level switch for benchmarking, and experimentation. It may be made controllable at the Socket level. To avoid the event threads being a bottleneck, ProcessorCount event threads are created.

tmds · 2020-05-12T19:48:52Z

@stephentoub @kouvel @adamsitnik I've rebased this PR on master, ptal.

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

stephentoub

Thanks, @tmds.

tmds · 2020-06-02T14:29:33Z

I broke the Windows build with the last change.

adamsitnik

I confirm that this PR is NOT regressing the PR.

I've switched to a new BenchmarkDriver2 and have not updated my parser yet, so I am not going to provide a nice table this time. If anyone is curious, please see the numbers in the details below.

--scenario plaintext --profile master --profile citrine

application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,761.99
Working Set (MB):     121
Build Time (ms):      5,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.66
Working Set (MB):     35
Build Time (ms):      3,501
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         14,298,212
Requests:             215,764,832
Mean latency (ms):    1.70
Max latency (ms):     142.47
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.55
Latency 75th (ms):    0.91
Latency 90th (ms):    5.43
Latency 99th (ms):    24.73

--scenario plaintext --profile pr --profile citrine

application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,763.13
Working Set (MB):     126
Build Time (ms):      4,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.66
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         14,270,636
Requests:             215,469,408
Mean latency (ms):    1.75
Max latency (ms):     66.48
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.56
Latency 75th (ms):    0.90
Latency 90th (ms):    5.44
Latency 99th (ms):    18.86

--scenario json --profile master --profile citrine

application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,783.66
Working Set (MB):     368
Build Time (ms):      4,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         1,143,918
Requests:             17,272,492
Mean latency (ms):    1.01
Max latency (ms):     60.24
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.40
Latency 75th (ms):    0.47
Latency 90th (ms):    0.69
Latency 99th (ms):    14.81

--scenario json --profile pr --profile citrine

application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,783.91
Working Set (MB):     365
Build Time (ms):      4,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         1,138,762
Requests:             17,194,935
Mean latency (ms):    0.92
Max latency (ms):     80.33
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.40
Latency 75th (ms):    0.47
Latency 90th (ms):    0.61
Latency 99th (ms):    13.69

--scenario fortunes --profile master --profile citrine

db
-------
## Host Process:
CPU Usage (%):        27
Raw CPU Usage (%):    1,280.63
Working Set (MB):     405
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        98
Raw CPU Usage (%):    2,743.75
Working Set (MB):     495
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         332,001
Requests:             5,012,947
Mean latency (ms):    1.69
Max latency (ms):     64.83
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.46
Latency 75th (ms):    1.64
Latency 90th (ms):    2.04
Latency 99th (ms):    8.30

--scenario fortunes --profile pr --profile citrine

db
-------
## Host Process:
CPU Usage (%):        27
Raw CPU Usage (%):    1,274.80
Working Set (MB):     385
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,758.65
Working Set (MB):     499
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         331,355
Requests:             5,003,322
Mean latency (ms):    1.74
Max latency (ms):     67.81
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.45
Latency 75th (ms):    1.65
Latency 90th (ms):    2.11
Latency 99th (ms):    9.99

--scenario fortunes --profile multiplexing --profile master --profile citrine

db
-------
## Host Process:
CPU Usage (%):        20
Raw CPU Usage (%):    966.94
Working Set (MB):     407
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    2,693.27
Working Set (MB):     510
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         414,397
Requests:             6,257,779
Mean latency (ms):    1.33
Max latency (ms):     44.77
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.13
Latency 75th (ms):    1.37
Latency 90th (ms):    1.84
Latency 99th (ms):    5.71

--scenario fortunes --profile multiplexing --profile pr --profile citrine

db
-------
## Host Process:
CPU Usage (%):        19
Raw CPU Usage (%):    932.19
Working Set (MB):     363
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        95
Raw CPU Usage (%):    2,648.83
Working Set (MB):     512
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     36
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         397,234
Requests:             5,998,115
Mean latency (ms):    1.38
Max latency (ms):     32.22
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.18
Latency 75th (ms):    1.45
Latency 90th (ms):    1.97
Latency 99th (ms):    5.45

--scenario updates --profile master --profile citrine

db
-------
## Host Process:
CPU Usage (%):        51
Raw CPU Usage (%):    2,459.05
Working Set (MB):     484
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        85
Raw CPU Usage (%):    2,392.32
Working Set (MB):     518
Build Time (ms):      4,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         17,331
Requests:             261,691
Mean latency (ms):    31.34
Max latency (ms):     293.16
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    24.33
Latency 75th (ms):    35.58
Latency 90th (ms):    55.78
Latency 99th (ms):    100.37

--scenario updates --profile pr --profile citrine

db
-------
## Host Process:
CPU Usage (%):        56
Raw CPU Usage (%):    2,666.49
Working Set (MB):     470
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        88
Raw CPU Usage (%):    2,455.37
Working Set (MB):     510
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         17,273
Requests:             260,750
Mean latency (ms):    31.39
Max latency (ms):     247.32
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    24.41
Latency 75th (ms):    35.79
Latency 90th (ms):    55.36
Latency 99th (ms):    99.74

--scenario single_query --profile master --profile citrine

db
-------
## Host Process:
CPU Usage (%):        33
Raw CPU Usage (%):    1,569.85
Working Set (MB):     405
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,767.50
Working Set (MB):     498
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         367,108
Requests:             5,543,032
Mean latency (ms):    1.69
Max latency (ms):     103.19
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.23
Latency 75th (ms):    1.62
Latency 90th (ms):    2.30
Latency 99th (ms):    12.51

--scenario single_query --profile pr --profile citrine

db
-------
## Host Process:
CPU Usage (%):        33
Raw CPU Usage (%):    1,562.56
Working Set (MB):     417
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,773.17
Working Set (MB):     496
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         369,740
Requests:             5,582,366
Mean latency (ms):    1.66
Max latency (ms):     117.19
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.22
Latency 75th (ms):    1.60
Latency 90th (ms):    2.28
Latency 99th (ms):    11.64

--scenario single_query --profile multiplexing --profile master --profile citrine

db
-------
## Host Process:
CPU Usage (%):        18
Raw CPU Usage (%):    850.92
Working Set (MB):     445
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        98
Raw CPU Usage (%):    2,748.24
Working Set (MB):     506
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     36
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         411,254
Requests:             6,209,791
Mean latency (ms):    1.29
Max latency (ms):     45.13
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.19
Latency 75th (ms):    1.31
Latency 90th (ms):    1.50
Latency 99th (ms):    3.99

--scenario single_query --profile multiplexing --profile pr --profile citrine

db
-------
## Host Process:
CPU Usage (%):        19
Raw CPU Usage (%):    918.42
Working Set (MB):     458
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        98
Raw CPU Usage (%):    2,737.21
Working Set (MB):     513
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         417,992
Requests:             6,312,109
Mean latency (ms):    1.27
Max latency (ms):     45.64
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.17
Latency 75th (ms):    1.29
Latency 90th (ms):    1.47
Latency 99th (ms):    4.03

--scenario multiple_queries --profile master --profile citrine

db
-------
## Host Process:
CPU Usage (%):        56
Raw CPU Usage (%):    2,699.38
Working Set (MB):     401
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,758.76
Working Set (MB):     503
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         38,372
Requests:             579,391
Mean latency (ms):    16.26
Max latency (ms):     696.40
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    12.19
Latency 75th (ms):    13.97
Latency 90th (ms):    17.24
Latency 99th (ms):    122.57

--scenario multiple_queries --profile pr --profile citrine

db
-------
## Host Process:
CPU Usage (%):        56
Raw CPU Usage (%):    2,681.85
Working Set (MB):     402
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,763.13
Working Set (MB):     507
Build Time (ms):      3,501
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         38,049
Requests:             574,530
Mean latency (ms):    14.67
Max latency (ms):     447.06
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    12.48
Latency 75th (ms):    14.31
Latency 90th (ms):    16.92
Latency 99th (ms):    68.57

--scenario plaintext --profile master --profile perf

application
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    1,158.88
Working Set (MB):     113
Build Time (ms):      5,501
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      7,502
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         7,272,838
Requests:             109,794,144
Mean latency (ms):    6.34
Max latency (ms):     113.60
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    2.42
Latency 75th (ms):    7.94
Latency 90th (ms):    17.58
Latency 99th (ms):    45.53

--scenario plaintext --profile pr --profile perf

application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    1,157.07
Working Set (MB):     114
Build Time (ms):      4,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         7,296,602
Requests:             110,133,008
Mean latency (ms):    7.00
Max latency (ms):     123.62
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    2.50
Latency 75th (ms):    8.47
Latency 90th (ms):    19.67
Latency 99th (ms):    52.57

--scenario json --profile master --profile perf

application
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    1,167.39
Working Set (MB):     164
Build Time (ms):      4,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         561,800
Requests:             8,481,768
Mean latency (ms):    2.03
Max latency (ms):     69.36
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.58
Latency 75th (ms):    1.56
Latency 90th (ms):    5.84
Latency 99th (ms):    18.05

--scenario json --profile pr --profile perf

application
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    1,166.85
Working Set (MB):     164
Build Time (ms):      4,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         568,684
Requests:             8,586,725
Mean latency (ms):    1.80
Max latency (ms):     60.87
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.67
Latency 75th (ms):    1.10
Latency 90th (ms):    4.58
Latency 99th (ms):    16.85

--scenario fortunes --profile master --profile perf

db
-------
## Host Process:
CPU Usage (%):        89
Raw CPU Usage (%):    1,072.04
Working Set (MB):     587
Build Time (ms):      4,002
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    1,155.93
Working Set (MB):     555
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         133,780
Requests:             2,018,666
Mean latency (ms):    4.12
Max latency (ms):     167.88
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    3.55
Latency 75th (ms):    4.31
Latency 90th (ms):    5.56
Latency 99th (ms):    15.10

--scenario fortunes --profile pr --profile perf

db
-------
## Host Process:
CPU Usage (%):        90
Raw CPU Usage (%):    1,083.89
Working Set (MB):     347
Build Time (ms):      4,001
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    1,152.57
Working Set (MB):     403
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     36
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         135,159
Requests:             2,040,783
Mean latency (ms):    4.22
Max latency (ms):     179.73
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    3.47
Latency 75th (ms):    4.26
Latency 90th (ms):    5.63
Latency 99th (ms):    16.72

--scenario updates --profile master --profile perf

db
-------
## Host Process:
CPU Usage (%):        80
Raw CPU Usage (%):    964.12
Working Set (MB):     517
Build Time (ms):      4,001
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        79
Raw CPU Usage (%):    944.04
Working Set (MB):     519
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         3,876
Requests:             58,485
Mean latency (ms):    140.87
Max latency (ms):     1,710.00
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    108.49
Latency 75th (ms):    164.57
Latency 90th (ms):    246.01
Latency 99th (ms):    500.64

--scenario updates --profile pr --profile perf

db
-------
## Host Process:
CPU Usage (%):        87
Raw CPU Usage (%):    1,045.26
Working Set (MB):     717
Build Time (ms):      4,002
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        79
Raw CPU Usage (%):    952.23
Working Set (MB):     632
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         3,946
Requests:             59,586
Mean latency (ms):    137.62
Max latency (ms):     1,810.00
Bad responses:        0
Socket errors:        3
Latency 50th (ms):    107.17
Latency 75th (ms):    161.81
Latency 90th (ms):    233.10
Latency 99th (ms):    494.23

--scenario single_query --profile master --profile perf

db
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    1,189.90
Working Set (MB):     508
Build Time (ms):      4,001
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    1,161.95
Working Set (MB):     512
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         133,933
Requests:             2,022,304
Mean latency (ms):    3.96
Max latency (ms):     104.35
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    3.69
Latency 75th (ms):    4.24
Latency 90th (ms):    5.04
Latency 99th (ms):    11.53

--scenario single_query --profile pr --profile perf

db
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    1,185.29
Working Set (MB):     481
Build Time (ms):      4,002
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    1,160.20
Working Set (MB):     473
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         132,185
Requests:             1,995,985
Mean latency (ms):    3.95
Max latency (ms):     99.69
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    3.81
Latency 75th (ms):    4.27
Latency 90th (ms):    4.90
Latency 99th (ms):    8.69

--scenario multiple_queries --profile master --profile perf

db
-------
## Host Process:
CPU Usage (%):        100
Raw CPU Usage (%):    1,198.97
Working Set (MB):     457
Build Time (ms):      4,002
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        95
Raw CPU Usage (%):    1,144.58
Working Set (MB):     487
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         6,565
Requests:             99,125
Mean latency (ms):    77.82
Max latency (ms):     185.37
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    79.43
Latency 75th (ms):    84.54
Latency 90th (ms):    95.56
Latency 99th (ms):    115.72

--scenario multiple_queries --profile pr --profile perf

db
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    1,191.61
Working Set (MB):     366
Build Time (ms):      4,001
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        94
Raw CPU Usage (%):    1,132.75
Working Set (MB):     443
Build Time (ms):      4,001
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         6,897
Requests:             104,077
Mean latency (ms):    73.97
Max latency (ms):     234.57
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    73.79
Latency 75th (ms):    79.90
Latency 90th (ms):    103.46
Latency 99th (ms):    144.43

--scenario plaintext --profile master --profile amd

application
-------
## Host Process:
CPU Usage (%):        82
Raw CPU Usage (%):    3,944.93
Working Set (MB):     144
Build Time (ms):      7,001
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,501
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         9,946,887
Requests:             150,197,888
Mean latency (ms):    4.13
Max latency (ms):     840.82
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.96
Latency 75th (ms):    1.69
Latency 90th (ms):    3.48
Latency 99th (ms):    74.43

--scenario plaintext --profile pr --profile amd

application
-------
## Host Process:
CPU Usage (%):        83
Raw CPU Usage (%):    3,986.86
Working Set (MB):     145
Build Time (ms):      3,500
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         10,034,846
Requests:             151,516,678
Mean latency (ms):    4.99
Max latency (ms):     458.03
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.00
Latency 75th (ms):    1.93
Latency 90th (ms):    5.49
Latency 99th (ms):    105.74

--scenario json --profile master --profile amd

application
-------
## Host Process:
CPU Usage (%):        71
Raw CPU Usage (%):    3,388.59
Working Set (MB):     393
Build Time (ms):      3,500
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         620,475
Requests:             9,368,560
Mean latency (ms):    1.02
Max latency (ms):     66.70
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.77
Latency 75th (ms):    1.04
Latency 90th (ms):    1.43
Latency 99th (ms):    6.19

--scenario json --profile pr --profile amd

application
-------
## Host Process:
CPU Usage (%):        72
Raw CPU Usage (%):    3,470.56
Working Set (MB):     401
Build Time (ms):      3,500
Published Size (KB):  101,671
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         636,834
Requests:             9,616,080
Mean latency (ms):    1.16
Max latency (ms):     51.92
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.77
Latency 75th (ms):    1.03
Latency 90th (ms):    1.45
Latency 99th (ms):    12.57

--scenario fortunes --profile master --profile amd

db
-------
## Host Process:
CPU Usage (%):        78
Raw CPU Usage (%):    2,184.69
Working Set (MB):     695
Build Time (ms):      9,002
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        86
Raw CPU Usage (%):    4,124.59
Working Set (MB):     3,866
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         334,740
Requests:             5,048,108
Mean latency (ms):    2.57
Max latency (ms):     269.62
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.44
Latency 75th (ms):    2.07
Latency 90th (ms):    3.01
Latency 99th (ms):    32.54

--scenario fortunes --profile pr --profile amd

db
-------
## Host Process:
CPU Usage (%):        90
Raw CPU Usage (%):    2,517.90
Working Set (MB):     699
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        85
Raw CPU Usage (%):    4,079.06
Working Set (MB):     3,808
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         372,317
Requests:             5,621,830
Mean latency (ms):    3.28
Max latency (ms):     199.04
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.19
Latency 75th (ms):    1.88
Latency 90th (ms):    4.00
Latency 99th (ms):    54.16

--scenario updates --profile master --profile amd

db
-------
## Host Process:
CPU Usage (%):        91
Raw CPU Usage (%):    2,538.64
Working Set (MB):     604
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        37
Raw CPU Usage (%):    1,792.44
Working Set (MB):     3,730
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         13,145
Requests:             198,246
Mean latency (ms):    40.34
Max latency (ms):     299.48
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    33.08
Latency 75th (ms):    45.03
Latency 90th (ms):    67.29
Latency 99th (ms):    116.21

--scenario updates --profile pr --profile amd

db
-------
## Host Process:
CPU Usage (%):        90
Raw CPU Usage (%):    2,521.69
Working Set (MB):     673
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        38
Raw CPU Usage (%):    1,816.27
Working Set (MB):     3,517
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         12,904
Requests:             194,613
Mean latency (ms):    41.99
Max latency (ms):     296.94
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    33.29
Latency 75th (ms):    46.25
Latency 90th (ms):    72.47
Latency 99th (ms):    135.04

--scenario single_query --profile master --profile amd

db
-------
## Host Process:
CPU Usage (%):        61
Raw CPU Usage (%):    1,719.66
Working Set (MB):     704
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        61
Raw CPU Usage (%):    2,919.10
Working Set (MB):     3,856
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         283,663
Requests:             4,283,165
Mean latency (ms):    3.42
Max latency (ms):     156.30
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.75
Latency 75th (ms):    2.08
Latency 90th (ms):    2.57
Latency 99th (ms):    70.46

--scenario single_query --profile pr --profile amd

db
-------
## Host Process:
CPU Usage (%):        62
Raw CPU Usage (%):    1,727.24
Working Set (MB):     709
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        61
Raw CPU Usage (%):    2,938.68
Working Set (MB):     3,884
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         287,854
Requests:             4,346,360
Mean latency (ms):    3.01
Max latency (ms):     154.99
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.75
Latency 75th (ms):    2.08
Latency 90th (ms):    2.57
Latency 99th (ms):    41.87

--scenario multiple_queries --profile master --profile amd

db
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    2,718.89
Working Set (MB):     703
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        52
Raw CPU Usage (%):    2,492.54
Working Set (MB):     3,775
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         22,471
Requests:             339,306
Mean latency (ms):    23.32
Max latency (ms):     180.66
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    22.22
Latency 75th (ms):    24.44
Latency 90th (ms):    27.10
Latency 99th (ms):    52.00

--scenario multiple_queries --profile pr --profile amd

db
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    2,776.39
Working Set (MB):     573
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        51
Raw CPU Usage (%):    2,458.95
Working Set (MB):     3,824
Build Time (ms):      3,500
Published Size (KB):  101,673
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     36
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         22,856
Requests:             345,074
Mean latency (ms):    22.59
Max latency (ms):     135.02
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    21.97
Latency 75th (ms):    23.47
Latency 90th (ms):    24.96
Latency 99th (ms):    36.25

--scenario plaintext --profile master --profile arm

application
-------
## Host Process:
CPU Usage (%):        98
Raw CPU Usage (%):    3,148.83
Working Set (MB):     119
Build Time (ms):      20,011
Published Size (KB):  110,305
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     36
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         7,515,835
Requests:             113,486,283
Mean latency (ms):    3.59
Max latency (ms):     424.31
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.15
Latency 75th (ms):    2.27
Latency 90th (ms):    5.31
Latency 99th (ms):    25.95

--scenario plaintext --profile pr --profile arm

application
-------
## Host Process:
CPU Usage (%):        98
Raw CPU Usage (%):    3,149.41
Working Set (MB):     121
Build Time (ms):      10,006
Published Size (KB):  110,305
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         7,625,430
Requests:             115,143,019
Mean latency (ms):    3.21
Max latency (ms):     464.26
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    1.12
Latency 75th (ms):    2.01
Latency 90th (ms):    4.65
Latency 99th (ms):    19.19

--scenario json --profile master --profile arm

application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    3,164.08
Working Set (MB):     212
Build Time (ms):      10,006
Published Size (KB):  110,305
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         534,588
Requests:             8,072,041
Mean latency (ms):    1.92
Max latency (ms):     416.36
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.70
Latency 75th (ms):    1.13
Latency 90th (ms):    3.34
Latency 99th (ms):    13.96

--scenario json --profile pr --profile arm

application
-------
## Host Process:
CPU Usage (%):        99
Raw CPU Usage (%):    3,160.21
Working Set (MB):     209
Build Time (ms):      10,505
Published Size (KB):  110,305
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         565,167
Requests:             8,527,061
Mean latency (ms):    1.57
Max latency (ms):     216.45
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    0.70
Latency 75th (ms):    1.00
Latency 90th (ms):    2.54
Latency 99th (ms):    10.78

--scenario fortunes --profile master --profile arm

db
-------
## Host Process:
CPU Usage (%):        7
Raw CPU Usage (%):    329.03
Working Set (MB):     393
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        95
Raw CPU Usage (%):    3,029.14
Working Set (MB):     415
Build Time (ms):      10,006
Published Size (KB):  110,306
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,501
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         90,891
Requests:             1,372,462
Mean latency (ms):    6.57
Max latency (ms):     390.00
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    3.82
Latency 75th (ms):    8.51
Latency 90th (ms):    13.52
Latency 99th (ms):    30.98

--scenario fortunes --profile pr --profile arm

db
-------
## Host Process:
CPU Usage (%):        7
Raw CPU Usage (%):    348.45
Working Set (MB):     389
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    3,062.87
Working Set (MB):     411
Build Time (ms):      10,003
Published Size (KB):  110,306
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.66
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         95,305
Requests:             1,439,046
Mean latency (ms):    7.31
Max latency (ms):     860.53
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    3.63
Latency 75th (ms):    7.69
Latency 90th (ms):    12.35
Latency 99th (ms):    29.51

--scenario updates --profile master --profile arm

db
-------
## Host Process:
CPU Usage (%):        37
Raw CPU Usage (%):    1,759.55
Working Set (MB):     468
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        92
Raw CPU Usage (%):    2,952.31
Working Set (MB):     466
Build Time (ms):      10,005
Published Size (KB):  110,306
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         12,486
Requests:             188,329
Mean latency (ms):    41.99
Max latency (ms):     494.69
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    34.18
Latency 75th (ms):    47.11
Latency 90th (ms):    67.32
Latency 99th (ms):    151.31

--scenario updates --profile pr --profile arm

db
-------
## Host Process:
CPU Usage (%):        37
Raw CPU Usage (%):    1,753.74
Working Set (MB):     465
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        92
Raw CPU Usage (%):    2,942.12
Working Set (MB):     476
Build Time (ms):      10,006
Published Size (KB):  110,306
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         12,657
Requests:             190,974
Mean latency (ms):    41.70
Max latency (ms):     439.16
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    33.96
Latency 75th (ms):    46.40
Latency 90th (ms):    65.32
Latency 99th (ms):    164.01

--scenario single_query --profile master --profile arm

db
-------
## Host Process:
CPU Usage (%):        10
Raw CPU Usage (%):    472.34
Working Set (MB):     401
Build Time (ms):      3,501
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        97
Raw CPU Usage (%):    3,088.42
Working Set (MB):     405
Build Time (ms):      9,507
Published Size (KB):  110,306
Swap (MB):            0
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         121,482
Requests:             1,834,302
Mean latency (ms):    5.23
Max latency (ms):     474.26
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    2.91
Latency 75th (ms):    6.34
Latency 90th (ms):    11.36
Latency 99th (ms):    25.36

--scenario single_query --profile pr --profile arm

db
-------
## Host Process:
CPU Usage (%):        10
Raw CPU Usage (%):    503.66
Working Set (MB):     405
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    3,060.86
Working Set (MB):     401
Build Time (ms):      10,004
Published Size (KB):  110,306
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         119,947
Requests:             1,811,249
Mean latency (ms):    5.30
Max latency (ms):     766.88
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    2.89
Latency 75th (ms):    5.89
Latency 90th (ms):    10.90
Latency 99th (ms):    26.15

--scenario multiple_queries --profile master --profile arm

db
-------
## Host Process:
CPU Usage (%):        24
Raw CPU Usage (%):    1,152.03
Working Set (MB):     403
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    3,073.34
Working Set (MB):     418
Build Time (ms):      10,005
Published Size (KB):  110,306
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.00
Working Set (MB):     35
Build Time (ms):      3,000
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         14,005
Requests:             211,140
Mean latency (ms):    41.73
Max latency (ms):     782.59
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    31.17
Latency 75th (ms):    44.54
Latency 90th (ms):    57.63
Latency 99th (ms):    299.99

--scenario multiple_queries --profile pr --profile arm

db
-------
## Host Process:
CPU Usage (%):        25
Raw CPU Usage (%):    1,215.64
Working Set (MB):     403
Build Time (ms):      3,500
Swap (MB):            0
application
-------
## Host Process:
CPU Usage (%):        96
Raw CPU Usage (%):    3,071.85
Working Set (MB):     418
Build Time (ms):      10,005
Published Size (KB):  110,306
Swap (MB):            15
load
-------
## Host Process:
CPU Usage (%):        0
Raw CPU Usage (%):    0.67
Working Set (MB):     35
Build Time (ms):      3,001
Published Size (KB):  80,011
Swap (MB):            0
## Benchmark:
Requests/sec:         14,359
Requests:             216,663
Mean latency (ms):    42.05
Max latency (ms):     1,460.00
Bad responses:        0
Socket errors:        0
Latency 50th (ms):    31.47
Latency 75th (ms):    44.23
Latency 90th (ms):    55.19
Latency 99th (ms):    302.05

stephentoub · 2020-06-10T22:06:57Z

Thanks for confirming, @adamsitnik!

stephentoub · 2020-06-11T17:28:25Z

@tmds, fyi, this now has conflicts from the token removal change.

tmds · 2020-06-16T08:57:17Z

@tmds, fyi, this now has conflicts from the token removal change.

@stephentoub #fixed.

adamsitnik · 2020-06-16T11:51:24Z

The Windows CI leg failed with following error:

C:\h\w\BC7E0A14\w\A2EF0929\e>"C:\h\w\BC7E0A14\p\dotnet.exe" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Net.Sockets.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Sockets.Tests (found 842 of 1201 test cases)
  Starting:    System.Net.Sockets.Tests (parallel test collections = on, max threads = 2)
    System.Net.Sockets.Tests.KeepAliveTest.Socket_KeepAlive_RetryCount_Success [SKIP]
      Condition(s) not met: "IsUnixOrWindowsAtLeast1703"
    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_ConnectAsyncUnixDomainSocketEndPoint_Success [FAIL]
      System.PlatformNotSupportedException : Operation is not supported on this platform.
      Stack Trace:
        /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/UnixDomainSocketEndPoint.cs(49,0): at System.Net.Sockets.UnixDomainSocketEndPoint..ctor(String path)
        /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/UnixDomainSocketTest.cs(43,0): at System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_ConnectAsyncUnixDomainSocketEndPoint_Success()
        --- End of stack trace from previous location ---
    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_SendReceive_Success [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_ConnectAsyncUnixDomainSocketEndPoint_NotServer [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_SendReceive_Clone_Success [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.UnixDomainSocketTest.UnixDomainSocketEndPoint_RemoteEndPointEqualsBindAddress [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.UnixDomainSocketTest.ConcurrentSendReceiveAsync [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.UnixDomainSocketTest.UnixDomainSocketEndPoint_InvalidPaths_Throws [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_SendReceiveAsync_Success [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"
    System.Net.Sockets.Tests.SocketOptionNameTest.ReuseUnicastPort_CreateSocketGetOption [SKIP]
      Condition(s) not met: "SocketsReuseUnicastPortSupport"
    System.Net.Sockets.Tests.SocketOptionNameTest.ReuseUnicastPort_CreateSocketSetOption [SKIP]
      Condition(s) not met: "SocketsReuseUnicastPortSupport"
  Finished:    System.Net.Sockets.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Sockets.Tests  Total: 1131, Errors: 0, Failed: 1, Skipped: 10, Time: 27.305s
----- end Tue 06/16/2020  9:51:01.07 ----- exit code 1 ----------------------------------------------------------

Is it related to this PR (I doubt so)?

stephentoub · 2020-06-16T11:53:06Z

Is it related to this PR (I doubt so)?

This PR was previously failing with that error, and I've not seen that error on other PRs... so it's likely from this somehow (though I don't see how).

tmds · 2020-06-16T12:10:46Z

I don't see how the test failure is related to the PR. The test should actually be skipped:

Some tests get skipped because PlatformSupportsUnixDomainSockets condition is not met:

    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_SendReceive_Success [SKIP]
      Condition(s) not met: "PlatformSupportsUnixDomainSockets"

but not Socket_ConnectAsyncUnixDomainSocketEndPoint_Success:

runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/UnixDomainSocketTest.cs

Lines 33 to 34 in bab2bc4

    
           [ConditionalFact(nameof(PlatformSupportsUnixDomainSockets))] 
        
           public async Task Socket_ConnectAsyncUnixDomainSocketEndPoint_Success()

which throws PNSE while executing:

System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_ConnectAsyncUnixDomainSocketEndPoint_Success [FAIL]
      System.PlatformNotSupportedException : Operation is not supported on this platform.
      Stack Trace:
        /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/UnixDomainSocketEndPoint.cs(49,0): at System.Net.Sockets.UnixDomainSocketEndPoint..ctor(String path)
        /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/UnixDomainSocketTest.cs(43,0): at System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_ConnectAsyncUnixDomainSocketEndPoint_Success()

tmds · 2020-06-16T12:12:37Z

Can you trigger another CI run, to see if it reproduces?

stephentoub · 2020-06-16T12:22:15Z

Triggered

stephentoub · 2020-06-16T14:09:47Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SafeSocketHandle.cs

+        // PreferInlineCompletions defaults to false and can be set to true using the DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS envvar.
+        internal static readonly bool InlineSocketCompletionsEnabled = Environment.GetEnvironmentVariable("DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS") == "1";
+
+        internal bool PreferInlineCompletions { get; set; } = InlineSocketCompletionsEnabled;


I just noticed these changes are in SafeSocketHandle.cs rather than SafeSocketHandle.Unix.cs; same for the ones below in Socket.cs rather than Socket.Unix.cs. Can they be moved to the Unix-specific files?

stephentoub · 2020-06-16T14:10:39Z

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs

 // The .NET Foundation licenses this file to you under the MIT license.
 // See the LICENSE file in the project root for more information.

+using System;


Was this necessary? If yes, can you move it down a line? But I'd also have expected it to not be needed, since everything below is in the System.Net.Sockets namespace, and thus System should be implicitly included.

stephentoub · 2020-06-16T14:17:11Z

@tmds, maybe try opening a new PR? That shouldn't help, but, the failure is very strange.

tmds · 2020-06-16T16:05:40Z

I've addressed the comment, and created a new PR: #37974.

Dotnet-GitSync-Bot added the area-System.Net.Sockets label Apr 14, 2020

am11 reviewed Apr 14, 2020

View reviewed changes

src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive.cs Outdated Show resolved Hide resolved

stephentoub reviewed Apr 14, 2020

View reviewed changes

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs Outdated Show resolved Hide resolved

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs Outdated Show resolved Hide resolved

adamsitnik reviewed Apr 14, 2020

View reviewed changes

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs Outdated Show resolved Hide resolved

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs Outdated Show resolved Hide resolved

lpereira reviewed Apr 14, 2020

View reviewed changes

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs Outdated Show resolved Hide resolved

tmds force-pushed the inline_socket branch from 5b11fb1 to ffcedd0 Compare April 16, 2020 14:58

davidfowl reviewed Apr 16, 2020

View reviewed changes

tmds mentioned this pull request Apr 17, 2020

Support inline application scheduling mode dotnet/aspnetcore#20952

Closed

stephentoub reviewed Apr 20, 2020

View reviewed changes

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs Outdated Show resolved Hide resolved

tmds force-pushed the inline_socket branch from 17732c5 to 16b6df0 Compare May 12, 2020 19:45

stephentoub reviewed Jun 2, 2020

View reviewed changes

src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs Show resolved Hide resolved

stephentoub approved these changes Jun 2, 2020

View reviewed changes

Don't use Unix specific SocketAsyncEngine in common SafeHandle code

674d7eb

stephentoub approved these changes Jun 8, 2020

View reviewed changes

adamsitnik approved these changes Jun 10, 2020

View reviewed changes

stephentoub closed this Jun 10, 2020

stephentoub reopened this Jun 10, 2020

Merge remote-tracking branch 'upstream/master' into inline_socket

3505eed

stephentoub reviewed Jun 16, 2020

View reviewed changes

PR feedback

6ea1256

tmds closed this Jun 16, 2020

tmds mentioned this pull request Jun 16, 2020

Unix: add mode to inline Socket continuations #37974

Merged

antonfirsov added a commit to antonfirsov/runtime that referenced this pull request Jun 17, 2020

adapt dotnet#34945 / dotnet#37974

93b15c2

antonfirsov mentioned this pull request Jun 17, 2020

Move epoll event handling to a non-inlined method #37138

Merged

halter73 mentioned this pull request Jul 1, 2020

Add inline scheduling mode to Kestrel's Socket Transport dotnet/aspnetcore#23591

Closed

Rodrigo-Andrade mentioned this pull request Jul 25, 2020

Performance regression in preview 4 to 7 dotnet/aspnetcore#24199

Closed

karelz added this to the 5.0.0 milestone Aug 18, 2020

ghost locked as resolved and limited conversation to collaborators Dec 9, 2020

Unix: add mode to inline Socket continuations #34945

Unix: add mode to inline Socket continuations #34945

Uh oh!

Conversation

tmds commented Apr 14, 2020

Uh oh!

ghost commented Apr 14, 2020

Uh oh!

tmds commented Apr 14, 2020

Uh oh!

Uh oh!

tmds commented Apr 14, 2020

Uh oh!

stephentoub commented Apr 14, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub commented Apr 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidfowl commented Apr 14, 2020

Uh oh!

stephentoub commented Apr 14, 2020

Uh oh!

davidfowl commented Apr 14, 2020

Uh oh!

stephentoub commented Apr 14, 2020

Uh oh!

scalablecory commented Apr 14, 2020

Uh oh!

stephentoub commented Apr 14, 2020

Uh oh!

davidfowl commented Apr 14, 2020

Uh oh!

stephentoub commented Apr 14, 2020

Uh oh!

davidfowl commented Apr 14, 2020

Uh oh!

Uh oh!

tmds commented Apr 15, 2020

Uh oh!

tmds commented Apr 15, 2020

Uh oh!

stephentoub commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmds commented Apr 16, 2020

Uh oh!

davidfowl Apr 16, 2020

Choose a reason for hiding this comment

Uh oh!

tmds Apr 16, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tmds commented May 12, 2020

Uh oh!

Uh oh!

stephentoub left a comment

Choose a reason for hiding this comment

Uh oh!

tmds commented Jun 2, 2020

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

stephentoub commented Jun 10, 2020

Uh oh!

stephentoub commented Jun 11, 2020

Uh oh!

tmds commented Jun 16, 2020

Uh oh!

adamsitnik commented Jun 16, 2020

Uh oh!

stephentoub commented Jun 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmds commented Jun 16, 2020

stephentoub commented Apr 14, 2020 •

edited

Loading

stephentoub commented Apr 15, 2020 •

edited

Loading

stephentoub commented Jun 16, 2020 •

edited

Loading