Sporadic loss of connectivity with 2.0.601

[paveliak]

Hi Marc

We are running ASP.NET services in Azure and use Azure Redis. After upgrading one of our services to 2.0.601 we started observing sporadic loss of connectivity. Our VMs maintain several Redis connections (using several multiplexers). They all share same (default) connection manager.

Occasionally one of multiplexers loses the connection and never recovers. We have to recycle app pool to bring it back to life. When this happens I see few errors in our logs (these are coming from ConnectionMultiplexer events) - InternalError followed by ConnectionFailed and ConfigurationChanged. All subsequent Redis requests then fail with "No connection is available to service this operation" error.

Here are the events coming from the ConnectionMultiplexer.

InternalError: endpoint=Unspecified/CACHENAME.redis.cache.windows.net:6380, connectionType=Interactive, origin=KeepAlive, exception=StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: PING; IOCP: (Busy=1,Free=999,Min=16,Max=1000), WORKER: (Busy=5,Free=32762,Min=48,Max=32767), Local-CPU: 29.7%
ConnectionFailed: endpoint=Unspecified/CACHENAME.redis.cache.windows.net:6380, connectionType=Interactive, failureType=SocketClosed, exception=StackExchange.Redis.RedisConnectionException: SocketClosed on CACHENAME.redis.cache.windows.net:6380/Interactive, Idle/MarkProcessed, last: EVAL, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 58s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402
ConfigurationChanged: endpoint=Unspecified/CACHENAME.redis.cache.windows.net:6380

Let me know if I can collect more data for you.

Thanks,
Pavel

[keithl8041]

You may want to tag @mgravell so he's aware of your post.

FWIW we're seeing the same issue, also using Azure Redis on 2.0.601.

[programcsharp]

Was just coming here to upgrade, guess I'll hold off for now 😁

Is 2.0.593 stable?

[vinayaktavildar]

Hi @mgravell ,

We also run ASP.NET service and use Azure Redis and we are planning to upgrade to 2.0.601 as we are facing lot of issues with older client version (1.2.6)
What do you suggest? Should we wait till this issue is fixed or should we upgrade to a version lower than 2.0.601?

Thanks,
Vinayak

[paveliak]

@mgravell just wanted to let you know that it happened again. Single multiplexer on a VM lost connection (a dozen more on the same VM keep working just fine). Symptoms are the same as described earlier.

FWIW first request going through this multiplexer fails with RedisConnectionException"An unknown error occurred when writing the message". Subsequent requests report StackExchange.Redis.RedisConnectionException "No connection is available to service this operation"

[mstavrev]

I believe, this is same described on #1147 and #1120.

[mts57017]

@mgravell i am facing the same issue as well, along with "No connection is available to service this operation" randomly

[ariel-balisi-bcs]

Any update on this issue? We're also encountering this errors.

[BoAlexander]

We are getting ready to deploy an upgrade using the following stack, which includes StackExchange.Redis, and I wondered if anyone had an update on the problem listed in this thread that might concern the reliability of my plan:

Microsoft.Web.RedisSessionStateProvider v4.0.1
StackExchange.Redis.2.0.601
Pipelines.Sockets.Unofficial 2.0.25
System.IO.Pipelines 4.5.1
System.Memory 4.5.3

Should I expect problems with 2.0.601?

Thanks in advance.
Bo

[AlexeiScherbakov]

2.0.601.3402 has this problem

[faiyazulnoor]

2.0.601 has issues**
We are using it in production and getting multiple errors like below making web app throw 500 on each request due to burst of these errors. First, we thought is due to load on server CPU usage etc seems below 25 %; Azure Redis Standard 2.5 GB we are using and peak load I can see Maximum 250 MB is used. We are using code like below and can see concurrent not more than 100 requests of GET, SET and Exist is being called.
Investigating as have one patch from Sitecore https://kb.sitecore.net/articles/464570 regarding similar by working on the thread. Looking at some comments online to downgrade to lower version or use StackExchange.Redis.StrongName" version="1.2.6"

Any suggestion ?

private static readonly Lazy<ConnectionMultiplexer> Redis = new Lazy<ConnectionMultiplexer>(
            () => ConnectionMultiplexer.Connect(ConnString));
        
		//To check key exist or NOT
		Redis.Value.GetDatabase().KeyExists("Key");
		//Set Key with serializedObject
		Redis.Value.StringSet(key, serializedObject, timespan);
		//Get Key
		Redis.ValueStringGet(key);

• An unknown error occurred when writing the message
  Source: StackExchange.Redis
  at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2235

• An existing connection was forcibly closed by the remote host; IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=5,Free=32762,Min=50,Max=32767), Local-CPU: n/a
  Nested Exception
  Exception: StackExchange.Redis.RedisConnectionException
  Message: SocketFailure on <website*********>.windows.net:6380/Interactive, Flushing/Faulted, last: SETEX, origin: ReadFromPipe, outstanding: 1, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 8 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402

• System.IO.IOException
  Message: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
  Source: System
  at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
  at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
  --- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at Pipelines.Sockets.Unofficial.StreamConnection.AsyncStreamPipe.d__13.MoveNext() in C:\Code\Pipelines.Sockets.Unofficial\src\Pipelines.Sockets.Unofficial\StreamConnection.AsyncStreamPipe.cs:line 97

[NickCraver]

This issue is finally tracked down in #1374 - going to close this out to fold the conversations in. Look out for a 2.1.x release soon as we verify and get that behavior change in.