NOAUTH Authentication required ERROR after ElastiCache patch

[bdlee420]

Redis Setup

1. ElastiCache Redis
2. Clustered
3. SSL
4. Version 3.2.6
5. Redis AUTH = Yes

Client

1. C# .Net Core console application
2. StackExchange.Redis: 2.0.601

Occasionally there are AWS updates applied to our Redis instance for ElastiCache. When this updates applies, all of our clients that use the StackExchance.Redis library start to receive "StackExchange.Redis.RedisServerException: NOAUTH Authentication required." errors. We have to restart the clients and then everything works fine.

We have been unable to duplicate this issue outside of an ElastiCache service update.

Things we have tried to duplicate the issue:

1. Shutdown Redis and restart it
2. Upgrade the version
3. Change SSL settings back and forth

In all these cases, the local client reconnected with no problems.

We are at a lose as to how to duplicate this issue and then how to resolve this issue. One thought I had was to create a heartbeat inside each client to try to connect and read a key, if it fails, then rebuild the connection to Redis, but we are blind since we can't duplicate it on demand. Any help would be greatly appreciated.

Our Redis provider constructor

public RedisProvider(RedisConnectionSettings connectionSettings, ILogger logger)
        {
            options = new ConfigurationOptions()
            {
                Ssl = connectionSettings.SslEnabled,
                DefaultDatabase = connectionSettings.Database,
                Password = connectionSettings.Password,
                EndPoints = { connectionSettings.Connection }
            };

            try
            {
                connectionMultiplexer = ConnectionMultiplexer.Connect(options);
            }
            catch (Exception ex)
            {
                logger.Error(ex.Message);
            }
            keysPrefix = $"{connectionSettings.KeyPrefix}:{_redisApplicationName}";
}

Our method to get the Redis Database used in each call to Redis

private IDatabase GetRedisDataBase()
{
            if (connectionMultiplexer == null)
            {
                connectionMultiplexer = ConnectionMultiplexer.Connect(options);
            }

            var redisDatabase = connectionMultiplexer.GetDatabase();

            return redisDatabase;
}

[mgravell]

Huh, that's odd. And the config (string or object) contains the auth password? This is confusing, because this is literally the first thing we send. I wonder whether elasticache works via a TCP proxy that changes the backend without severing the client connection (just: passing payloads to the new endpoint). That's the only way I can imagine of causing this... We don't currently expect NOAUTH in an established connection.

[bdlee420]

Yes the config contains the password. That never changes. Currently I am building a solution that will essentially catch that error and then close and reopen the connection and then try again. But it is virtually impossible to test since we have to wait for an AWS patch to duplicate the error.

private async Task HashSetAsync(string key, HashEntry[] hashEntries)
        {
            await Retry.Do(
                () => _redisProvider.HashSetAsync(key, hashEntries),
                () => _redisProvider.Reconnect(),
                TimeSpan.FromSeconds(5));
        }

public Task Reconnect()
        {
            if (connectionMultiplexer == null)
            {
                connectionMultiplexer = ConnectionMultiplexer.Connect(options);
            }
            else
            {
                connectionMultiplexer.Close();
                connectionMultiplexer = ConnectionMultiplexer.Connect(options);
            }
            return null;
        }

[shimingwu]

Hi @bdlee420 , we are not experiencing an AWS patch but we receive NOAUTH Authentication required ERROR after one of our slave node restarts.
For us, we are using v2.0.519 and .Net Core Web application and our case is that:
One of the slave node fails, and after it recovers, our containers start to receive NOAUTH Authentication required. However, all containers start after the recover (which means they directly connect to the new slave node), they won't receive this error message.
I try to reproduce this as well, but somehow AWS does not provide me an action to fail a slave node (they only provide the failover action, which is different from this case). I tried to delete the slave node and add it back, it does not help to reproduce this error as well.
I am not sure whether this will be linked to your issue, but it seems like the connection is getting issue for both cases as I think for Redis AUTH, the password is required for first time when the connection is build..

[effyteva]

+1 on this issue, we suffer from it as well, following a cluster update on AWS

[NickCraver]

This may also be a result of #1120/#1374 - let's poke here with the 2.1.x release from #1374 and see if it indeed resolves it.

[NickCraver]

Can you please give this a try on the 2.1.0 release now on NuGet?

[GREsau]

We've also experienced this after performing a failover with our elasticache redis cluster, even after upgrading to StackExchange.Redis 2.1.28

The first errors we see are:
StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: GET [key]; It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly.

Then after a couple of minutes,
StackExchange.Redis.RedisServerException: NOAUTH Authentication required.

Redis Setup

1. ElastiCache Redis
2. Clustered
3. SSL
4. Engine Version Compatibility 5.0.4
5. Redis AUTH = Yes

Client

1. C# .Net Core 3.1 console application on Alpine 3.10
2. StackExchange.Redis: 2.0.601 and 2.1.28 (reproduced on both)

[VicaryM]

We also experience this NOAUTH error after Redis patch.
StackExchange.Redis.RedisServerException: NOAUTH Authentication required.

We are using Redis AUTH + SSL.
We upgraded our StackExchange.Redis version to 2.1.30 but still having the issue.

[ExileLee]

any update?

we experience same issue

StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: HGETALL gis-user-profit:20200510; It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly. ConnectTimeout; IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=0,Free=32767,Min=2,Max=32767), Local-CPU: n/a ---> System.AggregateException: One or more errors occurred. (It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly. ConnectTimeout) (UnableToConnect on xxxxx:6228/Interactive, Idle/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 0, last-read: 2s ago, last-write: 2s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: never, global: 3s ago, v: 2.0.601.3402)

Redis Setup

• ElastiCache Redis
• Clustered
• SSL
• Engine Version Compatibility 5.0.4
• Redis AUTH = Yes

Client

• AWS Lambda
• C# .Net Core 2.1
• StackExchange.Redis: 2.0.601

[mageziyx]

Hi @NickCraver , according to AWS this happens due to the restarted Redis engine accepting connections before its auth token is populated. Would it be possible to modify the client to reconnect once ping has auth issue, or not connect until the restarted redis is populated with auth token? Thanks!

[mgravell]

We can perhaps try, but the problem is: none of the cloud hosts have a magic "now do crazy stuff in the background without severing connections" button that we can test on demand to replicate and validate things. Which makes this incredibly frustrating to develop.

…

On Tue, 30 Jun 2020, 15:21 Tat Lee Tang, ***@***.***> wrote: Hi @NickCraver <https://github.com/NickCraver> , according to AWS this happens due to the restarted Redis engine accepting connections before its auth token is populated. Would it be possible to modify the client to reconnect once ping has auth issue, or not connect until the restarted redis is populated with auth token? Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1273 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEHMAPBW66UDCXWRFDH53RZHYGJANCNFSM4JNQGYFA> .

[NickCraver]

Thanks to @TimLovellSmith's investigation here, this has bee fixed in the 2.2.50 release now on NuGet. It took some fresh eyes to see what was happening here in environments with some latency and races in play. Please grab 2.2.50 to eliminate this race!