GRAPH.QUERY support on redis read replica

[raghubirt]

I have setup a Redis cluster with 1 master and 2 replicas to distribute reads.
When executing read query on replica, I got below error
"(error) READONLY You can't write against a read only replica."

Is there a way to achieve it or is it coming in next release?

[swilly22]

Hi @raghubirt,
Reason you're getting this error is due to how we've registered the GRAPH.QUERY command,
when extending redis via a redis-module by introducing new commands we have to specify if a command is able to modify redis key-space, in which case it is a write command or not, a read command, because GRAPH.QUERY process both read and write queries we had no choice but to register it as a write command, as such redis will not allow you to call it on a replica.

We might consider introducing a read only command which will process read only queries.

[styk-tv]

@swilly22 well this is a BUG., you surly know after pre-parsing of the Cypher that a query is reading or writing data.

Maybe follow GraphQL for:

GRAPH.QUERY - for read only
GRAPH.MUTATION - for write

EDIT: however splitting those seems silly

[swilly22]

@styk-tv, just by inspecting at the query (AST) we can tell if it's a read or write query,
with the latest release of Redis 5.0.4 we'll be able to scale out reads without extending our API.

[styk-tv]

@swilly22 so you're telling me I can use edge RedisGraph which is built daily therefore has pulled 5.0.4 as a starting layer? Let me try this.

[styk-tv]

@swilly22 I most likely misunderstood what you are saying or please mark this ticket as a BUG because your hint does not really work. Below was installed with latest RedisGraph:edge (0d3c64f)

8510:C 10 Apr 2019 07:48:47.506 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=8510, just started

and

  File "/usr/local/lib/python3.7/site-packages/redisgraph/client.py", line 155, in query
    response = self.redis_con.execute_command("GRAPH.QUERY", self.name, q)
  File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 775, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 789, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 642, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.

[swilly22]

@styk-tv, @raghubirt Please allow me clarify, I'll start from the beginning;
the command GRAPH.QUERY is marked with a write flag, see

As such Redis default behavior is to block GRAPH.QUERY commands regardless if the under line query performs only reads.

I could have introduced another command e.g. GRAPH.QUERYRO (Query Read Only) which isn't marked with the write flag, the command will verify by looking at the query (AST) that indeed the query doesn't modifies the graph in anyway, in which case GRAPH.QUERYRO can be executed against both master and slaves. but I would like to avoid extending RedisGraph API and keep its command list to a minimum.

To do so Redis 5.0.4 introduce a new redis module context flag which allows a slave to figure the origin of a command; either the command was sent from a user/client or the command was sent by its master. with that I can remove the write flag from GRAPH.QUERY and only execute write queries on slaves if the origin is their master (part of a replication link)

Going back to the default behavior, you can configure slaves to execute write commands by issuing
CONFIG SET slave-read-only no this will enable slaves to execute GRAPH.QUERY or any other write command for that matter, beware, sending write commands to slaves might cause inconsistency within your cluster, it is your responsibility to make sure write commands are sent to the current master.

[styk-tv]

@swilly22 i'm sending these to sentinels, are they in any way making decisions as to where to send the request?

[swilly22]

Sorry I'm not that familiar with Sentinel, my assumption is that you're aware of your cluster architecture master/slave IPs and able to connect directly to anyone of them.

[styk-tv]

https://redis.io/topics/sentinel Sorry as well, my assumption would be that you are familiar with hundreds of configurations and strategic scenarios that you could play back as part of tests when you work with a cluster composed out of 100+ redis nodes. Best of which would make it into redis kubernetes charts.

I suggest starting with 5-25TB of data into 100-200 nodes cluster and play a performance test that can produce traffic of 30Gb/s as a mix of 80% reads and 20% writes. Then start ChaosMonkey both on Node and Pod level and see how the perftest is reacting to the drop and reconstruction of nodes/pods. That will give you idea why tracking masters/slaves manually is not desired.

Point being as an architect last thing I want to do is to know where the masters/slaves are. I want to rely on Sentinel to know that as I enjoy sending all of that traffic into a single virtual entrypoint for the entire cluster.

Let me explain my reasoning for wander into duality of this setup. Standard Sentinel setup from doc is:

>>> from redis.sentinel import Sentinel
>>> sentinel = Sentinel([('localhost', 26379)], socket_timeout=0.1)
>>> sentinel.discover_master('mymaster')
('127.0.0.1', 6379)
>>> sentinel.discover_slaves('mymaster')
[('127.0.0.1', 6380)]

and

>>> master = sentinel.master_for('mymaster', socket_timeout=0.1)
>>> slave = sentinel.slave_for('mymaster', socket_timeout=0.1)
>>> master.set('foo', 'bar')
>>> slave.get('foo')
'bar'

So this is already split. Now with this new flag from 5.0.4 I have to configure slaves to allow write even though I don't want to write to slaves as Sentinel is already enforcing the split by providing two isolated instances for access to masters and slaves.

However a big thanks for the hint because at least know it is back in the realm of possibility.

[vvippark]

@swilly22 , @styk-tv
Thanks both of you guys for giving a way to graph.query on redis replicas.

Just confirming again, you are able to do with CONFIG SET slave-read-only no?

[swilly22]

correct

[swilly22]

Use GRAPH.RO_QUERY #1364