StreamEntry (XREAD) does not support binary because it uses Map<String, String>

[raylau1234]

StreamEntry uses a Map<String, String> for fields.

As a result, binary data which contains illegal utf-8 sequences get mangled.

Reproduce by using XADD with a key or value which contains an illegal utf-8 sequence like 0xc3 0x28.

Retrieve via XREAD.

Expected result -- API to get byte array instead of a String, which is lossy with illegal utf-8 sequences.

[sazzad16]

First of all, what is XGET?

Secondly, there are xrange(byte[] key, ...) methods. Don't these help?

[raylau1234]

Sorry, I meant XREAD.

You are correct on the alternate XRANGE signature. But I am really using XREAD. Let me update the title and to clarify.... XREAD only is problematic.

[raylau1234]

I see there are alternate signatures for xread as well. Let me see of those help. I found the StreamEntry ones first. Give me a few minutes to explore and will udpate.

[raylau1234]

OK, List<byte[]> xread(XReadParams, Entry<byte[], byte[]>... streams) is a possible workaround, but is cumbersome bec I need to deserialize the byte into the map of key / message id / values. i.e. I need to replicate everything that List<Map, Entry<String, List<StreamEntry>>> xread(XReadParams, Map<String, StreamEntryID>) is doing in terms of decoding with STREAM_READ_RESPONSE, STREAM_ENTRY_LIST, etc. (but skipping the SafeEncoder.encode() and keeping the binary there).

[kmularise]

Hello, I noticed that there was a draft PR (#3803) opened for this issue quite a while ago, but it hasn’t been updated in some time.

I’m interested in picking up this work and continuing from where it left off — possibly by improving the previous approach or starting fresh while keeping backward compatibility in mind.

Would it be okay if I proceed with a new PR for this? Please let me know if there are any concerns or preferences before I begin

[thachlp]

@kmularise sure, please open the new pr for this issue, I will close mine 🙇

[kmularise]

@thachlp Thanks for the clarification and for allowing others to pick this up.
I’ll start working on a new PR for this issue. Appreciate the prior work done on #3803!