-
Notifications
You must be signed in to change notification settings - Fork 9.1k
HDFS-17808. EC: End block group in advance to prevent write failure for long-time running OutputStream #7810
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
Conversation
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
ee30da4
to
462f2ad
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
52b4698
to
e72198c
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
…or long-time running OutputStream.
e72198c
to
cac9ae7
Compare
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
Hi, @Hexiaoqiao @zhangshuyan0 @haiyang1987 could you please help review this pr when you are free? Thanks very much. |
🎊 +1 overall
This message was automatically generated. |
Thanks @hfutatzhanghb involve me here. Have quick review, I am not sure if close then create new stream will solve this issue, such as some DataNodes cost long time to restart or even can't restart again, then this solution will not be available, right? Compare to replica/pipeline case, there is one step to copy data then build pipeline again, but for EC, the first step could not be feasible. For one word, we need to define issue clearly. Thanks again. |
@Hexiaoqiao Thanks very much for reviewing. Please allow me to define issue clearly here. Recently, while exploring the use of HDFS Erasure Coding (EC) for hot-data storage, we encountered some problems and the current issue is one of them. Problem description (pseudo-code): DFSStripedOutputStream os = dfs.create(path);
// The task may run for several hours, so the output-stream object is also held open for hours.
while (task is not finished) {
data = doSomeComputeLogicAndGetData();
os.write(data);
}
os.close(); When we perform a rolling restart of DataNodes, the above task fails. So, this pr try to resolve such problem by ending block group in advance when meet failed streamers(count of streamer <= 3)to allocate new block, after allocating new block, we wil have sufficient data streamer. Hope to receiving your response~ Thanks again!!! |
Description of PR
Refer to HDFS-17808.
Recently, we met an EC problem in our production.
User creates an output stream to write ec files. That output stream writes some bytes and will be idle for a long time until data is ready. If we restart our cluster's datanodes to version up, those applications will finally fail due to not have enough healthy streamers.
This PR try to solve above problem by end block group in advance when we already have failed streamers but less than parity number.