Skip to content

HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads #5274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 15, 2023
Merged

HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads #5274

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e0c5ee8
Logged errors as info without stack trace for prefetching reads
Nov 21, 2022
b6d2d9c
changed error logging style for debug
Dec 1, 2022
6d01128
Logged failures during Read at error level and during prefetch at inf…
sauraank Dec 19, 2022
3364461
Addressed issues of throwing errors when read fails
sauraank Dec 31, 2022
6c81a1f
Improved spacing and indentation
sauraank Jan 4, 2023
2dd89ea
Improved error message and checkstyle issue
sauraank Jan 11, 2023
a4a2d2e
Added space and and other minor changes
Feb 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions ...t/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,12 @@ public void cancelPrefetches() {

private void read(BufferData data) throws IOException {
synchronized (data) {
readBlock(data, false, BufferData.State.BLANK);
try {
readBlock(data, false, BufferData.State.BLANK);
} catch (IOException e) {
LOG.error("error reading block {}", data.getBlockNumber(), e);
throw e;
}
}
}

Expand Down Expand Up @@ -362,9 +367,6 @@ private void readBlock(BufferData data, boolean isPrefetch, BufferData.State...
buffer.flip();
data.setReady(expectedState);
} catch (Exception e) {
String message = String.format("error during readBlock(%s)", data.getBlockNumber());
LOG.error(message, e);

if (isPrefetch && tracker != null) {
tracker.failed();
}
Expand Down Expand Up @@ -406,7 +408,8 @@ public Void get() {
try {
blockManager.prefetch(data, taskQueuedStartTime);
} catch (Exception e) {
LOG.error("error during prefetch", e);
LOG.info("error prefetching block {}. {}", data.getBlockNumber(), e.getMessage());
LOG.debug("error prefetching block {}", data.getBlockNumber(), e);
}
return null;
}
Expand Down Expand Up @@ -493,7 +496,8 @@ private void addToCacheAndRelease(BufferData data, Future<Void> blockFuture,
return;
}
} catch (Exception e) {
LOG.error("error waiting on blockFuture: {}", data, e);
LOG.info("error waiting on blockFuture: {}. {}", data, e.getMessage());
LOG.debug("error waiting on blockFuture: {}", data, e);
data.setDone();
return;
}
Expand Down Expand Up @@ -523,8 +527,8 @@ private void addToCacheAndRelease(BufferData data, Future<Void> blockFuture,
data.setDone();
} catch (Exception e) {
numCachingErrors.incrementAndGet();
String message = String.format("error adding block to cache after wait: %s", data);
LOG.error(message, e);
LOG.info("error adding block to cache after wait: {}. {}", data, e.getMessage());
LOG.debug("error adding block to cache after wait: {}", data, e);
data.setDone();
}

Expand Down