Skip to content

HADOOP-18246. Reduce lower limit on s3a prefetching/caching block size to 1 byte. #5120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 2, 2023
Merged

HADOOP-18246. Reduce lower limit on s3a prefetching/caching block size to 1 byte. #5120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0429180
Made the lower limit to 1
Nov 4, 2022
cdbed32
Lowered the Prefetch Limit, unchanged the Default size
Nov 8, 2022
563f0ff
Added the lower limit information in the Prefetch Readme
Nov 8, 2022
ddba5cd
Added new line after period in readme.
Nov 8, 2022
bf73b01
Updated the documentation of general s3a client configuration.
Nov 9, 2022
5705e99
Improved block description in s3a client configuration
Nov 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -525,8 +525,7 @@ public void initialize(URI name, Configuration originalConf)

this.prefetchEnabled = conf.getBoolean(PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT);
long prefetchBlockSizeLong =
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE,
PREFETCH_BLOCK_DEFAULT_SIZE);
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, 1);
if (prefetchBlockSizeLong > (long) Integer.MAX_VALUE) {
throw new IOException("S3A prefatch block size exceeds int limit");
}
Expand Down
1 change: 1 addition & 0 deletions hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,7 @@ options are covered in [Testing](./testing.md).
<value>8MB</value>
<description>
The size of a single prefetched block of data.
Decreasing this will increase the number of prefetches required, and may negatively impact performance.
</description>
</property>

Expand Down
4 changes: 4 additions & 0 deletions hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ Multiple blocks may be read in parallel.
|`fs.s3a.prefetch.block.size` |Size of a block |`8M` |
|`fs.s3a.prefetch.block.count` |Number of blocks to prefetch |`8` |

The default size of a block is 8MB, and the minimum allowed block size is 1 byte.
Decreasing block size will increase the number of blocks to be read for a file.
A smaller block size may negatively impact performance as the number of prefetches required will increase.

### Key Components

`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of
Expand Down