HADOOP-18507. VectorIO FileRange type to support a "reference" field #5076

steveloughran · 2022-10-25T17:05:59Z

Add new reference field; verifies it remains in merging.

also fix up those tests which used string comparison of range values into actual field checking.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Add new reference field; verifies it remains in merging. also fix up those tests which used string comparison of range values into actual field checking. Change-Id: I46fe5fb9d21f58a15bce0b42bd79cefd1bd0a0fe

mukund-thakur

looks good. Yetus failed but it doesn't show any error.

Also did you re-ran the vectored io integration tests? Which endpoint?

mukund-thakur · 2022-10-25T20:08:20Z

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileRange.java

+   * Get any reference passed in to the file range constructor.
+   * This is not used by any implementation code; it is to help
+   * bind this API to libraries retrieving multiple stripes of
+   * dat in parallel.j


mukund-thakur · 2022-10-25T20:25:47Z

...p-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestVectoredReadUtils.java

-        FileRange.createFileRange(1000, 100)
+        FileRange.createFileRange(3000, 100, "1"),
+        FileRange.createFileRange(2100, 100, "2"),
+        FileRange.createFileRange(1000, 100, "3")


Can we set one reference to null, such that null references gets tested as well.

good idea. i will do that for range #2

Change-Id: I50cf741e76189a53a587de31cd69928536a986dd

hadoop-yetus · 2022-10-26T15:54:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 40s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 40s		trunk passed
+1 💚	compile	25m 36s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	22m 9s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 27s		trunk passed
+1 💚	mvnsite	2m 54s		trunk passed
+1 💚	javadoc	2m 10s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 42s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 59s		trunk passed
+1 💚	shadedclient	23m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 44s		the patch passed
+1 💚	compile	24m 51s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	24m 51s		the patch passed
+1 💚	compile	22m 7s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 12s	/results-checkstyle-root.txt	root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
+1 💚	mvnsite	2m 40s		the patch passed
+1 💚	javadoc	2m 3s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 6s		the patch passed
+1 💚	shadedclient	24m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 26s		hadoop-common in the patch passed.
+1 💚	unit	0m 47s		hadoop-benchmark in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		245m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5076/2/artifact/out/Dockerfile
GITHUB PR	#5076
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ed4254f45df5 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bc88b60`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5076/2/testReport/
Max. process+thread count	1284 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-benchmark U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5076/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur

+1 pending checkstyle issues.

Change-Id: I79b8be3ae8498fe02f85e47c270990293663f940

hadoop-yetus · 2022-10-27T14:03:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 56s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 14s		trunk passed
+1 💚	compile	23m 12s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	20m 42s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 9s		trunk passed
+1 💚	mvnsite	2m 49s		trunk passed
+1 💚	javadoc	2m 21s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 8s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 57s		trunk passed
+1 💚	shadedclient	21m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 48s		the patch passed
+1 💚	compile	22m 30s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	22m 30s		the patch passed
+1 💚	compile	20m 56s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 57s		the patch passed
+1 💚	mvnsite	2m 48s		the patch passed
+1 💚	javadoc	2m 27s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 58s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 15s		the patch passed
+1 💚	shadedclient	21m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 39s		hadoop-common in the patch passed.
+1 💚	unit	0m 57s		hadoop-benchmark in the patch passed.
+1 💚	asflicense	1m 15s		The patch does not generate ASF License warnings.
		233m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5076/3/artifact/out/Dockerfile
GITHUB PR	#5076
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8e57679f801c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ba38b2a`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5076/3/testReport/
Max. process+thread count	3152 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-benchmark U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5076/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-10-31T21:20:42Z

in trunk; need to backport

…5076) Contributed by Steve Loughran

…pache#5076) Contributed by Steve Loughran

HADOOP-18507. VectorIO FileRange type to support a "reference" field

dc35f7e

Add new reference field; verifies it remains in merging. also fix up those tests which used string comparison of range values into actual field checking. Change-Id: I46fe5fb9d21f58a15bce0b42bd79cefd1bd0a0fe

steveloughran requested a review from mukund-thakur October 25, 2022 17:06

mukund-thakur reviewed Oct 25, 2022

View reviewed changes

HADOOP-18507. review changes and test failure

bc88b60

Change-Id: I50cf741e76189a53a587de31cd69928536a986dd

mukund-thakur approved these changes Oct 26, 2022

View reviewed changes

HADOOP-18507. checkstyle

ba38b2a

Change-Id: I79b8be3ae8498fe02f85e47c270990293663f940

steveloughran merged commit 3b10cb5 into apache:trunk Oct 31, 2022

asfgit pushed a commit that referenced this pull request Nov 1, 2022

HADOOP-18507. VectorIO FileRange type to support a "reference" field (#…

9bf9560

…5076) Contributed by Steve Loughran

asfgit pushed a commit that referenced this pull request Nov 8, 2022

HADOOP-18507. VectorIO FileRange type to support a "reference" field (#…

c392075

…5076) Contributed by Steve Loughran

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18507. VectorIO FileRange type to support a "reference" field (a…

2abd964

…pache#5076) Contributed by Steve Loughran

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18507. VectorIO FileRange type to support a "reference" field #5076

HADOOP-18507. VectorIO FileRange type to support a "reference" field #5076

Uh oh!

steveloughran commented Oct 25, 2022

Uh oh!

mukund-thakur left a comment

Uh oh!

mukund-thakur Oct 25, 2022

Uh oh!

mukund-thakur Oct 25, 2022

Uh oh!

steveloughran Oct 26, 2022

Uh oh!

hadoop-yetus commented Oct 26, 2022

Uh oh!

mukund-thakur left a comment

Uh oh!

hadoop-yetus commented Oct 27, 2022

Uh oh!

steveloughran commented Oct 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-18507. VectorIO FileRange type to support a "reference" field #5076

HADOOP-18507. VectorIO FileRange type to support a "reference" field #5076

Uh oh!

Conversation

steveloughran commented Oct 25, 2022

For code changes:

Uh oh!

mukund-thakur left a comment

Choose a reason for hiding this comment

Uh oh!

mukund-thakur Oct 25, 2022

Choose a reason for hiding this comment

Uh oh!

mukund-thakur Oct 25, 2022

Choose a reason for hiding this comment

Uh oh!

steveloughran Oct 26, 2022

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Oct 26, 2022

Uh oh!

mukund-thakur left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Oct 27, 2022

Uh oh!

steveloughran commented Oct 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants