Skip to content

Conversation

@elek
Copy link
Member

@elek elek commented Jun 3, 2019

Ozone tar file creation is a very time consuming step. I propose to make it optional and create the tar file only if the dist profile is enabled (-Pdist)

The tar file is not required to test ozone as the same content is available from hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT which is enough to run docker-compose pseudo clusters, smoketests.

If it's required, the tar file creation can be requested by the dist profile.

On my machine (ssd based) it can cause 5-10% time improvements as the tar size is ~500MB and it requires a lot of IO.

See: https://issues.apache.org/jira/browse/HDDS-1629

@elek elek added the ozone label Jun 3, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 46 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 593 trunk passed
+1 compile 312 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1766 branch has no errors when building and testing our client artifacts.
+1 javadoc 174 trunk passed
_ Patch Compile Tests _
+1 mvninstall 506 the patch passed
+1 compile 293 the patch passed
+1 javac 293 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 1 The patch has no ill-formed XML file.
+1 shadedclient 710 patch has no errors when building and testing our client artifacts.
+1 javadoc 173 the patch passed
_ Other Tests _
-1 unit 267 hadoop-hdds in the patch failed.
-1 unit 2088 hadoop-ozone in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
6277
Reason Tests
Failed junit tests hadoop.hdds.scm.block.TestBlockManager
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestBCSID
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/artifact/out/Dockerfile
GITHUB PR #887
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname Linux 2c094619b49a 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 4530f45
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/testReport/
Max. process+thread count 5338 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@anuengineer anuengineer merged commit e140a45 into apache:trunk Jun 4, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
Samza uses millisec as config value, while in rocksDB it's defined as int32. It's currently defined as integer in RocksDbTableDescriptor, the range isn't large enough to match, and it should be of Long type.

Author: Wei Song <[email protected]>

Reviewers: Hai Lu <[email protected]>

Closes apache#887 from weisong44/SAMZA-2076
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants