Skip to content

HDDS-1972. Provide example ha proxy with multiple s3 servers back end. #1304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Aug 19, 2019
Merged

HDDS-1972. Provide example ha proxy with multiple s3 servers back end. #1304

merged 6 commits into from
Aug 19, 2019

Conversation

@bharatviswa504
Copy link
Contributor

Tested it on my laptop:

$ docker-compose up
Creating network "ozones3-haproxy_default" with the default driver
Creating ozones3-haproxy_s3-proxy_1 ... done
Creating ozones3-haproxy_s3g3_1     ... done
Creating ozones3-haproxy_scm_1      ... done
Creating ozones3-haproxy_s3g1_1     ... done
Creating ozones3-haproxy_om_1       ... done
Creating ozones3-haproxy_s3g2_1     ... done
Creating ozones3-haproxy_datanode_1 ... done
Attaching to ozones3-haproxy_s3-proxy_1, ozones3-haproxy_om_1, ozones3-haproxy_datanode_1, ozones3-haproxy_s3g3_1, ozones3-haproxy_s3g1_1, ozones3-haproxy_scm_1, ozones3-haproxy_s3g2_1
s3-proxy_1  | [NOTICE] 226/212303 (1) : New worker #1 (6) forked

$ aws s3api --endpoint http://localhost:8081 create-bucket --bucket b12346
{
    "Location": "http://localhost:8081/b12346"
}
HW13865:ozones3-haproxy bviswanadham$ aws s3api --endpoint http://localhost:8081 create-bucket --bucket b1234
{
    "Location": "http://localhost:8081/b1234"
}
HW13865:ozones3-haproxy bviswanadham$ aws s3api --endpoint http://localhost:8081 create-bucket --bucket b123
{
    "Location": "http://localhost:8081/b123"
}
HW13865:ozones3-haproxy bviswanadham$ aws s3api --endpoint http://localhost:8081 list-buckets
{
    "Buckets": [
        {
            "CreationDate": "2019-08-15T21:23:49.643Z", 
            "Name": "b123"
        }, 
        {
            "CreationDate": "2019-08-15T21:23:45.330Z", 
            "Name": "b1234"
        }, 
        {
            "CreationDate": "2019-08-15T21:23:42.629Z", 
            "Name": "b12346"
        }
    ]
}

docker logs:

s3g1_1      | 2019-08-15 21:23:42 INFO  BucketEndpoint:206 - Location is /b12346
s3g2_1      | 2019-08-15 21:23:45 INFO  BucketEndpoint:206 - Location is /b1234
s3g3_1      | 2019-08-15 21:23:49 INFO  BucketEndpoint:206 - Location is /b123

@bharatviswa504 bharatviswa504 self-assigned this Aug 15, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 43 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 yamllint 1 yamllint was not available.
0 shelldocs 1 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 630 trunk passed
+1 compile 370 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 791 branch has no errors when building and testing our client artifacts.
+1 javadoc 169 trunk passed
_ Patch Compile Tests _
+1 mvninstall 562 the patch passed
+1 compile 380 the patch passed
+1 javac 380 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 689 patch has no errors when building and testing our client artifacts.
+1 javadoc 165 the patch passed
_ Other Tests _
+1 unit 286 hadoop-hdds in the patch passed.
-1 unit 1832 hadoop-ozone in the patch failed.
+1 asflicense 45 The patch does not generate ASF License warnings.
6185
Reason Tests
Failed junit tests hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/1/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux b004b49d3141 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 77d102c
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/1/testReport/
Max. process+thread count 4308 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/1/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

adoroszlai
adoroszlai requested changes Aug 16, 2019
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @bharatviswa504. The setup looks good, but there are a few small changes needed for the acceptance test to pass. S3 tests use http://s3g:9878 as endpoint. To allow it to run unmodified, the proxy service should be named s3g and it should listen on port 9878.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 45 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
0 yamllint 0 yamllint was not available.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 618 trunk passed
+1 compile 364 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 739 branch has no errors when building and testing our client artifacts.
+1 javadoc 153 trunk passed
_ Patch Compile Tests _
+1 mvninstall 540 the patch passed
+1 compile 379 the patch passed
+1 javac 379 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 632 patch has no errors when building and testing our client artifacts.
+1 javadoc 161 the patch passed
_ Other Tests _
+1 unit 297 hadoop-hdds in the patch passed.
-1 unit 179 hadoop-ozone in the patch failed.
+1 asflicense 40 The patch does not generate ASF License warnings.
4332
Reason Tests
Failed junit tests hadoop.ozone.security.TestOzoneDelegationTokenSecretManager
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/2/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux 4de79affb250 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 9b8359b
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/2/testReport/
Max. process+thread count 1327 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/2/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bharatviswa504 and others added 2 commits August 16, 2019 08:32
@bharatviswa504 @adoroszlai
Apply suggestions from code review
b6855d8 
Co-Authored-By: Doroszlai, Attila <[email protected]>
@bharatviswa504
Change S3 gateway ports.
2776de9 
As s3g port is used for s3 proxy.
@bharatviswa504
Copy link
Contributor Author

Thank You @adoroszlai for the review.
Thanks for the suggestion, now I think tests should pass.
In addition, I have done additional change, like one of S3G is exposed with port 9878, now changed that to 9879 and similarly to other s3g demons.

@bharatviswa504
Copy link
Contributor Author

On my desktop I see tests are passing.
Below is the result.

HW13865:ozones3-haproxy bviswanadham$ ./test.sh 
Removing network ozones3-haproxy_default
WARNING: Network ozones3-haproxy_default not found.
Creating network "ozones3-haproxy_default" with the default driver
Creating ozones3-haproxy_datanode_1 ... done
Creating ozones3-haproxy_datanode_2 ... done
Creating ozones3-haproxy_datanode_3 ... done
Creating ozones3-haproxy_scm_1      ... done
Creating ozones3-haproxy_om_1       ... done
Creating ozones3-haproxy_s3g_1      ... done
Creating ozones3-haproxy_s3g1_1     ... done
Creating ozones3-haproxy_s3g2_1     ... done
Creating ozones3-haproxy_s3g3_1     ... done
0 datanode is up and healthy (until now)
0 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
==============================================================================
ozones3-haproxy-basic :: Smoketest ozone cluster startup                      
==============================================================================
Check webui static resources                                          | PASS |
------------------------------------------------------------------------------
Start freon testing                                                   | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-basic :: Smoketest ozone cluster startup              | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozones3-haproxy/result/robot-ozones3-haproxy-ozones3-haproxy-basic-scm.xml
==============================================================================
ozones3-haproxy-s3                                                            
==============================================================================
ozones3-haproxy-s3.Awss3 :: S3 gateway test with aws cli                      
==============================================================================
File upload and directory list                                        | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Awss3 :: S3 gateway test with aws cli              | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Bucketcreate :: S3 gateway test with aws cli               
==============================================================================
Create bucket which already exists                                    | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Bucketcreate :: S3 gateway test with aws cli       | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Buckethead :: S3 gateway test with aws cli                 
==============================================================================
Head Bucket not existent                                              | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Buckethead :: S3 gateway test with aws cli         | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Bucketlist :: S3 gateway test with aws cli                 
==============================================================================
List buckets                                                          | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Bucketlist :: S3 gateway test with aws cli         | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.MultipartUpload :: S3 gateway test with aws cli            
==============================================================================
Test Multipart Upload                                                 | PASS |
------------------------------------------------------------------------------
Test Multipart Upload Complete                                        | PASS |
------------------------------------------------------------------------------
Test Multipart Upload Complete Entity too small                       | PASS |
------------------------------------------------------------------------------
Test Multipart Upload Complete Invalid part                           | PASS |
------------------------------------------------------------------------------
Test abort Multipart upload                                           | PASS |
------------------------------------------------------------------------------
Test abort Multipart upload with invalid uploadId                     | PASS |
------------------------------------------------------------------------------
Upload part with Incorrect uploadID                                   | PASS |
------------------------------------------------------------------------------
Test list parts                                                       | PASS |
------------------------------------------------------------------------------
Test Multipart Upload with the simplified aws s3 cp API               | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.MultipartUpload :: S3 gateway test with aws cli    | PASS |
9 critical tests, 9 passed, 0 failed
9 tests total, 9 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectcopy :: S3 gateway test with aws cli                 
==============================================================================
Copy Object Happy Scenario                                            | PASS |
------------------------------------------------------------------------------
Copy Object Where Bucket is not available                             | PASS |
------------------------------------------------------------------------------
Copy Object Where both source and dest are same with change to sto... | PASS |
------------------------------------------------------------------------------
Copy Object Where Key not available                                   | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectcopy :: S3 gateway test with aws cli         | PASS |
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectdelete :: S3 gateway test with aws cli               
==============================================================================
Delete file with s3api                                                | PASS |
------------------------------------------------------------------------------
Delete file with s3api, file doesn't exist                            | PASS |
------------------------------------------------------------------------------
Delete dir with s3api                                                 | PASS |
------------------------------------------------------------------------------
Delete file with s3api, file doesn't exist, prefix of a real file     | PASS |
------------------------------------------------------------------------------
Delete file with s3api, bucket doesn't exist                          | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectdelete :: S3 gateway test with aws cli       | PASS |
5 critical tests, 5 passed, 0 failed
5 tests total, 5 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectmultidelete :: S3 gateway test with aws cli          
==============================================================================
Delete file with multi delete                                         | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectmultidelete :: S3 gateway test with aws cli  | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectputget :: S3 gateway test with aws cli               
==============================================================================
Put object to s3                                                      | PASS |
------------------------------------------------------------------------------
Get object from s3                                                    | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset              | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(start off... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(end offse... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with only start offset                     | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset which are... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 to get last n bytes                        | PASS |
------------------------------------------------------------------------------
Incorrect values for end and start offset                             | PASS |
------------------------------------------------------------------------------
Zero byte file                                                        | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectputget :: S3 gateway test with aws cli       | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Webui :: S3 gateway web ui test                            
==============================================================================
S3 Gateway Web UI                                                     | PASS |
------------------------------------------------------------------------------
ozones3-haproxy-s3.Webui :: S3 gateway web ui test                    | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3                                                    | PASS |
34 critical tests, 34 passed, 0 failed
34 tests total, 34 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozones3-haproxy/result/robot-ozones3-haproxy-ozones3-haproxy-s3-scm.xml
Stopping ozones3-haproxy_datanode_2 ... done
Stopping ozones3-haproxy_s3g3_1     ... done
Stopping ozones3-haproxy_om_1       ... done
Stopping ozones3-haproxy_s3g1_1     ... done
Stopping ozones3-haproxy_s3g2_1     ... done
Stopping ozones3-haproxy_s3g_1      ... done
Stopping ozones3-haproxy_datanode_1 ... done
Stopping ozones3-haproxy_datanode_3 ... done
Stopping ozones3-haproxy_scm_1      ... done
Removing ozones3-haproxy_datanode_2 ... done
Removing ozones3-haproxy_s3g3_1     ... done
Removing ozones3-haproxy_om_1       ... done
Removing ozones3-haproxy_s3g1_1     ... done
Removing ozones3-haproxy_s3g2_1     ... done
Removing ozones3-haproxy_s3g_1      ... done
Removing ozones3-haproxy_datanode_1 ... done
Removing ozones3-haproxy_datanode_3 ... done
Removing ozones3-haproxy_scm_1      ... done
Removing network ozones3-haproxy_default
Log:     /Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3-haproxy/result/log.html
Report:  /Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3-haproxy/result/report.html

@bharatviswa504
Copy link
Contributor Author

Posted empty commit, to trigger Jenkins run.

@bharatviswa504
Copy link
Contributor Author

/retest

@bharatviswa504
Copy link
Contributor Author

Not sure why test is failing in Jenkins run, but passing locally.
Will look into it, if not able to figure out the issue, planning to disable S3 test suite run with proxy.
@adoroszlai any thoughts?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 76 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
0 yamllint 0 yamllint was not available.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 656 trunk passed
+1 compile 399 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 876 branch has no errors when building and testing our client artifacts.
+1 javadoc 178 trunk passed
_ Patch Compile Tests _
+1 mvninstall 578 the patch passed
+1 compile 391 the patch passed
+1 javac 391 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 743 patch has no errors when building and testing our client artifacts.
+1 javadoc 186 the patch passed
_ Other Tests _
+1 unit 358 hadoop-hdds in the patch passed.
-1 unit 2236 hadoop-ozone in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
6933
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/3/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux 55bc14b20cf9 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 8943e13
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/3/testReport/
Max. process+thread count 4573 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/3/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 32 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 yamllint 1 yamllint was not available.
0 shelldocs 1 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 637 trunk passed
+1 compile 400 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 860 branch has no errors when building and testing our client artifacts.
+1 javadoc 228 trunk passed
_ Patch Compile Tests _
+1 mvninstall 647 the patch passed
+1 compile 429 the patch passed
+1 javac 429 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 810 patch has no errors when building and testing our client artifacts.
+1 javadoc 205 the patch passed
_ Other Tests _
+1 unit 372 hadoop-hdds in the patch passed.
-1 unit 2048 hadoop-ozone in the patch failed.
+1 asflicense 61 The patch does not generate ASF License warnings.
6954
Reason Tests
Failed junit tests hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/4/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux 6a557ccbbeb0 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 8943e13
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/4/testReport/
Max. process+thread count 4499 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/4/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 83 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 yamllint 0 yamllint was not available.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 631 trunk passed
+1 compile 428 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 911 branch has no errors when building and testing our client artifacts.
+1 javadoc 187 trunk passed
_ Patch Compile Tests _
+1 mvninstall 638 the patch passed
+1 compile 415 the patch passed
+1 javac 415 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 742 patch has no errors when building and testing our client artifacts.
+1 javadoc 178 the patch passed
_ Other Tests _
+1 unit 373 hadoop-hdds in the patch passed.
-1 unit 1947 hadoop-ozone in the patch failed.
+1 asflicense 56 The patch does not generate ASF License warnings.
6796
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/5/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux ee7c84075a91 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c8675ec
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/5/testReport/
Max. process+thread count 4459 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/5/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bharatviswa504
Copy link
Contributor Author

/retest

@bharatviswa504
Copy link
Contributor Author

/retest

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 43 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 yamllint 0 yamllint was not available.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 621 trunk passed
+1 compile 365 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 731 branch has no errors when building and testing our client artifacts.
+1 javadoc 155 trunk passed
_ Patch Compile Tests _
+1 mvninstall 546 the patch passed
+1 compile 378 the patch passed
+1 javac 378 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 659 patch has no errors when building and testing our client artifacts.
+1 javadoc 163 the patch passed
_ Other Tests _
+1 unit 300 hadoop-hdds in the patch passed.
-1 unit 2080 hadoop-ozone in the patch failed.
+1 asflicense 55 The patch does not generate ASF License warnings.
6304
Reason Tests
Failed junit tests hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestCommitWatcher
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux a38bbfa4497e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / b8db5b9
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/testReport/
Max. process+thread count 5411 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bharatviswa504
Copy link
Contributor Author

bharatviswa504 commented Aug 19, 2019 •
edited
Loading

I see tests are passing locally on my laptop. I was able to test ha-proxy with aws cli. Not able to figure out why on jenkins it is showing not able to connect to http://s3g:9878.

For now, I have disabled the test. But this will give an example for users to use ha-proxy setup with S3Gateway Server. (I will open a new Jira(https://issues.apache.org/jira/browse/HDDS-1983) to enable s3 test suite for s3 proxy)

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 46 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 yamllint 0 yamllint was not available.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 601 trunk passed
+1 compile 375 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 723 branch has no errors when building and testing our client artifacts.
+1 javadoc 158 trunk passed
_ Patch Compile Tests _
+1 mvninstall 566 the patch passed
+1 compile 395 the patch passed
+1 javac 395 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 652 patch has no errors when building and testing our client artifacts.
+1 javadoc 170 the patch passed
_ Other Tests _
+1 unit 316 hadoop-hdds in the patch passed.
-1 unit 2696 hadoop-ozone in the patch failed.
+1 asflicense 49 The patch does not generate ASF License warnings.
6969
Reason Tests
Failed junit tests hadoop.ozone.dn.scrubber.TestDataScrubber
hadoop.ozone.container.ozoneimpl.TestOzoneContainer
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.ozone.client.rpc.TestReadRetries
hadoop.ozone.om.TestOzoneManagerHA
hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/artifact/out/Dockerfile
GITHUB PR #1304
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname Linux a05c19df5e06 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c765584
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/testReport/
Max. process+thread count 3723 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/7/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7
arp7 approved these changes Aug 19, 2019
View reviewed changes
Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@adoroszlai are you okay to commit this?

hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
command: ["ozone","scm"]
s3g1:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These can be collapsed into one entry, then we can use scale option. Not a blocker to commit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can do that, but we need to have ha-proxy image which can have some configuration, where we can dynamically configure s3proxy and start/stop.

@bharatviswa504
Copy link
Contributor Author

Thank You @arp7 and @adoroszlai for the review.
I will commit this to the trunk.

@bharatviswa504 bharatviswa504 merged commit 4f925af into apache:trunk Aug 19, 2019
@bharatviswa504
Copy link
Contributor Author

+1

@adoroszlai are you okay to commit this?

Got a +1 from @adoroszlai already, so committed this change.

asfgit pushed a commit that referenced this pull request Aug 21, 2019
@bharatviswa504
HDDS-1972. Provide example ha proxy with multiple s3 servers back end. (
551fd71 
#1304)

(cherry picked from commit 4f925af)
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
@bharatviswa504
HDDS-1972. Provide example ha proxy with multiple s3 servers back end. (
31f2c4a 
apache#1304)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adoroszlai adoroszlai adoroszlai approved these changes

@arp7 arp7 arp7 approved these changes

@anuengineer anuengineer Awaiting requested review from anuengineer

Assignees

@bharatviswa504 bharatviswa504

Labels

ozone

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bharatviswa504 @hadoop-yetus @arp7 @adoroszlai