- 
                Notifications
    
You must be signed in to change notification settings  - Fork 9.1k
 
HDFS-16767. RBF: Support observer node from Router-Based Federation #4127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| 
           💔 -1 overall 
 
 This message was automatically generated.  | 
    
5f34b2d    to
    2088b4a      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
2088b4a    to
    88c780c      
    Compare
  
            
          
                ...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
      | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
88c780c    to
    cbf3a15      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
        
          
                ...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...ect/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
      | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
562b783    to
    fdea891      
    Compare
  
    | 
           Hi @fengnanli could you please take a look and add folks in your team.  | 
    
| 
           🎊 +1 overall 
 
 This message was automatically generated.  | 
    
fdea891    to
    8324a37      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
8324a37    to
    98eee9f      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
7f71985    to
    77e41ba      
    Compare
  
    | 
           💔 -1 overall 
 
 This message was automatically generated.  | 
    
| 
           💔 -1 overall 
 
 This message was automatically generated.  | 
    
| 
           @goiri @omalley I've now split off the IPC related parts of this change into another pull request (#4311). Please take a look.  | 
    
77e41ba    to
    de6bb46      
    Compare
  
    | 
           💔 -1 overall 
 
 This message was automatically generated.  | 
    
de6bb46    to
    586216c      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
586216c    to
    986c613      
    Compare
  
    | 
           🎊 +1 overall 
 
 This message was automatically generated.  | 
    
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
3e5bda5    to
    2edd2b5      
    Compare
  
    There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm really worried about the performance characteristics with doing the all namespace msync. I think we'd be better off if the client doesn't have the federatedstateids set on the rpc header causing them to go to the active NN.
We can still send back the federatedstateids and so later calls will go to the observer NNs. That does mean that old clients will only use the active NN, but I think that is ok. It also means that it is easy to have clients always use the active NN by not including an alignmentcontext.
        
          
                ...rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/FederationRPCMetrics.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...op-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
      2edd2b5    to
    174a043      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
174a043    to
    54d2980      
    Compare
  
    54d2980    to
    61b0758      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
| 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
        
          
                .../main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...op-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                .../main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
              
          
                ...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
              
                Outdated
          
            Show resolved
            Hide resolved
        
      723e906    to
    12c90d4      
    Compare
  
    12c90d4    to
    b0e67a3      
    Compare
  
    | 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
| 
           💔 -1 overall 
 
 
 This message was automatically generated.  | 
    
| 
           🎊 +1 overall 
 
 This message was automatically generated.  | 
    
Fixes apache#4127 Signed-off-by: Owen O'Malley <[email protected]>
| 
           Why it is closed?  | 
    
| 
           Actually, it was committed to trunk. 6422eaf  | 
    
Fixes apache#4127 Signed-off-by: Owen O'Malley <[email protected]>
Fixes apache#4127 Signed-off-by: Owen O'Malley <[email protected]> ACLOVERRIDE
Fixes apache#4127 Signed-off-by: Owen O'Malley <[email protected]>
Description of PR
Enables routers to direct read calls to observer namenodes.
This is a refresh of a patch by Zhuobin Zhang to make it apply on current trunk. https://issues.apache.org/jira/secure/attachment/13027012/HDFS-13522.002.patch
How was this patch tested?
New unit tests
Deployed this in our test cluster and saw read operations in the observer audit log.
For code changes: