Provided with disabled fs cached #1
Conversation
…here Issue: Warning messages in DN logs, e.g. Exception in retrieving ReplicaInfo for block id XXXXX Cause: The datanode searches for the requested block in all storage volumes including mounted directories. The mounted volume search writes a warning log message when the block is not found even if the block was supposed to be missing, new block for e.g. Fix: 1. Replace a warning message with a debug log for missing blocks in mounted dirs 2. Prioritize searching the block in local dirs before looking in mounted dirs. This may avoid unnecessary lookups.
Tiered block reads fail on DNs when FS caching is disabled and the remote store requires credentials. This happens because the DN searches the wrong remoteFS. For e.g. there is a default provided volume created on DN. It uses the same alias map as other provided volumes. However it uses the default conf object. While reading a block, the volume filds the FileRegion for the required block from alias map. However, creation of a secured file system object will fail as the config object will not contain the desired credentials. So the DN will fail the read.
...oop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
Show resolved
Hide resolved
...ain/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ProvidedVolumeReplicaMap.java
Show resolved
Hide resolved
| fsVolume.addBlockPool(bpid, config, this.timer); | ||
| tempVolumeMap.addAll( | ||
| fsVolume.getVolumeMap(bpid, fsVolume, ramDiskReplicaTracker)); | ||
| tempVolumeMap = |
There was a problem hiding this comment.
Only those in the last nsInfo will go into the tempVolumeMap?
There was a problem hiding this comment.
Earlier, addAll was adding all the k-v in the ProvidedVolumeReplicaMap into the VolumeReplicaMap (tempVolumeMap ). Which means the replicaMap associated with the ProvidedVolume will still be of type VolumeReplicaMap and not ProvidedVolumeReplicaMap. The ProvidedVolume will never invoke aliasMap request if it is not connected to ProvidedVolumeReplicaMap.
There was a problem hiding this comment.
I meant the semantics associated with the {{for (final NamespaceInfo nsInfo : nsInfos) }} is not the same as before right?
There was a problem hiding this comment.
Good catch. I will fix this this. Thanks !
|
Regarding your question about the relation between the failure and fs.cache, the fix has two paths Part 1: Note that an abfs block is searched for in the Hence one part of the fix is to search for block presence early enough and terminate the flow with Part 2: |
2 participants
Tiered block reads fail on DNs when FS caching is disabled and the
remote store requires credentials. This happens because the DN searches
the wrong remoteFS. For e.g. there is a default provided volume created
on DN. It uses the same alias map as other provided volumes. However it
uses the default conf object. While reading a block, the volume filds
the FileRegion for the required block from alias map. However, creation
of a secured file system object will fail as the config object will not
contain the desired credentials. So the DN will fail the read.