S3guard/hadoop 13786 committer #1
Conversation
steveloughran
force-pushed
the
s3guard/HADOOP-13786-committer
branch
from
11c5c4f to
284cd10
Compare
ehiggs
force-pushed
the
s3guard/HADOOP-13786-committer
branch
from
1874cdb to
a7d4bfa
Compare
|
Rebased. |
steveloughran
force-pushed
the
s3guard/HADOOP-13786-committer
branch
from
284cd10 to
5417f2f
Compare
|
heh, I've just gone and broken things by squashing patches |
steveloughran
force-pushed
the
s3guard/HADOOP-13786-committer
branch
from
5417f2f to
284cd10
Compare
There was a problem hiding this comment.
don't like imports to be moved around. As well as a nominal rule of layout, changes here make merging a nightmare
There was a problem hiding this comment.
no. it's "pendingset" lower case.
There was a problem hiding this comment.
never knew of this. good to use something else by other people.
There was a problem hiding this comment.
Ignoring the fact that case-changing classnames is something very unlikely to take reliably in macos unless you can delete the old one first, I'm unsure about this. Why. It's not really a "set" in terms of java lang collection. At the same time, I'm not that fussy about classnames, though I do want it to stay all lower in the filesystem.
There was a problem hiding this comment.
Well it's two words. Pending and Set. So it's camel cased. Maybe PendingBatch?
The issue about case changing and macos doesn't make sense. The patch wasn't merged yet so no one will have issues with the name change.
|
overall +1 to the new pair class, guava stripping, etc |
ehiggs
force-pushed
the
s3guard/HADOOP-13786-committer
branch
from
a7d4bfa to
d3119f1
Compare
|
I rebased and added a few patched addressing @steveloughran's comments. I also fixed up some grammar or sentence fragments in the architecture doc. |
steveloughran
merged this pull request
into
steveloughran:s3guard/HADOOP-13786-committer
|
thanks, merged in |
…tify slow points Listing files is surprisingly slow. Theories * the listFiles() call is the wrong scan for local (and HDFS?) * over use of java 8 streams/maps, etc explore #2 and then worry about #1. We must stay with listFiles for the magic committers scans of s3, but for the staging committers, we just need to flat list the source dir with a filter Change-Id: I7e29b6004e71b146500a95c9822c5eed17390fb4
2 participants
Various small fixes to the Hadoop s3guard patch set.