Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
438 commits
Select commit Hold shift + click to select a range
b174410
HDFS-7589. Break the dependency between libnative_mini_dfs and libhdf…
cnauroth Jan 8, 2015
786108d
YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed …
jian-he Jan 8, 2015
22af33b
HDFS-7579. Improve log reporting during block report rpc failure. Con…
cnauroth Jan 8, 2015
8f2e17e
HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop…
Jan 9, 2015
ee3b5c6
YARN-3002. YARN documentation needs updating post-shell rewrite (aw)
aw-was-here Jan 9, 2015
bc251b7
HADOOP-11464. Reinstate support for launching Hadoop processes on Win…
cnauroth Jan 9, 2015
63947cc
HADOOP-9992. Modify the NN loadGenerator to optionally run as a MapRe…
Jan 10, 2015
4febc24
HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed…
Jing9 Jan 10, 2015
ce3d0c1
YARN-2956. Added missing links in YARN documentation. Contributed by …
jian-he Jan 10, 2015
d77a397
YARN-3014. Replaces labels on a host should update all NM's labels on…
jian-he Jan 10, 2015
1c5eb1b
HADOOP-11400. GraphiteSink does not reconnect to Graphite after 'brok…
raviprak Jan 10, 2015
aea79b1
HDFS-7596. NameNode should prune dead storages from storageMap. Contr…
cnauroth Jan 10, 2015
82c57ef
HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin. …
y-liu Jan 11, 2015
3e29d9b
YARN-3027. Scheduler should use totalAvailable resource from node ins…
rkanter Jan 12, 2015
6ee81a7
YARN-2957. Create unit test to automatically compare YarnConfiguratio…
rkanter Jan 12, 2015
0fd8ea1
HDFS-7326. Add documentation for hdfs debug commands (Vijay Bhat via …
Jan 12, 2015
3124609
HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade …
kihwal Jan 12, 2015
565a494
HADOOP-11473. test-patch says "-1 overall" even when all checks are +1
raviprak-altiscale Jan 12, 2015
db2abfc
HDFS-7598. Remove dependency on old version of Guava in TestDFSClient…
Jan 13, 2015
26f3b6e
HDFS-5445. PacketReceiver populates the packetLen field in PacketHead…
Jan 13, 2015
210f5c5
HDFS-7600. Refine hdfs admin classes to reuse common code. Contribute…
Jing9 Jan 13, 2015
2e213fc
YARN-2643. Don't create a new DominantResourceCalculator on every Fai…
rkanter Jan 13, 2015
b61ee33
HADOOP-11465. Fix findbugs warnings in hadoop-gridmix. (Contributed b…
arp7 Jan 13, 2015
215c2af
HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov.
pjeli Jan 13, 2015
9953e92
HDFS-7056. Snapshot support for truncate. Contributed by Konstantin S…
shvachko Jan 13, 2015
520f8bc
MAPREDUCE-6173. Document the configuration of deploying MR over
xgong Jan 13, 2015
69b6764
HDFS-7570. SecondaryNameNode need twice memory when calling reloadFro…
cnauroth Jan 14, 2015
6da20dd
YARN-2637. Fixed max-am-resource-percent calculation in CapacitySched…
jian-he Jan 14, 2015
d8b1de8
YARN-3019. Make work-preserving-recovery the default mechanism for RM…
JunpingDu Jan 14, 2015
c8f4e22
MAPREDUCE-6210. Use getApplicationAttemptId() instead of getApplicati…
aajisaka Jan 14, 2015
58d8821
YARN-2807. Option "--forceactive" not works as described in usage of
xgong Jan 14, 2015
69680cf
HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectu…
Jan 14, 2015
15f950c
HDFS-2219. Change fsck to support fully qualified paths so that a par…
Jan 14, 2015
e350126
HDFS-5782. Change BlockListAsLongs constructor to take Replica as par…
Jan 14, 2015
1cf49d5
HDFS-7067. ClassCastException while using a key created by keytool to…
Jan 15, 2015
446cb14
Add CHANGES.txt for HADOOP-11481
Jan 15, 2015
0b56e43
YARN-2217. [YARN-1492] Shared cache client side changes. (Chris Trezz…
kambatla Jan 15, 2015
acf951c
HADOOP-11483. HardLink.java should use the jdk7 createLink method
aajisaka Jan 15, 2015
6f0d9f4
YARN-3005. [JDK7] Use switch statement for String instead of if-else …
aajisaka Jan 15, 2015
51ac0b2
HDFS-7581. HDFS documentation needs updating post-shell rewrite (aw)
aw-was-here Jan 15, 2015
51df20c
HADOOP-11318. Update the document for hadoop fs -stat
aajisaka Jan 15, 2015
9910bd5
HDFS-7591. hdfs classpath command should support same options as hado…
arp7 Jan 15, 2015
f2ad2b5
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs…
jian-he Jan 15, 2015
30c4c6a
HADOOP-8757. Metrics should disallow names with invalid characters (r…
rkanter Jan 15, 2015
2408f31
HDFS-7615. Remove longReadLock. Contributed by Kihwal Lee.
kihwal Jan 15, 2015
93b7e86
HDFS-7457. DatanodeID generates excessive garbage. Contributed by Dar…
kihwal Jan 15, 2015
e4c9429
HADOOP-11350. The size of header buffer of HttpServer is too small wh…
Jan 16, 2015
64002ff
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin …
y-liu Jan 15, 2015
e473435
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerRe…
JunpingDu Jan 16, 2015
bf40a1f
HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel)
steveloughran Jan 16, 2015
a3a35b1
HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by C…
cnauroth Jan 16, 2015
c3efcde
HDFS-7632. MiniDFSCluster configures DataNode data directories incorr…
cnauroth Jan 16, 2015
af9c4f5
HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by…
shvachko Jan 16, 2015
8069247
YARN-2984. Metrics for container's actual memory usage. (kasha)
kambatla Jan 17, 2015
0fcf6ea
YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-ser…
zjshen14 Jan 17, 2015
d7174e9
HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemSt…
steveloughran Jan 17, 2015
b13d36a
HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas D…
steveloughran Jan 17, 2015
f2215d1
HDFS-7573. Consolidate the implementation of delete() into a single c…
Dec 26, 2014
d2298a3
YARN-3070. TestRMAdminCLI#testHelp fails for transitionToActive comma…
JunpingDu Jan 19, 2015
bc10940
YARN-3071. Remove invalid char from sample conf in doc of FairSchedul…
aajisaka Jan 19, 2015
ecda1cd
HDFS-7637. Fix the check condition for reserved path. Contributed by …
Jing9 Jan 19, 2015
29e7c31
HADOOP-10668. TestZKFailoverControllerStress#testExpireBackAndForth o…
cnauroth Jan 19, 2015
050512c
HDFS-5631. Change BlockMetadataHeader.readHeader(..), ChunkChecksum c…
Jan 19, 2015
f9ab802
YARN-3015. yarn classpath command should support same options as hado…
cnauroth Jan 19, 2015
0b025dd
YARN-2933. Capacity Scheduler preemption policy should only consider …
wangdatan Jan 20, 2015
548dad9
HDFS-7640. print NFS Client in the NFS log. Contributed by Brandon Li.
Jan 20, 2015
b438ec4
HDFS-7638: Small fix and few refinements for FSN#truncate. (yliu)
y-liu Jan 19, 2015
26da8ac
HADOOP-11489. Dropping dependency on io.netty from hadoop-nfs' pom.xm…
oza Jan 20, 2015
c554f07
HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by…
arp7 Jan 20, 2015
55f6a06
HDFS-7634. Fix CHANGES.txt
arp7 Jan 20, 2015
7ba6504
YARN-2731. Fixed RegisterApplicationMasterResponsePBImpl to properly …
wangdatan Jan 20, 2015
3111cfb
HDFS-7623. Add htrace configuration properties to core-default.xml an…
y-liu Jan 20, 2015
a6d1f21
HDFS-7641. Update archival storage user doc for list/set/get block st…
y-liu Jan 20, 2015
9014305
HDFS-7496. Fix FsVolume removal race conditions on the DataNode by re…
Jan 21, 2015
8c130ae
HDFS-7496: add to CHANGES.txt
Jan 21, 2015
ef4453d
HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu…
Jan 21, 2015
4995923
HDFS-7610. Add CHANGES.txt
Jan 21, 2015
cab6f96
HDFS-7643. Test case to ensure lazy persist files cannot be truncated…
arp7 Jan 21, 2015
bd457d3
HADOOP-11256. Some site docs have inconsistent appearance (Masatake I…
aw-was-here Jan 21, 2015
cf76280
HADOOP-11327. BloomFilter#not() omits the last bit, resulting in an i…
jlowe Jan 21, 2015
a865d7c
HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on AR…
Jan 21, 2015
dc9978a
HADOOP-11484: move CHANGES.txt entry to 3.0
Jan 21, 2015
8a690f9
HDFS-3443. Fix NPE when namenode transition to active during startup …
Jan 21, 2015
ce16949
HADOOP-10668. Addendum patch to fix TestZKFailoverController. Contrib…
cnauroth Jan 21, 2015
a8c59ba
HDFS-7548. Corrupt block reporting delayed until datablock scanner th…
kihwal Jan 21, 2015
c1ad0a8
MAPREDUCE-3283. mapred classpath CLI does not display the complete cl…
cnauroth Jan 21, 2015
f2a8eca
HADOOP-11490. Expose truncate API via FileSystem and shell command. C…
shvachko Jan 21, 2015
4ee89ce
HADOOP-11466. FastByteComparisons: do not use UNSAFE_COMPARER on the …
Jan 22, 2015
a691658
MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatical…
gerashegalov Jan 22, 2015
df4edd9
HDFS-7430. Refactor the BlockScanner to use O(1) memory and use multi…
Dec 17, 2014
73309fb
YARN-3078. LogCLIHelpers lacks of a blank space before string 'does n…
oza Jan 22, 2015
ea85ae4
HADOOP-11209. Configuration#updatingResource/finalParameters are not …
oza Jan 22, 2015
3f90fca
HADOOP-11500. InputStream is left unclosed in ApplicationClassLoader.…
oza Jan 22, 2015
abb0115
HDFS-7575. Upgrade should generate a unique storage ID for each volum…
arp7 Jan 22, 2015
c13d501
HADOOP-11008. Remove duplicated description about proxy-user in site …
aw-was-here Jan 22, 2015
8262acf
HDFS-7575. Fix CHANGES.txt
arp7 Jan 22, 2015
d3b92a1
HDFS-3519. Checkpoint upload may interfere with a concurrent saveName…
cnauroth Jan 23, 2015
c265a0e
HADOOP-11493. Fix some typos in kms-acls.xml description. (Contribute…
aajisaka Jan 23, 2015
8a96c6d
HDFS-7660. BlockReceiver#close() might be called multiple times, whic…
y-liu Jan 22, 2015
a06d2d6
YARN-3082. Non thread safe access to systemCredentials in NodeHeartbe…
oza Jan 23, 2015
7d62e90
YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disab…
oza Jan 23, 2015
5177c14
HADOOP-11507 Hadoop RPC Authentication problem with different user lo…
steveloughran Jan 23, 2015
2e02d86
HADOOP-11482. Use correct UGI when KMSClientProvider is called by a p…
umbrant Jan 23, 2015
d56e750
HDFS-7667. Various typos and improvements to HDFS Federation doc (Ch…
aw-was-here Jan 23, 2015
f3c4294
HDFS-7644. minor typo in HttpFS doc (Charles Lamb via aw)
aw-was-here Jan 23, 2015
cc9ed52
HDFS-3750. API docs don't include HDFS (Jolly Chen via aw)
aw-was-here Jan 23, 2015
f8b1ce9
HDFS-7320. The appearance of hadoop-hdfs-httpfs site docs is inconsis…
aw-was-here Jan 23, 2015
27c98cc
HDFS-7659. truncate should check negative value of the new length. Co…
y-liu Jan 24, 2015
e461d62
Move truncate related messages in CHANGES.txt to branch 2 section. (shv)
shvachko Jan 25, 2015
38cbafd
HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611. Contribute…
shvachko Jan 25, 2015
5adb125
HADOOP-11419 improve hadoop-maven-plugins. (Hervé Boutemy via stevel)
steveloughran Jan 25, 2015
f343538
YARN-3024. LocalizerRunner should give DIE action when all resources are
xgong Jan 26, 2015
b2de93d
HADOOP-11450. Cleanup DistCpV1 not to use deprecated methods and fix …
oza Jan 26, 2015
21f5c51
HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by…
kihwal Jan 26, 2015
b327379
YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if nativ…
jlowe Jan 26, 2015
4216800
MAPREDUCE-6141. History server leveldb recovery store. Contributed by…
jlowe Jan 26, 2015
7ab153e
HADOOP-11499. Check of executorThreadsStarted in ValueQueue#submitRef…
jlowe Jan 26, 2015
7c85bb3
HADOOP-11514. Raw Erasure Coder API for concrete encoding and decodin…
umamaheswararao Jan 29, 2015
9feb6b3
HADOOP-11466: move to 2.6.1
Jan 26, 2015
0111b57
HADOOP-6221 RPC Client operations cannot be interrupted (stevel)
steveloughran Jan 26, 2015
79d0787
HADOOP-11509. Change parsing sequence in GenericOptionsParser to parse
xgong Jan 26, 2015
21c74e6
YARN-3092. Created a common ResourceUsage class to track labeled reso…
jian-he Jan 26, 2015
6b67115
HDFS-49. MiniDFSCluster.stopDataNode will always shut down a node in …
steveloughran Jan 27, 2015
cfb00e7
YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. C…
oza Jan 27, 2015
803dd11
HADOOP-4297. Enable Java assertions when running tests. Contributed b…
Jan 27, 2015
2323609
HDFS-7683. Combine usages and percent stats in NameNode UI. Contribut…
Jan 27, 2015
32548f4
HDFS-3689. Add support for variable length block. Contributed by Jing…
Jing9 Jan 27, 2015
ec5b307
YARN-3011. Possible IllegalArgumentException in ResourceLocalizationS…
jian-he Jan 27, 2015
a966dff
HADOOP-10626. Limit Returning Attributes for LDAP search. Contributed…
atm Jan 27, 2015
94f3ffe
YARN-2897. CrossOriginFilter needs more log statements (Mit Desai via…
jteagles Jan 27, 2015
928bb8e
HDFS-7566. Remove obsolete entries from hdfs-default.xml (Ray Chiang …
aw-was-here Jan 27, 2015
c994cbc
YARN-3028. Better syntax for replaceLabelsOnNode in RMAdmin CLI. Cont…
wangdatan Jan 27, 2015
c690f6a
YARN-2932. Add entry for preemptable status (enabled/disabled) to sch…
wangdatan Jan 27, 2015
dad98fb
HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (y…
y-liu Jan 27, 2015
93f6e7a
HADOOP-11469. KMS should skip default.key.acl and whitelist.key.acl w…
y-liu Jan 27, 2015
f05809c
HDFS-7675. Remove unused member DFSClient.spanReceiverHost (cmccabe)
Jan 28, 2015
3397c23
HADOOP-11317. Increment SLF4J version to 1.7.10. Contributed by Tim R…
oza Jan 28, 2015
3a5a90d
HADOOP-11316. "mvn package -Pdist,docs -DskipTests -Dtar" fails becau…
oza Jan 28, 2015
5227f6f
HDFS-6673. Add delimited format support to PB OIV tool. Contributed b…
umbrant Jan 28, 2015
620dd74
YARN-3103. AMRMClientImpl does not update AMRM token properly. Contri…
jian-he Jan 28, 2015
d4edaed
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blo…
Jing9 Jan 28, 2015
dd5946a
MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToke…
jian-he Jan 28, 2015
8ced72c
HDFS-7681. Change ReplicaInputStreams constructor to take InputStream…
Jan 28, 2015
840d214
HDFS-7423. various typos and message formatting fixes in nfs daemon a…
y-liu Jan 28, 2015
1ca84c1
YARN-3079. Scheduler should also update maximumAllocation when update…
wangdatan Jan 29, 2015
899b5e1
HADOOP-10525. Remove DRFA.MaxBackupIndex config from log4j.properties…
aajisaka Jan 29, 2015
dd72a41
MAPREDUCE-6231. Grep example job is not working on a fully-distribute…
aajisaka Jan 29, 2015
a64335e
HADOOP-10574. Bump the maven plugin versions too -moving the numbers …
aajisaka Jan 29, 2015
4941e5e
YARN-2428. LCE default banned user list should have yarn (Varun Saxen…
aw-was-here Jan 29, 2015
570914e
HADOOP-9907. Webapp http://hostname:port/metrics link is not working.…
aajisaka Jan 29, 2015
0c15190
HDFS-7603. The background replication queue initialization may not le…
kihwal Jan 29, 2015
a0501d0
YARN-3029. FSDownload.unpack() uses local locale for FS case conversi…
oza Jan 29, 2015
afac550
HADOOP-11441. Hadoop-azure: Change few methods scope to public. Contr…
cnauroth Jan 29, 2015
d0f21bd
MAPREDUCE-6150. Update document of Rumen (Masatake Iwasaki via aw)
aw-was-here Jan 29, 2015
a38ace1
YARN-3108. ApplicationHistoryServer doesn't process -D arguments (Cha…
jteagles Jan 29, 2015
8de7e21
HADOOP-11403. Avoid using sys_errlist on Solaris, which lacks support…
Jan 29, 2015
cec3166
HADOOP-11523. StorageException complaining " no lease ID" when updati…
cnauroth Jan 30, 2015
fa38107
MAPREDUCE-6151. Update document of GridMix (Masatake Iwasaki via aw)
aw-was-here Jan 30, 2015
4acb207
HDFS-7706. Switch BlockManager logging to use slf4j.
umbrant Jan 30, 2015
a3394ba
HDFS-7339. Allocating and persisting block groups in NameNode. Contri…
zhe-thoughts Jan 31, 2015
d4b9b40
HADOOP-11534. Minor improvements for raw erasure coders ( Contributed…
Feb 2, 2015
d8ddc15
YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use Resour…
jian-he Jan 30, 2015
97c76ab
HADOOP-9137. Support connection limiting in IPC server. Contributed b…
kihwal Jan 30, 2015
03b87a1
HADOOP-11498. Bump the version of HTrace to 3.1.0-incubating (Masatak…
Jan 31, 2015
22bfdec
YARN-3077. Fixed RM to create zk root path recursively. Contributed b…
jian-he Jan 31, 2015
4361326
HADOOP-11432. Fix SymlinkBaseTest#testCreateLinkUsingPartQualPath2. (…
gerashegalov Jan 31, 2015
7419212
HADOOP-11488. Difference in default connection timeout for S3A FS. Co…
Jan 31, 2015
89268fa
HADOOP-11494. Lock acquisition on WrappedInputStream#unwrappedRpcBuff…
Feb 2, 2015
0f374ad
YARN-3113. Release audit warning for Sorting icons.psd. Contributed b…
kihwal Feb 2, 2015
aaeb43c
HADOOP-10181. GangliaContext does not work with multicast ganglia set…
cnauroth Feb 2, 2015
1c1aae4
HADOOP-11442. hadoop-azure: Create test jar. Contributed by Shashank …
cnauroth Feb 2, 2015
3834375
YARN-2808. Made YARN CLI list attempt’s finished containers of a runn…
zjshen14 Feb 2, 2015
164410b
YARN-2216. Fixed the change log.
zjshen14 Feb 2, 2015
a97fd5d
Move HDFS-5631, HDFS-5782 and HDFS-7681 to branch-2.
Feb 2, 2015
1b396d9
HDFS-7696. In FsDatasetImpl, the getBlockInputStream(..) and getTmpIn…
Feb 2, 2015
7e47b03
MAPREDUCE-6143. add configuration for mapreduce speculative execution…
rkanter Feb 2, 2015
6f47c84
HDFS-6651. Deletion failure can leak inodes permanently. Contributed …
Feb 3, 2015
117fd5f
YARN-3085. Application summary should include the application type. C…
jlowe Feb 3, 2015
3330dea
HADOOP-11045. Introducing a tool to detect flaky tests of hadoop jenk…
oza Feb 3, 2015
adf943e
YARN-3022. Expose Container resource information from NodeManager for…
rkanter Feb 3, 2015
663b759
HADOOP-11529. Fix findbugs warnings in hadoop-archives. Contributed b…
Feb 3, 2015
f0ebd20
HDFS-7721. The HDFS BlockScanner may run fast during the first hour (…
Feb 3, 2015
89e440a
YARN-3098. Created common QueueCapacities class in Capacity Scheduler…
jian-he Feb 3, 2015
99ef0c1
HDFS-7707. Edit log corruption due to delayed block removal again. Co…
kihwal Feb 3, 2015
0fe9b25
YARN-3075. NodeLabelsManager implementation to retrieve label to node…
wangdatan Feb 3, 2015
be0b649
YARN-1393. SLS: Add how-to-use instructions. (Wei Yan via kasha)
kambatla Feb 3, 2015
c6ca702
MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructor…
aajisaka Feb 3, 2015
8fb3f66
HDFS-7712. Switch blockStateChangeLog to use slf4j.
umbrant Feb 3, 2015
b6c8a0f
YARN-3056. Add verification for containerLaunchDuration in
xgong Feb 3, 2015
4224297
YARN-2543. Made resource usage be published to the timeline server to…
zjshen14 Feb 4, 2015
6369fdf
YARN-3058. Fix error message of tokens' activation delay configuratio…
oza Feb 4, 2015
47278a2
HADOOP-11544. Remove unused configuration keys for tracing. Contribut…
aajisaka Feb 4, 2015
258b1a4
HADOOP-11546. Checkstyle failing: Unable to instantiate DoubleChecked…
oza Feb 4, 2015
d0399a1
MAPREDUCE-6243. Fix findbugs warnings in hadoop-rumen. Contributed by…
aajisaka Feb 4, 2015
03d05b3
HADOOP-11548. checknative should display a nicer error message when o…
cnauroth Feb 4, 2015
12cbbd2
HDFS-7734. Class cast exception in NameNode#main. Contributed by Yi Liu.
umbrant Feb 4, 2015
b537751
HADOOP-11547. hadoop-common native compilation fails on Windows due t…
cnauroth Feb 4, 2015
0809cb7
HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some i…
Feb 4, 2015
fe770f6
YARN-1723. AMRMClientAsync missing blacklist addition and removal
sidseth Feb 4, 2015
4fe401c
HADOOP-11492. Bump up curator version to 2.7.1. (Arun Suresh and Kart…
kambatla Feb 4, 2015
300a6f9
HADOOP-11549. flaky test detection tool failed to handle special cont…
aajisaka Feb 4, 2015
d31af04
YARN-3123. Made YARN CLI show a single completed container even if th…
zjshen14 Feb 5, 2015
b08f019
HDFS-7733. NFS: readdir/readdirplus return null directory attribute o…
arp7 Feb 5, 2015
484aff7
HADOOP-11460. Deprecate shell vars (John Smith via aw)
aw-was-here Feb 5, 2015
fa58390
HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smit…
aw-was-here Feb 5, 2015
6722247
MAPREDUCE-6059. Speed up history server startup time (Siqi Li via aw)
aw-was-here Feb 5, 2015
d2f66c0
MAPREDUCE-5988. Fix dead links to the javadocs in mapreduce project. …
aajisaka Feb 5, 2015
b8ea1c6
HADOOP-10976. moving the source code of hadoop-tools docs to the dire…
aw-was-here Feb 5, 2015
07c2147
HADOOP-10062. race condition in MetricsSystemImpl#publishMetricsNow t…
JunpingDu Feb 5, 2015
9b559cb
HADOOP-11543. Improve help message for hadoop/yarn command. Contribut…
oza Feb 5, 2015
8959852
HDFS-7709. Fix findbug warnings in httpfs. Contributed by Rakesh R.
oza Feb 5, 2015
0a4089a
HADOOP-11463 Replace method-local TransferManager object with S3AFile…
steveloughran Feb 5, 2015
14d8926
HADOOP-7713. dfs -count -q should label output column (Jonathan Allen…
aw-was-here Feb 5, 2015
c3973f1
Update CHANGES.txt to move MR-6059 to 2.7
jlowe Feb 5, 2015
650542e
YARN-3101. In Fair Scheduler, fix canceling of reservations for excee…
sryza Feb 5, 2015
4ea2e7d
HADOOP-9044. add FindClass main class to provide classpath checking o…
aw-was-here Feb 5, 2015
b7eb696
HDFS-7270. Add congestion signaling capability to DataNode write prot…
Feb 5, 2015
bb13b9c
YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting pe…
jlowe Feb 5, 2015
c6b6a0d
HADOOP-9044. add FindClass main class to provide classpath checking o…
aw-was-here Feb 5, 2015
4b06a86
MAPREDUCE-6186. Redundant call to requireJob() while displaying the c…
jlowe Feb 5, 2015
cf8d8af
MAPREDUCE-6233. org.apache.hadoop.mapreduce.TestLargeSort.testLargeSo…
rkanter Feb 5, 2015
db29be8
YARN-3149. Fix typo in message for invalid application id. Contributed
xgong Feb 5, 2015
237996a
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler…
jian-he Feb 6, 2015
04fe613
HDFS-7655. Expose truncate API for Web HDFS. (yliu)
y-liu Feb 5, 2015
0f143b2
HDFS-7698. Fix locking on HDFS read statistics and add a method for c…
y-liu Feb 5, 2015
ee9c025
HADOOP-11526. Memory leak in Bzip2Compressor and Bzip2Decompressor. C…
cnauroth Feb 6, 2015
f9a837c
HADOOP-6964. Allow compact property description in xml (Kengo Seki vi…
aw-was-here Feb 6, 2015
e684e78
Move HADOOP-11543 from BUG-FIX to IMPROVEMENT in CHANGES.txt.
oza Feb 6, 2015
aad0211
HDFS-7741. Remove unnecessary synchronized in FSDataInputStream and H…
y-liu Feb 5, 2015
1664362
YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationH…
acmurthy Feb 6, 2015
9258205
YARN-1537. Fix race condition in TestLocalResourcesTrackerImpl.testLo…
acmurthy Feb 6, 2015
a5e5f30
HADOOP-11506. Configuration variable expansion regex expensive for lo…
gerashegalov Jan 31, 2015
bde495d
HADOOP-11535 TableMapping related tests failed due to 'successful' r…
steveloughran Feb 6, 2015
04290f6
HDFS-7732. Fix the order of the parameters in DFSConfigKeys. Contribu…
aajisaka Feb 6, 2015
0d1ddeb
YARN-2694. Ensure only single node label specified in ResourceRequest…
jian-he Feb 6, 2015
2bb2df4
YARN-3089. LinuxContainerExecutor does not handle file arguments to d…
jlowe Feb 6, 2015
6d4e9e3
YARN-3144. Configuration for making delegation token failures to time…
jlowe Feb 6, 2015
425c4b6
HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Tho…
steveloughran Feb 6, 2015
2fc501a
HADOOP-11485. Pluggable shell integration (aw)
aw-was-here Feb 6, 2015
e031ade
HADOOP-11541. Raw XOR coder
Feb 7, 2015
8652e8f
YARN-3143. RM Apps REST API can return NPE or entries missing id and …
jlowe Feb 6, 2015
bdc21e1
HDFS-7720. Quota by Storage Type API, tools and ClientNameNode Protoc…
arp7 Feb 7, 2015
e907a58
HDFS-7710. Remove dead code in BackupImage.java. Contributed by Xiaoy…
aajisaka Feb 7, 2015
d0b53b4
HDFS-7738. Revise the exception message for recover lease; add more t…
Feb 7, 2015
2fef715
HDFS-7751. Fix TestHDFSCLI for quota with storage type. Contributed …
Feb 7, 2015
26ec876
MAPREDUCE-6227. DFSIO for truncate. (shv via yliu)
y-liu Feb 7, 2015
63fd1ac
HDFS-7743. Code cleanup of BlockInfo and rename BlockInfo to BlockInf…
Jing9 Feb 8, 2015
b5625a7
Fix Compilation Error in TestAddBlockgroup.java after the merge
Jing9 Feb 9, 2015
71a67e9
Added the missed entry for commit of HADOOP-11541
Feb 9, 2015
c4362d5
HDFS-7652. Process block reports for erasure coded blocks. Contribute…
zhe-thoughts Feb 9, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
12 changes: 9 additions & 3 deletions BUILDING.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Build instructions for Hadoop
Requirements:

* Unix System
* JDK 1.6+
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
Expand Down Expand Up @@ -204,15 +204,19 @@ Building on Windows
Requirements:

* Windows System
* JDK 1.6+
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK or Visual Studio 2010 Professional
* Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
tools must be present on your PATH.

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/download/win.

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
Expand All @@ -221,6 +225,8 @@ download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

Cygwin is neither required nor supported.

----------------------------------------------------------------------------------
Building:

Expand Down
204 changes: 204 additions & 0 deletions dev-support/determine-flaky-tests-hadoop.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
#!/usr/bin/env python
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Given a jenkins test job, this script examines all runs of the job done
# within specified period of time (number of days prior to the execution
# time of this script), and reports all failed tests.
#
# The output of this script includes a section for each run that has failed
# tests, with each failed test name listed.
#
# More importantly, at the end, it outputs a summary section to list all failed
# tests within all examined runs, and indicate how many runs a same test
# failed, and sorted all failed tests by how many runs each test failed.
#
# This way, when we see failed tests in PreCommit build, we can quickly tell
# whether a failed test is a new failure, or it failed before and how often it
# failed, so to have idea whether it may just be a flaky test.
#
# Of course, to be 100% sure about the reason of a test failure, closer look
# at the failed test for the specific run is necessary.
#
import sys
import platform
sysversion = sys.hexversion
onward30 = False
if sysversion < 0x020600F0:
sys.exit("Minimum supported python version is 2.6, the current version is " +
"Python" + platform.python_version())

if sysversion == 0x030000F0:
sys.exit("There is a known bug with Python" + platform.python_version() +
", please try a different version");

if sysversion < 0x03000000:
import urllib2
else:
onward30 = True
import urllib.request

import datetime
import json as simplejson
import logging
from optparse import OptionParser
import time

# Configuration
DEFAULT_JENKINS_URL = "https://builds.apache.org"
DEFAULT_JOB_NAME = "Hadoop-Common-trunk"
DEFAULT_NUM_PREVIOUS_DAYS = 14

SECONDS_PER_DAY = 86400

# total number of runs to examine
numRunsToExamine = 0

""" Parse arguments """
def parse_args():
parser = OptionParser()
parser.add_option("-J", "--jenkins-url", type="string",
dest="jenkins_url", help="Jenkins URL",
default=DEFAULT_JENKINS_URL)
parser.add_option("-j", "--job-name", type="string",
dest="job_name", help="Job name to look at",
default=DEFAULT_JOB_NAME)
parser.add_option("-n", "--num-days", type="int",
dest="num_prev_days", help="Number of days to examine",
default=DEFAULT_NUM_PREVIOUS_DAYS)

(options, args) = parser.parse_args()
if args:
parser.error("unexpected arguments: " + repr(args))
return options

""" Load data from specified url """
def load_url_data(url):
if onward30:
ourl = urllib.request.urlopen(url)
codec = ourl.info().get_param('charset')
content = ourl.read().decode(codec)
data = simplejson.loads(content, strict=False)
else:
ourl = urllib2.urlopen(url)
data = simplejson.load(ourl, strict=False)
return data

""" List all builds of the target project. """
def list_builds(jenkins_url, job_name):
url = "%(jenkins)s/job/%(job_name)s/api/json?tree=builds[url,result,timestamp]" % dict(
jenkins=jenkins_url,
job_name=job_name)

try:
data = load_url_data(url)

except:
logging.error("Could not fetch: %s" % url)
raise
return data['builds']

""" Find the names of any tests which failed in the given build output URL. """
def find_failing_tests(testReportApiJson, jobConsoleOutput):
ret = set()
try:
data = load_url_data(testReportApiJson)

except:
logging.error(" Could not open testReport, check " +
jobConsoleOutput + " for why it was reported failed")
return ret

for suite in data['suites']:
for cs in suite['cases']:
status = cs['status']
errDetails = cs['errorDetails']
if (status == 'REGRESSION' or status == 'FAILED' or (errDetails is not None)):
ret.add(cs['className'] + "." + cs['name'])

if len(ret) == 0:
logging.info(" No failed tests in testReport, check " +
jobConsoleOutput + " for why it was reported failed.")
return ret

""" Iterate runs of specfied job within num_prev_days and collect results """
def find_flaky_tests(jenkins_url, job_name, num_prev_days):
global numRunsToExamine
all_failing = dict()
# First list all builds
builds = list_builds(jenkins_url, job_name)

# Select only those in the last N days
min_time = int(time.time()) - SECONDS_PER_DAY * num_prev_days
builds = [b for b in builds if (int(b['timestamp']) / 1000) > min_time]

# Filter out only those that failed
failing_build_urls = [(b['url'] , b['timestamp']) for b in builds
if (b['result'] in ('UNSTABLE', 'FAILURE'))]

tnum = len(builds)
num = len(failing_build_urls)
numRunsToExamine = tnum
logging.info(" THERE ARE " + str(num) + " builds (out of " + str(tnum)
+ ") that have failed tests in the past " + str(num_prev_days) + " days"
+ ((".", ", as listed below:\n")[num > 0]))

for failed_build_with_time in failing_build_urls:
failed_build = failed_build_with_time[0];
jobConsoleOutput = failed_build + "Console";
testReport = failed_build + "testReport";
testReportApiJson = testReport + "/api/json";

ts = float(failed_build_with_time[1]) / 1000.
st = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
logging.info("===>%s" % str(testReport) + " (" + st + ")")
failing = find_failing_tests(testReportApiJson, jobConsoleOutput)
if failing:
for ftest in failing:
logging.info(" Failed test: %s" % ftest)
all_failing[ftest] = all_failing.get(ftest,0)+1

return all_failing

def main():
global numRunsToExamine
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.INFO)

# set up logger to write to stdout
soh = logging.StreamHandler(sys.stdout)
soh.setLevel(logging.INFO)
logger = logging.getLogger()
logger.removeHandler(logger.handlers[0])
logger.addHandler(soh)

opts = parse_args()
logging.info("****Recently FAILED builds in url: " + opts.jenkins_url
+ "/job/" + opts.job_name + "")

all_failing = find_flaky_tests(opts.jenkins_url, opts.job_name,
opts.num_prev_days)
if len(all_failing) == 0:
raise SystemExit(0)
logging.info("\nAmong " + str(numRunsToExamine) + " runs examined, all failed "
+ "tests <#failedRuns: testName>:")

# print summary section: all failed tests sorted by how many times they failed
for tn in sorted(all_failing, key=all_failing.get, reverse=True):
logging.info(" " + str(all_failing[tn])+ ": " + tn)

if __name__ == "__main__":
main()
72 changes: 0 additions & 72 deletions dev-support/test-patch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -857,74 +857,6 @@ findModules () {
rm $TMP_MODULES
echo $CHANGED_MODULES
}
###############################################################################
### Run the test-contrib target
runContribTests () {
echo ""
echo ""
echo "======================================================================"
echo "======================================================================"
echo " Running contrib tests."
echo "======================================================================"
echo "======================================================================"
echo ""
echo ""

if [[ `$GREP -c 'test-contrib' build.xml` == 0 ]] ; then
echo "No contrib tests in this project."
return 0
fi

### Kill any rogue build processes from the last attempt
$PS auxwww | $GREP ${PROJECT_NAME}PatchProcess | $AWK '{print $2}' | /usr/bin/xargs -t -I {} /bin/kill -9 {} > /dev/null

#echo "$ANT_HOME/bin/ant -Dversion="${VERSION}" $ECLIPSE_PROPERTY -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib"
#$ANT_HOME/bin/ant -Dversion="${VERSION}" $ECLIPSE_PROPERTY -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib
echo "NOP"
if [[ $? != 0 ]] ; then
JIRA_COMMENT="$JIRA_COMMENT

{color:red}-1 contrib tests{color}. The patch failed contrib unit tests."
return 1
fi
JIRA_COMMENT="$JIRA_COMMENT

{color:green}+1 contrib tests{color}. The patch passed contrib unit tests."
return 0
}

###############################################################################
### Run the inject-system-faults target
checkInjectSystemFaults () {
echo ""
echo ""
echo "======================================================================"
echo "======================================================================"
echo " Checking the integrity of system test framework code."
echo "======================================================================"
echo "======================================================================"
echo ""
echo ""

### Kill any rogue build processes from the last attempt
$PS auxwww | $GREP ${PROJECT_NAME}PatchProcess | $AWK '{print $2}' | /usr/bin/xargs -t -I {} /bin/kill -9 {} > /dev/null

#echo "$ANT_HOME/bin/ant -Dversion="${VERSION}" -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no -Dcompile.c++=yes -Dforrest.home=$FORREST_HOME inject-system-faults"
#$ANT_HOME/bin/ant -Dversion="${VERSION}" -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no -Dcompile.c++=yes -Dforrest.home=$FORREST_HOME inject-system-faults
echo "NOP"
return 0
if [[ $? != 0 ]] ; then
JIRA_COMMENT="$JIRA_COMMENT

{color:red}-1 system test framework{color}. The patch failed system test framework compile."
return 1
fi
JIRA_COMMENT="$JIRA_COMMENT

{color:green}+1 system test framework{color}. The patch passed system test framework compile."
return 0
}

###############################################################################
### Submit a comment to the defect's Jira
submitJiraComment () {
Expand Down Expand Up @@ -1059,11 +991,7 @@ checkReleaseAuditWarnings
if [[ $JENKINS == "true" || $RUN_TESTS == "true" ]] ; then
runTests
(( RESULT = RESULT + $? ))
runContribTests
(( RESULT = RESULT + $? ))
fi
checkInjectSystemFaults
(( RESULT = RESULT + $? ))
JIRA_COMMENT_FOOTER="Test results: $BUILD_URL/testReport/
$JIRA_COMMENT_FOOTER"

Expand Down
4 changes: 2 additions & 2 deletions hadoop-assemblies/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@
<configuration>
<rules>
<requireMavenVersion>
<version>[3.0.0,)</version>
<version>${enforced.maven.version}</version>
</requireMavenVersion>
<requireJavaVersion>
<version>1.6</version>
<version>${enforced.java.version}</version>
</requireJavaVersion>
</rules>
</configuration>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@
</includes>
<fileMode>0755</fileMode>
</fileSet>
<fileSet>
<directory>${basedir}/src/main/shellprofile.d</directory>
<outputDirectory>/libexec/shellprofile.d</outputDirectory>
<fileMode>0755</fileMode>
</fileSet>
<fileSet>
<directory>${basedir}/src/main/bin</directory>
<outputDirectory>/sbin</outputDirectory>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@
</includes>
<fileMode>0755</fileMode>
</fileSet>
<fileSet>
<directory>shellprofile.d</directory>
<outputDirectory>libexec/shellprofile.d</outputDirectory>
<fileMode>0755</fileMode>
</fileSet>
<fileSet>
<directory>bin</directory>
<outputDirectory>sbin</outputDirectory>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@
</includes>
<fileMode>0755</fileMode>
</fileSet>
<fileSet>
<directory>hadoop-yarn/shellprofile.d</directory>
<outputDirectory>libexec/shellprofile.d</outputDirectory>
<fileMode>0755</fileMode>
</fileSet>
<fileSet>
<directory>hadoop-yarn/bin</directory>
<outputDirectory>sbin</outputDirectory>
Expand Down
2 changes: 1 addition & 1 deletion hadoop-client/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.hadoop</groupId>
Expand Down
Loading