hadoop.git
13 hours agoHDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node... trunk
Viraj Jasani [Sat, 21 May 2022 01:28:59 +0000 (18:28 -0700)] 
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323)

Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
19 hours agoYARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's...
slfan1989 [Fri, 20 May 2022 20:19:42 +0000 (13:19 -0700)] 
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation (#4317)

21 hours agoHADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267)
Viraj Jasani [Fri, 20 May 2022 18:21:13 +0000 (11:21 -0700)] 
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
21 hours agoHDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229)
Ashutosh Gupta [Fri, 20 May 2022 17:53:14 +0000 (18:53 +0100)] 
HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
30 hours agoHADOOP-18120. Hadoop auth does not handle HTTP Headers in a case-insensitive way...
9uapaw [Fri, 20 May 2022 08:08:19 +0000 (10:08 +0200)] 
HADOOP-18120. Hadoop auth does not handle HTTP Headers in a case-insensitive way. Contributed by Janos Makai.

43 hours agoHADOOP-18245 Extend KMS related exceptions that get mapped to ConnectException (...
Ritesh H Shukla [Thu, 19 May 2022 20:20:24 +0000 (13:20 -0700)] 
HADOOP-18245 Extend KMS related exceptions that get mapped to ConnectException (#4329)

2 days agoYARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitial...
9uapaw [Mon, 16 May 2022 08:40:46 +0000 (10:40 +0200)] 
YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation

2 days agoYARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators...
9uapaw [Wed, 11 May 2022 16:07:04 +0000 (18:07 +0200)] 
YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe

2 days agoHDFS-16540. Addendum: Data locality is lost when DataNode pod restarts in kubernetes...
Hexiaoqiao [Wed, 18 May 2022 15:28:45 +0000 (23:28 +0800)] 
HDFS-16540. Addendum: Data locality is lost when DataNode pod restarts in kubernetes. (#4326)

3 days agoHADOOP-18229. Fix Hadoop-Common JavaDoc Errors (#4292)
slfan1989 [Wed, 18 May 2022 11:12:04 +0000 (04:12 -0700)] 
HADOOP-18229. Fix Hadoop-Common JavaDoc Errors (#4292)

Contributed by slfan1989

3 days agoHADOOP-18237. Upgrade Apache Xerces Java to 2.12.2 (#4318)
Ashutosh Gupta [Tue, 17 May 2022 19:34:12 +0000 (20:34 +0100)] 
HADOOP-18237. Upgrade Apache Xerces Java to 2.12.2 (#4318)

Upgrade Apache Xerces Java to 2.12.2 due to handle vulnerability CVE-2022-23437

Contributed by Ashutosh Gupta

3 days agoHADOOP-18198. Release 3.3.3: release notes and jdiff files.
Steve Loughran [Tue, 17 May 2022 18:00:09 +0000 (19:00 +0100)] 
HADOOP-18198. Release 3.3.3: release notes and jdiff files.

* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d

4 days agoHADOOP-18234. Fix s3a access point xml examples (#4309)
Ashutosh Gupta [Mon, 16 May 2022 16:47:14 +0000 (17:47 +0100)] 
HADOOP-18234. Fix s3a access point xml examples (#4309)

Contributed by Ashutosh Gupta

4 days agoYARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280)
Zilong Zhu [Mon, 16 May 2022 16:40:09 +0000 (00:40 +0800)] 
YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280)

* Update QueueConfigurationsPBImpl.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

5 days agoHADOOP-18228. Update hadoop-vote to use HADOOP_RC_VERSION dir (#4272)
Viraj Jasani [Mon, 16 May 2022 14:34:39 +0000 (07:34 -0700)] 
HADOOP-18228. Update hadoop-vote to use HADOOP_RC_VERSION dir (#4272)

Signed-off-by: stack@apache.org
5 days agoYARN-11092. Upgrade jquery ui to 1.13.1 (#4260)
Ashutosh Gupta [Mon, 16 May 2022 14:11:17 +0000 (15:11 +0100)] 
YARN-11092. Upgrade jquery ui to 1.13.1 (#4260)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
5 days agoYARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274)
slfan1989 [Sun, 15 May 2022 16:16:06 +0000 (09:16 -0700)] 
YARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274)

7 days agoYARN-10080. Support show app id on localizer thread pool (#4283)
Ashutosh Gupta [Fri, 13 May 2022 16:41:06 +0000 (17:41 +0100)] 
YARN-10080. Support show app id on localizer thread pool (#4283)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
7 days agoMAPREDUCE-7377. Remove unused imports in MapReduce project (#4299)
Ashutosh Gupta [Fri, 13 May 2022 16:34:19 +0000 (17:34 +0100)] 
MAPREDUCE-7377. Remove unused imports in MapReduce project (#4299)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
7 days agoYARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases...
Jian Chen [Fri, 13 May 2022 16:11:42 +0000 (09:11 -0700)] 
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110)

Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
8 days agoYARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2...
Szilard Nemeth [Sun, 8 May 2022 21:05:08 +0000 (23:05 +0200)] 
YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth

8 days agoRevert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" ...
Hui Fei [Fri, 13 May 2022 06:44:41 +0000 (14:44 +0800)] 
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306)

8 days agoHDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)
Felix Nguyen [Fri, 13 May 2022 06:28:53 +0000 (14:28 +0800)] 
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)

9 days agoYARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed...
Szilard Nemeth [Thu, 12 May 2022 11:42:06 +0000 (13:42 +0200)] 
YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok

9 days agoHDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmi...
singer-bin [Thu, 12 May 2022 08:53:09 +0000 (16:53 +0800)] 
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122)

9 days agoHADOOP-18193:Support nested mount points in INodeTree
Lei Yang [Mon, 9 May 2022 18:52:15 +0000 (11:52 -0700)] 
HADOOP-18193:Support nested mount points in INodeTree

Fixes #4181

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
9 days agoHDFS-16465. Remove redundant strings.h inclusions (#4279)
Gautham B A [Wed, 11 May 2022 17:34:22 +0000 (23:04 +0530)] 
HDFS-16465. Remove redundant strings.h inclusions (#4279)

* Remove redundant strings.h inclusions

* strings.h was included in a bunch of
  C/C++ files and were redundant.
* Also, strings.h is not available on
  Windows and thus isn't cross-platform
  compatible.

* Build for all platforms in CI

* Revert "Build for all platforms in CI"

This reverts commit 2650f047bd6791a5908cfbe50cc8e70d42c512cb.

* Debug failure on Centos 8

* Skipping pipeline run on
  Centos 7 to debug the
  failure on Centos 8.

* Revert "Debug failure on Centos 8"

This reverts commit e365e34d6fab9df88f4df622910ddb28a8c8796f.

9 days agoYARN-11114. RMWebServices returns only apps matching exactly the submitted queue...
Szilard Nemeth [Wed, 20 Apr 2022 17:39:47 +0000 (19:39 +0200)] 
YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth

10 days agoMAPREDUCE-7379. RMContainerRequestor#makeRemoteRequest has confusing log message...
Szilard Nemeth [Wed, 11 May 2022 14:55:19 +0000 (16:55 +0200)] 
MAPREDUCE-7379. RMContainerRequestor#makeRemoteRequest has confusing log message. Contributed by Ashutosh Gupta

10 days agoYARN-10850. TimelineService v2 lists containers for all attempts when filtering for...
Szilard Nemeth [Wed, 11 May 2022 12:39:42 +0000 (14:39 +0200)] 
YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke

10 days agoYARN-11141. Capacity Scheduler does not support ambiguous queue names when moving...
Szilard Nemeth [Wed, 11 May 2022 12:27:34 +0000 (14:27 +0200)] 
YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori

10 days agoYARN-11130. removed unused import (#4276)
slfan1989 [Wed, 11 May 2022 06:44:52 +0000 (23:44 -0700)] 
YARN-11130. removed unused import (#4276)

10 days agoHDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes...
Viraj Jasani [Wed, 11 May 2022 00:10:03 +0000 (17:10 -0700)] 
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264)

Signed-off-by: Tao Li <tomscut@apache.org>
10 days agoHADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple...
hchaverr [Thu, 5 May 2022 19:39:58 +0000 (12:39 -0700)] 
HADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple times

Fixes #4266

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
11 days agoFix TestDFSStripedInputStreamWithRandomECPolicy (#4291)
daimin [Tue, 10 May 2022 00:01:16 +0000 (08:01 +0800)] 
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291)

11 days agoMAPREDUCE-7376. AggregateWordCount fetches wrong results. (#4257). Contributed by...
Ayush Saxena [Mon, 9 May 2022 17:26:14 +0000 (22:56 +0530)] 
MAPREDUCE-7376. AggregateWordCount fetches wrong results. (#4257). Contributed by Ayush Saxena.

Reviewed-by: Steve Loughran <stevel@apache.org>
12 days agoHDFS-16572. Fix typo in readme of hadoop-project-dist
Gautham B A [Sun, 8 May 2022 18:17:13 +0000 (23:47 +0530)] 
HDFS-16572. Fix typo in readme of hadoop-project-dist

13 days agoHADOOP-17479. Fix the examples of hadoop config prefix (#4197)
Ashutosh Gupta [Sat, 7 May 2022 23:09:24 +0000 (00:09 +0100)] 
HADOOP-17479. Fix the examples of hadoop config prefix (#4197)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
13 days agoHDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194)
Ashutosh Gupta [Sat, 7 May 2022 23:05:34 +0000 (00:05 +0100)] 
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
13 days agoHADOOP-16515. Update the link to compatibility guide (#4226)
Ashutosh Gupta [Sat, 7 May 2022 22:38:32 +0000 (23:38 +0100)] 
HADOOP-16515. Update the link to compatibility guide (#4226)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoHDFS-16520. Improve EC pread: avoid potential reading whole block (#4104)
daimin [Fri, 6 May 2022 17:30:32 +0000 (01:30 +0800)] 
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104)

Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2 weeks agoHDFS-16564. Use uint32_t for hdfs_find (#4245)
Gautham B A [Wed, 4 May 2022 14:49:08 +0000 (20:19 +0530)] 
HDFS-16564. Use uint32_t for hdfs_find (#4245)

* hdfs_find uses u_int32_t type for
  storing the value for the max-depth
  command line argument.
* The type u_int32_t isn't standard,
  isn't available on Windows and thus
  breaks cross-platform compatibility.
  We need to replace this with uint32_t
  which is available on all platforms since
  it's part of the C++ standard.

2 weeks agoHADOOP-18168. Fix S3A ITestMarkerTool use of purged public bucket. (#4140)
Daniel Carl Jones [Tue, 3 May 2022 13:26:52 +0000 (14:26 +0100)] 
HADOOP-18168. Fix S3A ITestMarkerTool use of purged public bucket. (#4140)

This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones

Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257

2 weeks agoRevert "HADOOP-18168. . (#4140)"
Steve Loughran [Tue, 3 May 2022 13:27:52 +0000 (14:27 +0100)] 
Revert "HADOOP-18168. . (#4140)"

This reverts commit 6ab7b72cd61a16f7bcf0d7cae15d1d13b04e1eb1.

2 weeks agoHADOOP-18168. . (#4140)
Daniel Carl Jones [Tue, 3 May 2022 13:26:52 +0000 (14:26 +0100)] 
HADOOP-18168. . (#4140)

This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated  test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones

2 weeks agoHADOOP-18198. add -mvnargs option to create-release command line (#4239)
Steve Loughran [Wed, 27 Apr 2022 16:31:17 +0000 (17:31 +0100)] 
HADOOP-18198. add -mvnargs option to create-release command line (#4239)

This allows for builds to be run with options like
--mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false"

Contributed by Ayush Saxena.

Change-Id: I396e82d0915d679657d063a948f865041bcdde29

2 weeks agoHDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240)
Murali Krishna [Tue, 3 May 2022 12:12:45 +0000 (17:42 +0530)] 
HDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240)

2 weeks agoHDFS-16521. DFS API to retrieve slow datanodes (#4107)
Viraj Jasani [Mon, 2 May 2022 21:05:40 +0000 (14:05 -0700)] 
HDFS-16521. DFS API to retrieve slow datanodes (#4107)

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2 weeks agoYARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter...
jteagles [Mon, 2 May 2022 18:14:26 +0000 (13:14 -0500)] 
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242)

Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoYARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer mainta...
Ashutosh Gupta [Mon, 2 May 2022 18:08:30 +0000 (19:08 +0100)] 
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoHADOOP-18219. Fix shaded client test failure (#4254)
Akira Ajisaka [Mon, 2 May 2022 11:25:09 +0000 (20:25 +0900)] 
HADOOP-18219. Fix shaded client test failure (#4254)

This patch MUST be cherrypicked into all branches with
HADOOP-15983 Use jersey-json that is built to use jackson2

Contributed by Akira Ajisaka

2 weeks agoHDFS-16256. Minor fix in HDFS Fedbalance document (#4192)
Ashutosh Gupta [Mon, 2 May 2022 00:08:12 +0000 (01:08 +0100)] 
HDFS-16256. Minor fix in HDFS Fedbalance document (#4192)

3 weeks agoHDFS-16468. Define ssize_t for Windows (#4228)
Gautham B A [Fri, 29 Apr 2022 16:55:09 +0000 (22:25 +0530)] 
HDFS-16468. Define ssize_t for Windows (#4228)

* Some C/C++ files use ssize_t data type.
  This isn't available for Windows and we
  need to define an alias for this and set it
  to an appropriate type to make it cross
  platform compatible.

3 weeks agoHDFS-16553. Fix checkstyle for the length of BlockManager construction method over...
smarthan [Fri, 29 Apr 2022 14:59:23 +0000 (22:59 +0800)] 
HDFS-16553. Fix checkstyle for the length of BlockManager construction method over limit. (#4211). Contributed by Chengwei Wang.

3 weeks agoHDFS-16528. Reconfigure slow peer enable for Namenode (#4186)
Viraj Jasani [Fri, 29 Apr 2022 09:00:17 +0000 (02:00 -0700)] 
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186)

3 weeks agoHDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170)
stack [Thu, 28 Apr 2022 17:28:01 +0000 (10:28 -0700)] 
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170)

This reverts the previous commit 4e47eb66d123014b653d2a2c278a35366902d8a0
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.

3 weeks agoRevert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes...
stack [Thu, 28 Apr 2022 17:26:46 +0000 (10:26 -0700)] 
Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)"

Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf90ae64b32dbaf7b42413e18598e434d.

3 weeks agoHDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)
huaxiangsun [Thu, 28 Apr 2022 17:24:37 +0000 (10:24 -0700)] 
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)

When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.

3 weeks agoHADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)
PJ Fanning [Thu, 28 Apr 2022 05:18:19 +0000 (07:18 +0200)] 
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
3 weeks agoHDFS-16539. RBF: Support refreshing/changing router fairness policy controller withou...
Felix Nguyen [Wed, 27 Apr 2022 06:42:30 +0000 (14:42 +0800)] 
HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168)

3 weeks agoHDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (...
smarthan [Wed, 27 Apr 2022 03:44:38 +0000 (11:44 +0800)] 
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang.

3 weeks agoHADOOP-18216. io.file.buffer.size must be positive (#4220)
liever18 [Tue, 26 Apr 2022 18:53:11 +0000 (02:53 +0800)] 
HADOOP-18216. io.file.buffer.size must be positive (#4220)

Documents that the buffer size must be > 0

Contributed by Jingxuan Fu.

3 weeks agoHADOOP-18167. Add metrics to track delegation token secret manager op… (#4092)
hchaverri [Tue, 26 Apr 2022 16:20:11 +0000 (09:20 -0700)] 
HADOOP-18167. Add metrics to track delegation token secret manager op… (#4092)

* HADOOP-18167. Add metrics to track delegation token secret manager operations

3 weeks agoHDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234)
litao [Tue, 26 Apr 2022 03:48:06 +0000 (11:48 +0800)] 
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234)

3 weeks agoYARN-10303. Fixing rest api example, status code error and other typos in ResourceMan...
Ashutosh Gupta [Tue, 26 Apr 2022 01:51:11 +0000 (02:51 +0100)] 
YARN-10303. Fixing rest api example, status code error and other typos in ResourceManagerRest.md (#4221)

3 weeks agoMAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updatin...
Ashutosh Gupta [Mon, 25 Apr 2022 05:29:35 +0000 (06:29 +0100)] 
MAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updating the datatype of appId to "string". (#4223)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
3 weeks agoHDFS-16255. RBF: Fix dead link to fedbalance document (#4193)
Ashutosh Gupta [Mon, 25 Apr 2022 05:11:41 +0000 (06:11 +0100)] 
HDFS-16255. RBF: Fix dead link to fedbalance document (#4193)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
3 weeks agoHADOOP-16202. Enhanced openFile(): hadoop-azure changes. (#2584/4)
Steve Loughran [Sun, 24 Apr 2022 16:30:41 +0000 (17:30 +0100)] 
HADOOP-16202. Enhanced openFile(): hadoop-azure changes. (#2584/4)

Stops the abfs connector warning if openFile().withFileStatus()
is invoked with a FileStatus is not an abfs VersionedFileStatus.

Contributed by Steve Loughran.

Change-Id: I85076b365eb30aaef2ed35139fa8714efd4d048e

3 weeks agoHADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3)
Steve Loughran [Sun, 24 Apr 2022 16:23:19 +0000 (17:23 +0100)] 
HADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3)

S3A input stream support for the few fs.option.openfile settings.
As well as supporting the read policy option and values,
if the file length is declared in fs.option.openfile.length
then no HEAD request will be issued when opening a file.
This can cut a few tens of milliseconds off the operation.

The patch adds a new openfile parameter/FS configuration option
fs.s3a.input.async.drain.threshold (default: 16000).
It declares the number of bytes remaining in the http input stream
above which any operation to read and discard the rest of the stream,
"draining", is executed asynchronously.
This asynchronous draining offers some performance benefit on seek-heavy
file IO.

Contributed by Steve Loughran.

Change-Id: I9b0626bbe635e9fd97ac0f463f5e7167e0111e39

3 weeks agoHADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2)
Steve Loughran [Sun, 24 Apr 2022 16:10:34 +0000 (17:10 +0100)] 
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2)

These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94

3 weeks agoHADOOP-16202. Enhanced openFile(): hadoop-common changes. (#2584/1)
Steve Loughran [Sun, 24 Apr 2022 16:03:59 +0000 (17:03 +0100)] 
HADOOP-16202. Enhanced openFile(): hadoop-common changes. (#2584/1)

This defines standard option and values for the
openFile() builder API for opening a file:

fs.option.openfile.read.policy
 A list of the desired read policy, in preferred order.
 standard values are
 adaptive, default, random, sequential, vector, whole-file

fs.option.openfile.length
 How long the file is.

fs.option.openfile.split.start
 start of a task's split

fs.option.openfile.split.end
 end of a task's split

These can be used by filesystem connectors to optimize their
reading of the source file, including but not limited to
* skipping existence/length probes when opening a file
* choosing a policy for prefetching/caching data

The hadoop shell commands which read files all declare "whole-file"
and "sequential", as appropriate.

Contributed by Steve Loughran.

Change-Id: Ia290f79ea7973ce8713d4f90f1315b24d7a23da1

3 weeks agoRevert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232)
Hui Fei [Sun, 24 Apr 2022 09:18:31 +0000 (17:18 +0800)] 
Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232)

3 weeks agoHDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)
litao [Sun, 24 Apr 2022 06:54:41 +0000 (14:54 +0800)] 
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)

4 weeks agoHDFS-16519. Add throttler to EC reconstruction (#4101)
daimin [Sat, 23 Apr 2022 03:45:22 +0000 (11:45 +0800)] 
HDFS-16519. Add throttler to EC reconstruction (#4101)

Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
4 weeks agoHDFS-16552. Fix NPE for TestBlockManager (#4210)
litao [Sat, 23 Apr 2022 03:14:06 +0000 (11:14 +0800)] 
HDFS-16552. Fix NPE for TestBlockManager (#4210)

4 weeks agoHDFS-16556. Fix typos in distcp (#4217)
GuoPhilipse [Fri, 22 Apr 2022 18:01:20 +0000 (02:01 +0800)] 
HDFS-16556. Fix typos in distcp (#4217)

4 weeks agoHADOOP-17564. Fix typo in UnixShellGuide.html (#4195)
Ashutosh Gupta [Fri, 22 Apr 2022 16:59:41 +0000 (17:59 +0100)] 
HADOOP-17564. Fix typo in UnixShellGuide.html (#4195)

contributed by Ashutosh Gupta

4 weeks agoYARN-11111. Recovery failure when node-label configure-type transit from delegated...
Junfan Zhang [Fri, 22 Apr 2022 02:37:28 +0000 (10:37 +0800)] 
YARN-11111. Recovery failure when node-label configure-type transit from delegated-centralized to centralized (#4200)

4 weeks agoHADOOP-17551. Upgrade maven-site-plugin to 3.11.0 (#4196)
Ashutosh Gupta [Thu, 21 Apr 2022 13:16:12 +0000 (14:16 +0100)] 
HADOOP-17551. Upgrade maven-site-plugin to 3.11.0 (#4196)

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
4 weeks agoHDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold confi...
smarthan [Thu, 21 Apr 2022 04:32:12 +0000 (12:32 +0800)] 
HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. (#4061). Contributed by Chengwei Wang.

4 weeks agoRevert "HDFS-16531. Avoid setReplication writing an edit record if old replication...
S O'Donnell [Wed, 20 Apr 2022 19:34:43 +0000 (20:34 +0100)] 
Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell."

This reverts commit dbeeee03639f41a022dd07d5fc04e3aa65a94b5f.

4 weeks agoHDFS-16526. Addendum Add metrics for slow DataNode (#4191)
Renukaprasad C [Wed, 20 Apr 2022 13:27:43 +0000 (18:57 +0530)] 
HDFS-16526. Addendum Add metrics for slow DataNode (#4191)

4 weeks agoHDFS-16544. EC decoding failed due to invalid buffer (#4179)
qinyuren [Wed, 20 Apr 2022 06:04:27 +0000 (14:04 +0800)] 
HDFS-16544. EC decoding failed due to invalid buffer (#4179)

4 weeks agoHADOOP-18172: Changed scope for isRootInternalDir/getRootFallbackLink for InodeTree...
Xing Lin [Wed, 20 Apr 2022 05:47:02 +0000 (22:47 -0700)] 
HADOOP-18172: Changed scope for isRootInternalDir/getRootFallbackLink for InodeTree (#4106)

* HADOOP-18172: Change scope of InodeTree and its member methods to make them accessible from outside package.

Co-authored-by: Xing Lin <xinglin@linkedin.com>
4 weeks agoHDFS-16035. Remove DummyGroupMapping as it is not longer used anywhere. (#4183)
Ashutosh Gupta [Tue, 19 Apr 2022 05:35:23 +0000 (06:35 +0100)] 
HDFS-16035. Remove DummyGroupMapping as it is not longer used anywhere. (#4183)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
4 weeks agoHDFS-16538. EC decoding failed due to not enough valid inputs (#4167)
qinyuren [Tue, 19 Apr 2022 04:37:28 +0000 (12:37 +0800)] 
HDFS-16538. EC decoding failed due to not enough valid inputs (#4167)

Co-authored-by: liubingxing <liubingxing@bigo.sg>
4 weeks agoHADOOP-18202. create-release fails fatal: unsafe repository (#4188)
Steve Loughran [Mon, 18 Apr 2022 18:24:35 +0000 (19:24 +0100)] 
HADOOP-18202. create-release fails fatal: unsafe repository (#4188)

Since April 2022/CVE-2022-24765, git refuses to work in directories
whose owner != the current user, unless explicitly told to trust it.

This patches the create-release script to trust the /build/source
dir mounted from the hosting OS, whose userid is inevitably different
from that of the account in the container running git.

Contributed by: Steve Loughran, Ayush Saxena and the new git error messages

4 weeks agoHDFS-16541. Fix a typo in NameNodeLayoutVersion (#4176)
Happy-shi [Mon, 18 Apr 2022 17:23:57 +0000 (01:23 +0800)] 
HDFS-16541. Fix a typo in NameNodeLayoutVersion (#4176)

4 weeks agoHDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819)
jianghuazhu [Mon, 18 Apr 2022 03:21:51 +0000 (11:21 +0800)] 
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819)

Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
4 weeks agoHDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158)
Quanlong Huang [Mon, 18 Apr 2022 02:32:29 +0000 (10:32 +0800)] 
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158)

Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
4 weeks agoHDFS-16531. Avoid setReplication writing an edit record if old replication equals...
Stephen O'Donnell [Sun, 17 Apr 2022 12:05:11 +0000 (13:05 +0100)] 
HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell.

4 weeks agoHDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141...
He Xiaoqiao [Sun, 17 Apr 2022 11:21:27 +0000 (19:21 +0800)] 
HDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141) Contributed by limingxiang.

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
5 weeks agoHDFS-16526. Add metrics for slow DataNode (#4162)
Renukaprasad C [Fri, 15 Apr 2022 16:07:05 +0000 (21:37 +0530)] 
HDFS-16526. Add metrics for slow DataNode (#4162)

5 weeks agoHADOOP-18196. Remove replace-guava from replacer plugin (#4152)
Viraj Jasani [Fri, 15 Apr 2022 14:24:02 +0000 (19:54 +0530)] 
HADOOP-18196. Remove replace-guava from replacer plugin (#4152)

5 weeks agoYARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist...
章锡平 [Fri, 15 Apr 2022 08:50:49 +0000 (16:50 +0800)] 
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. (#4175)

5 weeks agoHADOOP-18201. Remove endpoint config overrides for ITestS3ARequesterPays (#4169)
Daniel Carl Jones [Thu, 14 Apr 2022 15:21:34 +0000 (16:21 +0100)] 
HADOOP-18201. Remove endpoint config overrides for ITestS3ARequesterPays (#4169)

Contributed by Daniel Carl Jones.

5 weeks agoHDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number...
caozhiqiang [Thu, 14 Apr 2022 09:42:39 +0000 (17:42 +0800)] 
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126)

5 weeks agoHDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by...
daimin [Thu, 14 Apr 2022 03:07:06 +0000 (11:07 +0800)] 
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin.

5 weeks agoHDFS-16479. EC: NameNode should not send a reconstruction work when the source datano...
Takanobu Asanuma [Thu, 14 Apr 2022 02:23:38 +0000 (11:23 +0900)] 
HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138)

5 weeks agoYARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does...
章锡平 [Fri, 8 Apr 2022 07:19:17 +0000 (15:19 +0800)] 
YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1