bigtop.git
21 months agoWebsite updated to 6aa901c9 asf-staging
jenkins [Sat, 26 Sep 2020 13:16:27 +0000 (13:16 +0000)] 
Website updated to 6aa901c9

21 months agoWebsite updated to abe18953
jenkins [Sat, 26 Sep 2020 13:11:16 +0000 (13:11 +0000)] 
Website updated to abe18953

21 months agoWebsite updated to c24b40ab
jenkins [Sat, 26 Sep 2020 12:42:27 +0000 (12:42 +0000)] 
Website updated to c24b40ab

21 months agoWebsite updated to eae17026
jenkins [Fri, 25 Sep 2020 12:48:53 +0000 (12:48 +0000)] 
Website updated to eae17026

21 months agoWebsite updated to d8a05745
jenkins [Wed, 23 Sep 2020 14:29:26 +0000 (14:29 +0000)] 
Website updated to d8a05745

21 months agoWebsite updated to 94be68c8
jenkins [Tue, 22 Sep 2020 17:21:16 +0000 (17:21 +0000)] 
Website updated to 94be68c8

21 months agoWebsite updated to b74baee5
jenkins [Tue, 22 Sep 2020 02:23:07 +0000 (02:23 +0000)] 
Website updated to b74baee5

21 months agoWebsite updated to 76333496
jenkins [Sun, 20 Sep 2020 18:18:53 +0000 (18:18 +0000)] 
Website updated to 76333496

21 months agoWebsite updated to 1c08eb57
jenkins [Sun, 20 Sep 2020 18:18:27 +0000 (18:18 +0000)] 
Website updated to 1c08eb57

21 months agoWebsite updated to 0e3d4599
jenkins [Sun, 20 Sep 2020 13:38:06 +0000 (13:38 +0000)] 
Website updated to 0e3d4599

21 months agoWebsite updated to 1a55291c
jenkins [Sat, 19 Sep 2020 21:00:04 +0000 (21:00 +0000)] 
Website updated to 1a55291c

21 months agoWebsite updated to e4febc22
jenkins [Sat, 19 Sep 2020 20:36:40 +0000 (20:36 +0000)] 
Website updated to e4febc22

21 months agoWebsite updated to 42ed1fa0
Olaf Flebbe [Sat, 19 Sep 2020 19:55:27 +0000 (21:55 +0200)] 
Website updated to 42ed1fa0

21 months agoMerge pull request #671 from guyuqi/BIGTOP-3401
Yuqi Gu [Wed, 16 Sep 2020 01:21:27 +0000 (09:21 +0800)] 
Merge pull request #671 from guyuqi/BIGTOP-3401

BIGTOP-3401: Failed to build Centos-8 docker images for Arm64

21 months agoUpdate the puppetlibs-4.12's modulepath into build.gradle. 671/head
Yuqi Gu [Tue, 15 Sep 2020 10:13:22 +0000 (10:13 +0000)] 
Update the puppetlibs-4.12's modulepath into build.gradle.

Change-Id: I1aff27f304016eda580a51849de55109070b3248
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3405. Make gpg_check in docker-hadoop.sh configurable. (#672)
Kengo Seki [Tue, 15 Sep 2020 08:51:54 +0000 (17:51 +0900)] 
BIGTOP-3405. Make gpg_check in docker-hadoop.sh configurable. (#672)

21 months agoFix docker provisioner on Arm64 Centos-8
Yuqi Gu [Fri, 11 Sep 2020 06:19:20 +0000 (06:19 +0000)] 
Fix docker provisioner on Arm64 Centos-8

Modify Puppet modulepath for puppetlabs-4.12 to
fix "Unknown function: 'any2array'" issue.
Fix 'shellcheck' warnings.

Change-Id: I9a41eabd2e5472def0b0c551ed5744e3819da939
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3401: Failed to build Centos-8 docker images for Arm64
Yuqi Gu [Fri, 11 Sep 2020 06:11:24 +0000 (06:11 +0000)] 
BIGTOP-3401: Failed to build Centos-8 docker images for Arm64

The patch is to fix this issues.

Change-Id: I3e5f1bd864af6ada64c37657af6d8fea15c1df98
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3399. Fix bigtop-groovy build failure. (#670)
Kengo Seki [Wed, 9 Sep 2020 08:15:13 +0000 (17:15 +0900)] 
BIGTOP-3399. Fix bigtop-groovy build failure. (#670)

21 months agoBIGTOP-3398. Fix GPDB puppet manifest to work. (#668)
Kengo Seki [Tue, 8 Sep 2020 11:55:35 +0000 (20:55 +0900)] 
BIGTOP-3398. Fix GPDB puppet manifest to work. (#668)

21 months agoBIGTOP-3396. Building Ambari is failing on Fedora 31 for ARM64. (#667)
Kengo Seki [Sun, 6 Sep 2020 21:05:20 +0000 (06:05 +0900)] 
BIGTOP-3396. Building Ambari is failing on Fedora 31 for ARM64. (#667)

21 months agoBIGTOP-3395. Building Kafka is failing on Fedora 31 for ARM64. (#666)
Kengo Seki [Sun, 6 Sep 2020 21:05:07 +0000 (06:05 +0900)] 
BIGTOP-3395. Building Kafka is failing on Fedora 31 for ARM64. (#666)

21 months agoRevert "BIGTOP-3382. Upgrade Flink to 1.11.1 (#665)"
Kengo Seki [Sun, 6 Sep 2020 15:09:13 +0000 (00:09 +0900)] 
Revert "BIGTOP-3382. Upgrade Flink to 1.11.1 (#665)"

This reverts commit 6f920a14401a65b072a31d5f935b3156840c6c28.

21 months agoBIGTOP-3382. Upgrade Flink to 1.11.1 (#665)
okumurays [Fri, 4 Sep 2020 02:53:40 +0000 (11:53 +0900)] 
BIGTOP-3382. Upgrade Flink to 1.11.1 (#665)

21 months agoBIGTOP-3391: Support GPDB on Perl 5.30+ environment (#664)
Jun He [Thu, 3 Sep 2020 11:34:38 +0000 (19:34 +0800)] 
BIGTOP-3391: Support GPDB on Perl 5.30+ environment (#664)

GPDB is using a special variable $[ in its Perl script,
but it was deprecated from v5.30. So it can't be built
on some distros, e.g., Fedora 31 and Ubuntu 20.04.

This patch fixes indexs used in array and substr functions.

Change-Id: I92c793243c37ce57ffe146b71a5f732488f5e4a9
Signed-off-by: Jun He <jun.he@arm.com>
Co-authored-by: Jun He <jun.he@arm.com>
22 months agoBIGTOP-3392. All packaging jobs are failing on Fedora 31 for aarch64 and ppc64le...
Kengo Seki [Mon, 31 Aug 2020 13:24:37 +0000 (22:24 +0900)] 
BIGTOP-3392. All packaging jobs are failing on Fedora 31 for aarch64 and ppc64le. (#663)

22 months agoBIGTOP-3320: Add Kibana as Bigtop component (#657)
Yuqi Gu [Wed, 26 Aug 2020 08:22:02 +0000 (16:22 +0800)] 
BIGTOP-3320: Add Kibana as Bigtop component (#657)

22 months agoBIGTOP-3390. Building Alluxio fails on Fedora 31. (#662)
Kengo Seki [Tue, 25 Aug 2020 13:17:04 +0000 (22:17 +0900)] 
BIGTOP-3390. Building Alluxio fails on Fedora 31. (#662)

22 months agoBIGTOP-3387: Building QFS fails on CentOS 8 (#658)
Kengo Seki [Tue, 25 Aug 2020 03:28:20 +0000 (12:28 +0900)] 
BIGTOP-3387: Building QFS fails on CentOS 8 (#658)

22 months agoBIGTOP-3389 (addendum). Extend timeout for installing R packages again. (#661)
Kengo Seki [Mon, 24 Aug 2020 12:41:03 +0000 (21:41 +0900)] 
BIGTOP-3389 (addendum). Extend timeout for installing R packages again. (#661)

22 months agoBIGTOP-3389. Extend timeout for installing protobuf and R packages. (#660)
Kengo Seki [Mon, 24 Aug 2020 01:35:19 +0000 (10:35 +0900)] 
BIGTOP-3389. Extend timeout for installing protobuf and R packages. (#660)

22 months agoBIGTOP-3386: Building GPDB fails on CentOS 8 (#659)
Kengo Seki [Sat, 22 Aug 2020 13:31:42 +0000 (22:31 +0900)] 
BIGTOP-3386: Building GPDB fails on CentOS 8 (#659)

23 months agoBIGTOP-3371. Unable to build Apache Ambari on RHEL8. (#656)
Kengo Seki [Thu, 30 Jul 2020 23:17:17 +0000 (08:17 +0900)] 
BIGTOP-3371. Unable to build Apache Ambari on RHEL8. (#656)

23 months agoBIGTOP-3380. Bump Tomcat to 8.5.57. (#655)
Masatake Iwasaki [Mon, 27 Jul 2020 23:14:29 +0000 (08:14 +0900)] 
BIGTOP-3380. Bump Tomcat to 8.5.57. (#655)

23 months agoBIGTOP-3377. Fix deployment failures on Fedora 31 (#652)
Kengo Seki [Sat, 25 Jul 2020 08:34:02 +0000 (17:34 +0900)] 
BIGTOP-3377. Fix deployment failures on Fedora 31 (#652)

23 months agoBIGTOP-3378. Fix Livy deployment failure on Debian and Ubuntu. (#653)
Kengo Seki [Fri, 24 Jul 2020 16:37:42 +0000 (01:37 +0900)] 
BIGTOP-3378. Fix Livy deployment failure on Debian and Ubuntu. (#653)

23 months agoBIGTOP-3379. Fix Spark build failure on CentOS 8. (#654)
Kengo Seki [Fri, 24 Jul 2020 12:59:05 +0000 (21:59 +0900)] 
BIGTOP-3379. Fix Spark build failure on CentOS 8. (#654)

23 months agoBIGTOP-3375. Bump Livy to 0.7.0 and build it with Python3. (#651)
Kengo Seki [Mon, 20 Jul 2020 04:36:09 +0000 (13:36 +0900)] 
BIGTOP-3375. Bump Livy to 0.7.0 and build it with Python3. (#651)

23 months agoBIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner. (...
Kengo Seki [Tue, 7 Jul 2020 14:46:35 +0000 (23:46 +0900)] 
BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner. (#634)

23 months agoBIGTOP-3374. Add dependencies on the utilities provided by Bigtop to bigtop.bom....
Kengo Seki [Tue, 7 Jul 2020 05:55:02 +0000 (14:55 +0900)] 
BIGTOP-3374. Add dependencies on the utilities provided by Bigtop to bigtop.bom. (#650)

23 months agoBIGTOP-3355. [Puppet] Make GPG check for repos a configuration and default to true...
Kengo Seki [Sun, 5 Jul 2020 12:53:46 +0000 (21:53 +0900)] 
BIGTOP-3355. [Puppet] Make GPG check for repos a configuration and default to true (#649)

23 months agoBIGTOP-3319: Add Logstash as Bigtop component (#620)
Yuqi Gu [Sun, 5 Jul 2020 12:33:09 +0000 (20:33 +0800)] 
BIGTOP-3319: Add Logstash as Bigtop component (#620)

23 months agoBIGTOP-3372. Building Spark fails on Debian 9 and Ubuntu 16.04. (#648)
Kengo Seki [Fri, 3 Jul 2020 11:52:37 +0000 (20:52 +0900)] 
BIGTOP-3372. Building Spark fails on Debian 9 and Ubuntu 16.04. (#648)

2 years agoBIGTOP-3364: Leverage pip3/python3 to install setuptools
Yuqi Gu [Thu, 11 Jun 2020 09:01:26 +0000 (09:01 +0000)] 
BIGTOP-3364: Leverage pip3/python3 to install setuptools

setuptools could not be installed by pip/pip2 on Ubuntu-16.04/18.04 and centos-7.
From https://packaging.python.org/tutorials/installing-packages/#requirements-for-installing-packages,
it suggests to leverage python3/pip3 to install setuptools package.

Change-Id: I42b6eb483d3f0e66637e6fc6bbd1fa6eb0a4e60c
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3236: Zeppelin smoke tests implementation (#577)
Yuqi [Fri, 19 Jun 2020 01:30:14 +0000 (09:30 +0800)] 
BIGTOP-3236: Zeppelin smoke tests implementation (#577)

2 years agoBIGTOP-3363. Drop Apex, Hama and Tajo in the 1.5.0 release. (#647)
Masatake Iwasaki [Tue, 16 Jun 2020 15:47:01 +0000 (00:47 +0900)] 
BIGTOP-3363. Drop Apex, Hama and Tajo in the 1.5.0 release. (#647)

* dropped Apex.

* dropped Hama.

* dropped Tajo.

2 years agoBIGTOP-3369: RHEL8 does not have PowerTools repo but CodeReady Builder (#645)
masatana [Tue, 9 Jun 2020 05:19:10 +0000 (14:19 +0900)] 
BIGTOP-3369: RHEL8 does not have PowerTools repo but CodeReady Builder (#645)

2 years agoBIGTOP-3361: No libcrypt.so.1 in Fedora-31
Yuqi Gu [Fri, 29 May 2020 02:38:19 +0000 (02:38 +0000)] 
BIGTOP-3361: No libcrypt.so.1 in Fedora-31

libcrypt.so.1 is not available in Fedora-31 both for Arm64 (both in bigtop/puppet and bigtop/slaves)
and x86_64 (just in bigtop/puppet).
It would cause some dependency issues when some pacakges need to link it such as jffi.
Add libxcrypt-compat in Bigtop_toolchain to fix it.

Change-Id: Ia3b67c17ae710f4e1d06fe3f213c02a83e99b3df
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3362. Unable to install spark-python package on CentOS8 due to lack of python...
masatana [Thu, 4 Jun 2020 15:15:58 +0000 (00:15 +0900)] 
BIGTOP-3362. Unable to install spark-python package on CentOS8 due to lack of python package. (#644)

2 years agoBIGTOP-3359. Fix failure of installing Hadoop due to conflicting .build-id entry...
Masatake Iwasaki [Wed, 27 May 2020 05:15:03 +0000 (14:15 +0900)] 
BIGTOP-3359. Fix failure of installing Hadoop due to conflicting .build-id entry on CentOS 8. (#642)

2 years agoBIGTOP-3356: Add CentOS 8 support to the Docker provisioner (#640)
Kengo Seki [Wed, 27 May 2020 02:03:41 +0000 (11:03 +0900)] 
BIGTOP-3356: Add CentOS 8 support to the Docker provisioner (#640)

2 years agoBIGTOP-3358: get_roles() function should fail if a component is not in roles_map...
masatana [Mon, 25 May 2020 09:51:37 +0000 (18:51 +0900)] 
BIGTOP-3358: get_roles() function should fail if a component is not in roles_map (#641)

2 years agoBIGTOP-3327: Fix Zeppelin build issue after upgrading Spark-2.4.5 (#631)
Yuqi [Fri, 15 May 2020 14:53:07 +0000 (22:53 +0800)] 
BIGTOP-3327: Fix Zeppelin build issue after upgrading Spark-2.4.5 (#631)

When upgrading Spark-2.4.5, Zepplin-0.7.3 is not compatible with Spark-2.4.5.
The patch is to bump Zeppelin to 0.8.2 which is the latest and stable release,
and to fix the zeppelin build issues.

Change-Id: I1e4ff8a18f80bcd99896b68a37a7206002adfd0d
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3348. Fix Hive test failures caused by NoSuchMethodError of ApplicationId...
Masatake Iwasaki [Wed, 13 May 2020 15:21:55 +0000 (00:21 +0900)] 
BIGTOP-3348. Fix Hive test failures caused by NoSuchMethodError of ApplicationId.fromString. (#636)

2 years agoBIGTOP-3353: hive.hwi.war.file configuration is no longer needed (#638)
masatana [Wed, 6 May 2020 18:26:50 +0000 (03:26 +0900)] 
BIGTOP-3353: hive.hwi.war.file configuration is no longer needed (#638)

2 years agoBIGTOP-3347. Fix test failures of Hive ProxyFileSystem against Hadoop 2.10.0. (#633)
Masatake Iwasaki [Tue, 5 May 2020 07:01:58 +0000 (16:01 +0900)] 
BIGTOP-3347. Fix test failures of Hive ProxyFileSystem against Hadoop 2.10.0. (#633)

2 years agoBIGTOP-3352: Fix qfs build error on Ubuntu 18.04
Yuqi Gu [Wed, 29 Apr 2020 08:52:36 +0000 (08:52 +0000)] 
BIGTOP-3352: Fix qfs build error on Ubuntu 18.04

dh-python package missing.

Change-Id: I581885650361a6939dd8fd2c254f53302324ad91
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3351: Fix sqoop build error on Ubuntu 18.04
Yuqi Gu [Wed, 29 Apr 2020 08:08:54 +0000 (08:08 +0000)] 
BIGTOP-3351: Fix sqoop build error on Ubuntu 18.04

rsync package missing

Change-Id: I08fad6f4343abf4fc49d6d04c83b38ff7adb1a88
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3333. Fix Hive build after upgrading Spark to 2.4.5 (#622)
Kengo Seki [Mon, 27 Apr 2020 23:42:43 +0000 (08:42 +0900)] 
BIGTOP-3333. Fix Hive build after upgrading Spark to 2.4.5 (#622)

2 years agoBIGTOP-3337. Remove Node.js from the toolchain. (#626)
Kengo Seki [Thu, 23 Apr 2020 11:30:43 +0000 (20:30 +0900)] 
BIGTOP-3337. Remove Node.js from the toolchain. (#626)

2 years agoBIGTOP-3339. Fix Ambari build failure due to using old Bower. (#628)
Kengo Seki [Tue, 21 Apr 2020 05:15:58 +0000 (14:15 +0900)] 
BIGTOP-3339. Fix Ambari build failure due to using old Bower. (#628)

2 years agoBIGTOP-3341. Support for Openssl 1.1.1. (#630)
Masatake Iwasaki [Mon, 20 Apr 2020 04:20:31 +0000 (13:20 +0900)] 
BIGTOP-3341. Support for Openssl 1.1.1. (#630)

2 years agoBIGTOP-3340. Add Masatake Iwasaki to team list. (#629)
Masatake Iwasaki [Sat, 18 Apr 2020 00:52:04 +0000 (09:52 +0900)] 
BIGTOP-3340. Add Masatake Iwasaki to team list. (#629)

2 years agoBIGTOP-3338. Fix `./gradlew repo` to work with GPDB and QFS on Ubuntu 18.04. (#627)
Kengo Seki [Fri, 10 Apr 2020 18:02:27 +0000 (03:02 +0900)] 
BIGTOP-3338. Fix `./gradlew repo` to work with GPDB and QFS on Ubuntu 18.04. (#627)

2 years agoBIGTOP-3336. Fix Livy build on CentOS and Fedora (#625)
Kengo Seki [Fri, 10 Apr 2020 14:10:00 +0000 (23:10 +0900)] 
BIGTOP-3336. Fix Livy build on CentOS and Fedora (#625)

2 years agoBIGTOP-3335. Fix Mahout build after upgrading Spark to 2.4.5 (#624)
Kengo Seki [Fri, 10 Apr 2020 11:11:52 +0000 (20:11 +0900)] 
BIGTOP-3335. Fix Mahout build after upgrading Spark to 2.4.5 (#624)

2 years agoBIGTOP-3334. Fix Oozie build after upgrading Spark to 2.4.5 (#623)
Kengo Seki [Fri, 10 Apr 2020 06:16:03 +0000 (15:16 +0900)] 
BIGTOP-3334. Fix Oozie build after upgrading Spark to 2.4.5 (#623)

2 years agoBIGTOP-3331: Provisioner Deployment failed on Fedora (#619)
Yuqi [Thu, 2 Apr 2020 12:22:37 +0000 (20:22 +0800)] 
BIGTOP-3331: Provisioner Deployment failed on Fedora (#619)

'yum.conf' doesn't exist in Fedora-29/31 environment.
It causes format error when echo "gpgcheck=0" >> /etc/yum.conf
So the provisioner deployment failed in Fedora-29/31.

Change-Id: I798769bbb38cb2eb6fa9d178c2569cb63c22876c
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3332: Add new provisioner yaml configuration for Fedora 31
Yuqi Gu [Mon, 30 Mar 2020 07:10:12 +0000 (07:10 +0000)] 
BIGTOP-3332: Add new provisioner yaml configuration for Fedora 31

Change-Id: I9d0fe4b890906cd9e24f6010543408881f22e5aa
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3330: avoid duplicate jars between oozie and oozie-client
Luca Toscano [Sun, 22 Mar 2020 17:25:40 +0000 (18:25 +0100)] 
BIGTOP-3330: avoid duplicate jars between oozie and oozie-client

Add a workaround to the install_oozie.sh script to avoid duplicate
jars between oozie and oozie-client. This issue leads to override
errors when installing the oozie package on Debian (oozie-client
is among its dependencies).

Credis to Olaf Flebbe for the solution.

2 years agoBIGTOP-3301. Add smoke-tests for hadoop-kms. (#617)
Masatake Iwasaki [Wed, 18 Mar 2020 18:36:21 +0000 (03:36 +0900)] 
BIGTOP-3301. Add smoke-tests for hadoop-kms. (#617)

* BIGTOP-3301. Add smoke-tests for hadoop-kms.

* refactored variable names.

2 years agoBIGTOP-3286: Bump Ignite to 2.7.6
Yuqi Gu [Mon, 9 Mar 2020 03:14:12 +0000 (03:14 +0000)] 
BIGTOP-3286: Bump Ignite to 2.7.6

Bump Ignite to 2.7.6 for release-1.5.0.

Change-Id: Ib03d2103d60fa0d03d549851420f442af8ddecbe
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3300. Add puppet manifests for hadoop-kms. (#615)
Masatake Iwasaki [Sun, 15 Mar 2020 15:17:10 +0000 (00:17 +0900)] 
BIGTOP-3300. Add puppet manifests for hadoop-kms. (#615)

* BIGTOP-3300. Add puppet manifests for hadoop-kms.

* fixed role assingment, kms kerberos configs and bugs in kerberos module.

* tightened permission of kms-env.sh containing keystore password.

2 years agoBIGTOP-3305. Bump Flume to 1.9.0. (#599)
Masatake Iwasaki [Sun, 15 Mar 2020 13:47:29 +0000 (22:47 +0900)] 
BIGTOP-3305. Bump Flume to 1.9.0. (#599)

2 years agoBIGTOP-3325. Spark LICENSE files should be exempted by RAT check (#616)
Evans Ye [Sun, 15 Mar 2020 07:24:13 +0000 (15:24 +0800)] 
BIGTOP-3325. Spark LICENSE files should be exempted by RAT check (#616)

2 years agoMerge pull request #612 from iwasakims/BIGTOP-3322
Olaf Flebbe [Sat, 14 Mar 2020 16:56:34 +0000 (17:56 +0100)] 
Merge pull request #612 from iwasakims/BIGTOP-3322

2 years agoBIGTOP-3234: TEZ smoke tests implementation (#568)
Yuqi [Sat, 14 Mar 2020 15:36:56 +0000 (23:36 +0800)] 
BIGTOP-3234: TEZ smoke tests implementation (#568)

* BIGTOP-3234: TEZ smoke tests implementation

Add smoke tests for Bigtop Tez

Change-Id: I1963609293d6d55a0db50657611b8a17faa5e566
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
* make the version *

Change-Id: I2e9c683c12bc84b80d26c5fef5f4b2f80847611e

* Set proper tez execution runtime in Bigtop-deploy

Change-Id: I8a41eba2c4af202a94dfc5cffead568ea6e057a1

2 years agoBIGTOP-3165. Bump Spark to 2.4.5.
Masayoshi Tsuzuki [Fri, 13 Mar 2020 10:03:08 +0000 (19:03 +0900)] 
BIGTOP-3165. Bump Spark to 2.4.5.

Signed-off-by: Kengo Seki <sekikn@apache.org>
2 years agoBIGTOP-3237: Add smoke tests for YCSB
Yuqi Gu [Mon, 2 Dec 2019 03:07:08 +0000 (03:07 +0000)] 
BIGTOP-3237: Add smoke tests for YCSB

Change-Id: Id6f5cea6c01b8ada33885d403c61eebc5485f73f
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3317: fix broken symlink for hadoop-hdfs.jar
Luca Toscano [Sat, 29 Feb 2020 09:54:17 +0000 (10:54 +0100)] 
BIGTOP-3317: fix broken symlink for hadoop-hdfs.jar

On Debian 9 the hadoop-hdfs.jar is broken, and causes failures like
oozie sharedlib creation on HDFS. This change replaces it with
hadoop-hdfs-client.jar, that seems to work.

Apply the same fix also for rpm packages.

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
2 years agoBIGTOP-3312: Fix Ambari build failure as maven repo only accepts https now
Jun He [Wed, 19 Feb 2020 02:20:39 +0000 (10:20 +0800)] 
BIGTOP-3312: Fix Ambari build failure as maven repo only accepts https now

Maven repo has been upgrade to accpet https access only.
The URLs defined in ambari pom.xml need to be changed to https.

Change-Id: I77ff02631800cd2ba7abfef071e903262039ee17
Signed-off-by: Jun He <jun.he@arm.com>
2 years agoBIGTOP-3322. Lintian Warnings in Hadoop. 612/head
Masatake Iwasaki [Mon, 9 Mar 2020 05:22:25 +0000 (05:22 +0000)] 
BIGTOP-3322. Lintian Warnings in Hadoop.

2 years agoBIGTOP-3293. Add hadoop-kms package.
Masatake Iwasaki [Tue, 28 Jan 2020 15:04:56 +0000 (15:04 +0000)] 
BIGTOP-3293. Add hadoop-kms package.

fixed kms misconfiguration.

added deb resources for hadoop-kms.

fixed invalid path in tomcat-deployment script.

added temporary workaround for init script not working under systemctl redirect.

runuser must be used instead of su to make init script work under systemd control.

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
2 years agoBIGTOP-3316: Fix HBase build failure on Arm platform
Jun He [Mon, 24 Feb 2020 11:04:52 +0000 (19:04 +0800)] 
BIGTOP-3316: Fix HBase build failure on Arm platform

After HBase is bumped to 1.5.0, jruby patch command in build
script doesn't apply due to regex not match.
This patch refines sed command to make it more versatile to
various asciidoctorj-pdf versions.

Change-Id: Ib414ba3e7b66cf08e2080c75e1c83c563fa88224
Signed-off-by: Jun He <jun.he@arm.com>
2 years agoBIGTOP-3306. Bump Giraph to 1.2.0
Kengo Seki [Sat, 15 Feb 2020 05:14:28 +0000 (14:14 +0900)] 
BIGTOP-3306. Bump Giraph to 1.2.0

2 years agoBIGTOP-3318: Use secure repo in bigtop-tests
Jonathan Leitschuh [Tue, 11 Feb 2020 00:05:19 +0000 (19:05 -0500)] 
BIGTOP-3318: Use secure repo in bigtop-tests

This fixes a security vulnerability in this project where the `pom.xml`
files were configuring Maven to resolve dependencies over HTTP instead of
HTTPS.

Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
2 years agoBIGTOP-3308: add support for openssl 1.1.0 on Debian 611/head
Luca Toscano [Sat, 29 Feb 2020 10:03:23 +0000 (11:03 +0100)] 
BIGTOP-3308: add support for openssl 1.1.0 on Debian

In BIGTOP-2932 a patch was backported from Hadoop upstream to better
support openssl 1.1.x. Some time later a new commit was filed to address
some corner cases, but it was never backported to BigTop.

On Debian 9 openssl 1.1.0 is the default, and BigTop builds against it.
When executing the command "hadoop checknative -a" though an error shows up:

openssl: false EVP_CIPHER_CTX_cleanup

The issue is that EVP_CIPHER_CTX_cleanup was removed from openssl 1.1.0, but
OpensslCipher.c kept using it until a proper if guard was added by the
patch provided by this pull request.

In my case this issue caused Spark 2.4's RPC encryption, based on Openssl native
lib, to fail when establishing new connections between workers and AM on Yarn.
I rebuilt the hadoop deb package with the patch and tested in the Debian 9 Docker
image, the checknative util now returns correctly a positive "true" for openssl.
I also tested it with Spark 2.4 on a test Hadoop cluster and it seems working fine.

2 years agoBIGTOP-3314: Fix flink build failure with upgraded hadoop version (#609)
Jun [Fri, 28 Feb 2020 04:03:19 +0000 (12:03 +0800)] 
BIGTOP-3314: Fix flink build failure with upgraded hadoop version (#609)

Flink 1.6.4 failed to build with new hadoop 2.10.0. The root cause
is ApplicationReport newInstance API is changed.
Add a patch to upgrade API call in flink.

Change-Id: I5ae0f0e786a92f805feace67d5fe2facea0d9314
Signed-off-by: Jun He <jun.he@arm.com>
2 years agoBIGTOP-3310: URL of maven and ant package list page is changed
Jun He [Wed, 19 Feb 2020 02:13:00 +0000 (10:13 +0800)] 
BIGTOP-3310: URL of maven and ant package list page is changed

bigtop_toolchain uses packages list page hosted on
https://www.apache.org/dist/ to detect latest version of maven and ant.
This url has been moved to https://downloads.apache.org/ and latest_*_binary
functions need to be updated to reflect this.
To adapt to future URL change,'-L' option is added too.

Change-Id: Ia732be3c79f5d54b3db0eaa010adbfc8ff0d7395
Signed-off-by: Jun He <jun.he@arm.com>
2 years agoBIGTOP-3315. Bump Alluxio to 1.8.2. (#606)
Masatake Iwasaki [Sat, 22 Feb 2020 05:50:09 +0000 (14:50 +0900)] 
BIGTOP-3315. Bump Alluxio to 1.8.2. (#606)

2 years agoBIGTOP-3311: Fix Alluxio build failure with upgraded hadoop version (#602)
Jun [Sat, 22 Feb 2020 05:48:06 +0000 (13:48 +0800)] 
BIGTOP-3311: Fix Alluxio build failure with upgraded hadoop version (#602)

Alluxio build failed after hadoop is bumpped to 2.10.0. The reason is
the version check regex in integration/yarn/pom.xml doesn't cover
version higher than 2.9.x.
Spring release repo is upgraded to https access.

Change-Id: I83e38af640881e8d61213d58a636af7d50393cdb
Signed-off-by: Jun He <jun.he@arm.com>
2 years agoBIGTOP-3285. Bump Kafka to 2.4.0 (#601)
okumurays [Fri, 21 Feb 2020 09:11:23 +0000 (18:11 +0900)] 
BIGTOP-3285. Bump Kafka to 2.4.0 (#601)

2 years agoBIGTOP-3313: Bump solr to 6.6.6 (#604)
Jun [Fri, 21 Feb 2020 05:59:16 +0000 (13:59 +0800)] 
BIGTOP-3313: Bump solr to 6.6.6 (#604)

Bump solr to latest version of 6.x branch.
Also updated maven repo to https access

Change-Id: I7f09d72d51d6c89b172e796189261590b37444c4
Signed-off-by: Jun He <jun.he@arm.com>
2 years agoBIGTOP-3302. init scripts using su do not work on recent systemd. (#600)
Masatake Iwasaki [Fri, 21 Feb 2020 04:13:07 +0000 (13:13 +0900)] 
BIGTOP-3302. init scripts using su do not work on recent systemd. (#600)

2 years agoBIGTOP-3304. Fix build failure of Flume against hive-2.3.6. (#596)
Masatake Iwasaki [Fri, 14 Feb 2020 22:56:32 +0000 (07:56 +0900)] 
BIGTOP-3304. Fix build failure of Flume against hive-2.3.6. (#596)

2 years agoBIGTOP-3298. Bump Phoenix to 4.15.0-HBase-1.5. (#589)
Masatake Iwasaki [Thu, 13 Feb 2020 00:06:07 +0000 (09:06 +0900)] 
BIGTOP-3298. Bump Phoenix to 4.15.0-HBase-1.5. (#589)

2 years agoBIGTOP-3303. Fix build failure of hbase-1.5.0 on Ubuntu. (#593)
Masatake Iwasaki [Wed, 12 Feb 2020 13:52:46 +0000 (22:52 +0900)] 
BIGTOP-3303. Fix build failure of hbase-1.5.0 on Ubuntu. (#593)

2 years agoBIGTOP-3295. Fix build failure of hadoop-2.10.0 on CentOS 8 (#594)
Kengo Seki [Wed, 12 Feb 2020 04:58:18 +0000 (13:58 +0900)] 
BIGTOP-3295. Fix build failure of hadoop-2.10.0 on CentOS 8 (#594)

2 years agoBIGTOP-3281. Bump Tez to 0.9.2. (#591)
Masatake Iwasaki [Fri, 7 Feb 2020 07:28:14 +0000 (16:28 +0900)] 
BIGTOP-3281. Bump Tez to 0.9.2. (#591)

2 years agoBIGTOP-3282. Bump Hive to 2.3.6. (#592)
Masatake Iwasaki [Fri, 7 Feb 2020 04:22:25 +0000 (13:22 +0900)] 
BIGTOP-3282. Bump Hive to 2.3.6. (#592)