bigtop.git
18 months agoAdditional fix of release notes for 1.5.0 RC0 rel/1.5.0 release-1.5.0
Kengo Seki [Sun, 6 Dec 2020 15:12:07 +0000 (00:12 +0900)] 
Additional fix of release notes for 1.5.0 RC0

18 months agoBIGTOP-3462. Deploying Livy fails if the SPARK_HOME directory doesn't exist. (#709)
Kengo Seki [Sun, 6 Dec 2020 00:59:20 +0000 (09:59 +0900)] 
BIGTOP-3462. Deploying Livy fails if the SPARK_HOME directory doesn't exist. (#709)

(cherry picked from commit 77d41573bb4d8c41e5f4b606c905698dcc49ee81)

19 months agoBIGTOP-3460. Extend timeout for Ambari Server setup. (#708)
Kengo Seki [Thu, 3 Dec 2020 13:33:12 +0000 (22:33 +0900)] 
BIGTOP-3460. Extend timeout for Ambari Server setup. (#708)

(cherry picked from commit 87733f52077d54cf0d68502e0d78baab22d2499c)

19 months agoBIGTOP-3458. Fix broken dependency on groovy-all in the test framework. (#706)
Kengo Seki [Wed, 2 Dec 2020 13:52:40 +0000 (22:52 +0900)] 
BIGTOP-3458. Fix broken dependency on groovy-all in the test framework. (#706)

(cherry picked from commit b622fc64d3b396df9678245d3e174364b6d48f27)

19 months agoBIGTOP-3459. Deploying GPDB fails on ARM64 CentOS 7. (#707)
Kengo Seki [Wed, 2 Dec 2020 10:47:43 +0000 (19:47 +0900)] 
BIGTOP-3459. Deploying GPDB fails on ARM64 CentOS 7. (#707)

(cherry picked from commit 67df377ab7c9f708916b2952018eb3acf72e9870)

19 months agoUpdate default values in hieradata for release 1.5.0
Kengo Seki [Tue, 1 Dec 2020 23:11:02 +0000 (08:11 +0900)] 
Update default values in hieradata for release 1.5.0

(cherry picked from commit 9715ca7ab9ae049b6e7b058c972e0ddd8aedf293)

19 months agoBigtop 1.5.0 release
Kengo Seki [Fri, 27 Nov 2020 12:47:49 +0000 (21:47 +0900)] 
Bigtop 1.5.0 release

19 months agoBIGTOP-3455: add hive patch to fix HIVE-17368 (#705)
Luca Toscano [Thu, 26 Nov 2020 12:23:09 +0000 (13:23 +0100)] 
BIGTOP-3455: add hive patch to fix HIVE-17368 (#705)

(cherry picked from commit c6afdccaa2ff5cf6909a7ac89b96842ecd6ec8e0)

19 months agoBIGTOP-3451. Fix smoke test for SparkR. (#702)
Kengo Seki [Mon, 16 Nov 2020 23:34:19 +0000 (08:34 +0900)] 
BIGTOP-3451. Fix smoke test for SparkR. (#702)

(cherry picked from commit 0469fa579c0b23a799e110a18c3fdf0f9e3ca639)

19 months agoBIGTOP-3449. Failed to build Haddop-2.10.1 on Arm64. (#701)
Masatake Iwasaki [Mon, 16 Nov 2020 13:11:57 +0000 (22:11 +0900)] 
BIGTOP-3449. Failed to build Haddop-2.10.1 on Arm64. (#701)

(cherry picked from commit 0b45e14fec8e17af4ae7b2660933783d6b71448c)

19 months agoBIGTOP-3448. Building Alluxio fails on all distros. (#700)
Kengo Seki [Mon, 16 Nov 2020 13:02:08 +0000 (22:02 +0900)] 
BIGTOP-3448. Building Alluxio fails on all distros. (#700)

(cherry picked from commit bae3cd98538493dfa15a54e8c36786be255164d6)

19 months agoBIGTOP-3446. Fix toolchain task failure due to undefined variable in renv.pp. (#698)
Masatake Iwasaki [Fri, 13 Nov 2020 03:11:57 +0000 (12:11 +0900)] 
BIGTOP-3446. Fix toolchain task failure due to undefined variable in renv.pp. (#698)

Megerd, thanks @iwasakims

(cherry picked from commit 26614ccbb10fce25f17a71f152ebb2a00590642d)

19 months agoBIGTOP-3443: Building Spark fails on Ubuntu 16.04 (#697)
Yuqi Gu [Thu, 12 Nov 2020 09:18:57 +0000 (17:18 +0800)] 
BIGTOP-3443: Building Spark fails on Ubuntu 16.04 (#697)

* BIGTOP-3443: Building Spark fails on Ubuntu 16.04

R version in Ubuntu16.04 is 3.2.3.
The packages 'digest' and 'htmltools' are not available (for R version 3.2.3).
There are some other dependency issues in R-3.2 which seems to be deprecated.

Upgrade the R version to 3.4 for ubuntu-16.04.

Change-Id: I1b5b5116ed09c24da701e6247ce5108ad28f1a79
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
* Fix typo and unify code style

Change-Id: If606e03e225077d14177062f630b943bba21397b
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
(cherry picked from commit b4784868fa9dc26f42d0e60271586e448486b754)

19 months agoBIGTOP-3425. Deploying Ambari fails on Fedora 31. (#695)
Kengo Seki [Thu, 12 Nov 2020 00:42:29 +0000 (09:42 +0900)] 
BIGTOP-3425. Deploying Ambari fails on Fedora 31. (#695)

(cherry picked from commit e2160e26ef803cdeeaa1e4a966c4ba4b13c83fd3)

19 months agoBIGTOP-3444. Bump Hadoop to 2.10.1. (#693)
Masatake Iwasaki [Wed, 11 Nov 2020 22:54:35 +0000 (07:54 +0900)] 
BIGTOP-3444. Bump Hadoop to 2.10.1. (#693)

* BIGTOP-3444. Bump Hadoop to 2.10.1.

* addressed API change of azure-storage.

(cherry picked from commit d661a5643471c9fd1057e7979670a08256659a41)

19 months agoBIGTOP-3440: disable flume solr sink to avoid build failures (#694)
Luca Toscano [Mon, 9 Nov 2020 14:02:11 +0000 (15:02 +0100)] 
BIGTOP-3440: disable flume solr sink to avoid build failures (#694)

The solr sink currently depends on kite, that it is a library not
developed anymore. There are currently build failures due to
the ua_parser library, that was available in a Twitter maven repository
not maintained anymore (since it throws HTTP 503s consistently).
Flume upstream moved the dependency to another repository,
https://github.com/apache/flume/commit/9e0bc5bf4865673e6989ba887d27f468991c1e27
that fixes the Twitter problem but that brings in another build failure
for the 'rome' dependency.

This commit removes the solr sink from the build list, resolving the
overall build problem for Flume.

(cherry picked from commit 8e1bf51585ef78a914640d7d3eba367a174d2a76)

19 months agoBIGTOP-3429. QFS smoke test fails on Ubuntu 16.04. (#696)
Kengo Seki [Wed, 11 Nov 2020 05:41:31 +0000 (14:41 +0900)] 
BIGTOP-3429. QFS smoke test fails on Ubuntu 16.04. (#696)

(cherry picked from commit b22191d2253d69202573df6cdec8bbbbd05cd8a1)

19 months agoChanging version to 1.5.0
Kengo Seki [Fri, 6 Nov 2020 08:31:18 +0000 (17:31 +0900)] 
Changing version to 1.5.0

19 months agoPreparing for release 1.5.0
Kengo Seki [Fri, 6 Nov 2020 07:56:42 +0000 (16:56 +0900)] 
Preparing for release 1.5.0

19 months agoBIGTOP-3441. Suppress YCSB smoke test output. (#691)
Kengo Seki [Fri, 6 Nov 2020 08:16:16 +0000 (17:16 +0900)] 
BIGTOP-3441. Suppress YCSB smoke test output. (#691)

19 months agoBIGTOP-3427. Deploying GPDB fails due to the paramiko installation on some distros...
Kengo Seki [Thu, 5 Nov 2020 09:55:42 +0000 (18:55 +0900)] 
BIGTOP-3427. Deploying GPDB fails due to the paramiko installation on some distros (addendum). (#690)

19 months agoBIGTOP-3439. Oozie's smoke test fails on CentOS and Fedora. (#689)
Masatake Iwasaki [Thu, 5 Nov 2020 00:25:19 +0000 (09:25 +0900)] 
BIGTOP-3439. Oozie's smoke test fails on CentOS and Fedora. (#689)

19 months agoBIGTOP-3427. Deploying GPDB fails due to the paramiko installation on some distros...
Kengo Seki [Wed, 4 Nov 2020 23:34:38 +0000 (08:34 +0900)] 
BIGTOP-3427. Deploying GPDB fails due to the paramiko installation on some distros. (#688)

20 months agoBIGTOP-3422. Fix ambari-server setup failure due to PostgreSQL initdb error. (#687)
Kengo Seki [Fri, 30 Oct 2020 05:28:15 +0000 (14:28 +0900)] 
BIGTOP-3422. Fix ambari-server setup failure due to PostgreSQL initdb error. (#687)

20 months agoBIGTOP-3406. Fix Oozie smoke test to work. (#676)
Kengo Seki [Thu, 29 Oct 2020 06:58:03 +0000 (15:58 +0900)] 
BIGTOP-3406. Fix Oozie smoke test to work. (#676)

20 months agoBIGTOP-3437. Building Logstash fails due to amazing_print version mismatch. (#686)
Kengo Seki [Thu, 29 Oct 2020 02:32:30 +0000 (11:32 +0900)] 
BIGTOP-3437. Building Logstash fails due to amazing_print version mismatch. (#686)

20 months agoBIGTOP-3397. Building Spark is failing on CentOS 7 for ARM64 and PPC64. (#685)
Masatake Iwasaki [Wed, 28 Oct 2020 05:44:43 +0000 (14:44 +0900)] 
BIGTOP-3397. Building Spark is failing on CentOS 7 for ARM64 and PPC64. (#685)

20 months agoMerge pull request #678 from guyuqi/BIGTOP-3411
Yuqi Gu [Wed, 7 Oct 2020 04:12:51 +0000 (12:12 +0800)] 
Merge pull request #678 from guyuqi/BIGTOP-3411

BIGTOP-3411: Disable unnecessary scm check in Buildnumber-maven-plugin for Mpack

20 months agoBIGTOP-3431, README.md updated to contain contextual information for how to build...
Mandruff [Fri, 2 Oct 2020 16:35:10 +0000 (12:35 -0400)] 
BIGTOP-3431, README.md updated to contain contextual information for how to build with git.

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
21 months agoBIGTOP-3416. Fix YCSB to work with CentOS 8 and Fedora 31. (#681)
Kengo Seki [Tue, 29 Sep 2020 13:16:07 +0000 (22:16 +0900)] 
BIGTOP-3416. Fix YCSB to work with CentOS 8 and Fedora 31. (#681)

21 months agoBIGTOP-3418. Fix Ambari deployment on CentOS 8, Fedora 31 and Debian 10. (#683)
Kengo Seki [Tue, 29 Sep 2020 11:45:11 +0000 (20:45 +0900)] 
BIGTOP-3418. Fix Ambari deployment on CentOS 8, Fedora 31 and Debian 10. (#683)

21 months agoBIGTOP-3413. QFS doesn't work with Hadoop's TeraGen. (#682)
Kengo Seki [Tue, 29 Sep 2020 06:34:19 +0000 (15:34 +0900)] 
BIGTOP-3413. QFS doesn't work with Hadoop's TeraGen. (#682)

21 months agoBIGTOP-3414. Fix the default value of GPDB configuration for smoke test. (#679)
Kengo Seki [Mon, 28 Sep 2020 11:20:29 +0000 (20:20 +0900)] 
BIGTOP-3414. Fix the default value of GPDB configuration for smoke test. (#679)

21 months agoBIGTOP-3419: fix condition
Olaf Flebbe [Sat, 26 Sep 2020 13:11:49 +0000 (15:11 +0200)] 
BIGTOP-3419: fix condition

21 months agoIntroduce Jenkinsfile.site for publishing to website
Olaf Flebbe [Fri, 25 Sep 2020 20:02:26 +0000 (22:02 +0200)] 
Introduce Jenkinsfile.site for publishing to website

21 months agoBIGTOP-3415. Extend timeout for Spark smoke test. (#680)
Kengo Seki [Fri, 25 Sep 2020 12:47:03 +0000 (21:47 +0900)] 
BIGTOP-3415. Extend timeout for Spark smoke test. (#680)

21 months agoBIGTOP-3411: Disable unnecessary scm check in Buildnumber-maven-plugin for Mpack 678/head
Yuqi Gu [Wed, 23 Sep 2020 03:19:23 +0000 (03:19 +0000)] 
BIGTOP-3411: Disable unnecessary scm check in Buildnumber-maven-plugin for Mpack

Make it possible to use Buildnumber-maven-plugin to generate build number
without unnecessary scm check. And fix 'rat' issues.

Change-Id: I1ebc24ad46f8167e75f6a35bb64df27e8184627b
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3410. Fix bigtop-tomcat build failure due to downloading problem. (#677)
Kengo Seki [Wed, 23 Sep 2020 03:56:52 +0000 (12:56 +0900)] 
BIGTOP-3410. Fix bigtop-tomcat build failure due to downloading problem. (#677)

21 months agoBIGTOP-3394: Building Logstash is failing on CentOS 8, Debian 10 and Ubuntu 18.04...
Yuqi Gu [Tue, 22 Sep 2020 17:19:41 +0000 (01:19 +0800)] 
BIGTOP-3394: Building Logstash is failing on CentOS 8, Debian 10 and Ubuntu 18.04 (#675)

when rvm install ruby-2.4.0 under user: jenkins,
it fails to "try_sudo" to install autolibs dependencies.
Disable unnecessary autolibs installation.

Change-Id: Ib779074d2180d274dd8526166ca1bf4ed4ac3a3e
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoAdd RPM scripts for making Mpack as a standalone component
Yuqi Gu [Tue, 8 Sep 2020 08:45:24 +0000 (08:45 +0000)] 
Add RPM scripts for making Mpack as a standalone component

Change-Id: If996dd00259f1de4dbfc5c9a1876a7d91468729a
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3376: Make Mpack as a standalone component in Bigtop
Yuqi Gu [Mon, 13 Jul 2020 14:47:36 +0000 (14:47 +0000)] 
BIGTOP-3376: Make Mpack as a standalone component in Bigtop

For users/Bigtopers could easily modify Mapck and make contributions to it like bugfix
and adding more services, the task is to decouple Mpack from Ambari and
put Mpack as a standalone component in Bigtop, like bigtop-packages/src/common/bigtop-mpack.

Change-Id: If1613910325763e4ef4f5299f23456c1f2a20d85
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3407. Ensure the procps package is installed in Debian 10. (#673)
Kengo Seki [Sun, 20 Sep 2020 18:17:34 +0000 (03:17 +0900)] 
BIGTOP-3407. Ensure the procps package is installed in Debian 10. (#673)

21 months agoBIGTOP-3408. docker-hadoop.sh generates broken site.yaml from config.yaml containing...
Masatake Iwasaki [Sun, 20 Sep 2020 18:14:51 +0000 (03:14 +0900)] 
BIGTOP-3408. docker-hadoop.sh generates broken site.yaml from config.yaml containing multiple components. (#674)

21 months agoBIGTOP-3409: minimal staging yaml
Olaf Flebbe [Sat, 19 Sep 2020 20:42:34 +0000 (22:42 +0200)] 
BIGTOP-3409: minimal staging yaml

21 months agoMerge pull request #671 from guyuqi/BIGTOP-3401
Yuqi Gu [Wed, 16 Sep 2020 01:21:27 +0000 (09:21 +0800)] 
Merge pull request #671 from guyuqi/BIGTOP-3401

BIGTOP-3401: Failed to build Centos-8 docker images for Arm64

21 months agoUpdate the puppetlibs-4.12's modulepath into build.gradle. 671/head
Yuqi Gu [Tue, 15 Sep 2020 10:13:22 +0000 (10:13 +0000)] 
Update the puppetlibs-4.12's modulepath into build.gradle.

Change-Id: I1aff27f304016eda580a51849de55109070b3248
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3405. Make gpg_check in docker-hadoop.sh configurable. (#672)
Kengo Seki [Tue, 15 Sep 2020 08:51:54 +0000 (17:51 +0900)] 
BIGTOP-3405. Make gpg_check in docker-hadoop.sh configurable. (#672)

21 months agoFix docker provisioner on Arm64 Centos-8
Yuqi Gu [Fri, 11 Sep 2020 06:19:20 +0000 (06:19 +0000)] 
Fix docker provisioner on Arm64 Centos-8

Modify Puppet modulepath for puppetlabs-4.12 to
fix "Unknown function: 'any2array'" issue.
Fix 'shellcheck' warnings.

Change-Id: I9a41eabd2e5472def0b0c551ed5744e3819da939
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3401: Failed to build Centos-8 docker images for Arm64
Yuqi Gu [Fri, 11 Sep 2020 06:11:24 +0000 (06:11 +0000)] 
BIGTOP-3401: Failed to build Centos-8 docker images for Arm64

The patch is to fix this issues.

Change-Id: I3e5f1bd864af6ada64c37657af6d8fea15c1df98
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
21 months agoBIGTOP-3399. Fix bigtop-groovy build failure. (#670)
Kengo Seki [Wed, 9 Sep 2020 08:15:13 +0000 (17:15 +0900)] 
BIGTOP-3399. Fix bigtop-groovy build failure. (#670)

21 months agoBIGTOP-3398. Fix GPDB puppet manifest to work. (#668)
Kengo Seki [Tue, 8 Sep 2020 11:55:35 +0000 (20:55 +0900)] 
BIGTOP-3398. Fix GPDB puppet manifest to work. (#668)

21 months agoBIGTOP-3396. Building Ambari is failing on Fedora 31 for ARM64. (#667)
Kengo Seki [Sun, 6 Sep 2020 21:05:20 +0000 (06:05 +0900)] 
BIGTOP-3396. Building Ambari is failing on Fedora 31 for ARM64. (#667)

21 months agoBIGTOP-3395. Building Kafka is failing on Fedora 31 for ARM64. (#666)
Kengo Seki [Sun, 6 Sep 2020 21:05:07 +0000 (06:05 +0900)] 
BIGTOP-3395. Building Kafka is failing on Fedora 31 for ARM64. (#666)

21 months agoRevert "BIGTOP-3382. Upgrade Flink to 1.11.1 (#665)"
Kengo Seki [Sun, 6 Sep 2020 15:09:13 +0000 (00:09 +0900)] 
Revert "BIGTOP-3382. Upgrade Flink to 1.11.1 (#665)"

This reverts commit 6f920a14401a65b072a31d5f935b3156840c6c28.

22 months agoBIGTOP-3382. Upgrade Flink to 1.11.1 (#665)
okumurays [Fri, 4 Sep 2020 02:53:40 +0000 (11:53 +0900)] 
BIGTOP-3382. Upgrade Flink to 1.11.1 (#665)

22 months agoBIGTOP-3391: Support GPDB on Perl 5.30+ environment (#664)
Jun He [Thu, 3 Sep 2020 11:34:38 +0000 (19:34 +0800)] 
BIGTOP-3391: Support GPDB on Perl 5.30+ environment (#664)

GPDB is using a special variable $[ in its Perl script,
but it was deprecated from v5.30. So it can't be built
on some distros, e.g., Fedora 31 and Ubuntu 20.04.

This patch fixes indexs used in array and substr functions.

Change-Id: I92c793243c37ce57ffe146b71a5f732488f5e4a9
Signed-off-by: Jun He <jun.he@arm.com>
Co-authored-by: Jun He <jun.he@arm.com>
22 months agoBIGTOP-3392. All packaging jobs are failing on Fedora 31 for aarch64 and ppc64le...
Kengo Seki [Mon, 31 Aug 2020 13:24:37 +0000 (22:24 +0900)] 
BIGTOP-3392. All packaging jobs are failing on Fedora 31 for aarch64 and ppc64le. (#663)

22 months agoBIGTOP-3320: Add Kibana as Bigtop component (#657)
Yuqi Gu [Wed, 26 Aug 2020 08:22:02 +0000 (16:22 +0800)] 
BIGTOP-3320: Add Kibana as Bigtop component (#657)

22 months agoBIGTOP-3390. Building Alluxio fails on Fedora 31. (#662)
Kengo Seki [Tue, 25 Aug 2020 13:17:04 +0000 (22:17 +0900)] 
BIGTOP-3390. Building Alluxio fails on Fedora 31. (#662)

22 months agoBIGTOP-3387: Building QFS fails on CentOS 8 (#658)
Kengo Seki [Tue, 25 Aug 2020 03:28:20 +0000 (12:28 +0900)] 
BIGTOP-3387: Building QFS fails on CentOS 8 (#658)

22 months agoBIGTOP-3389 (addendum). Extend timeout for installing R packages again. (#661)
Kengo Seki [Mon, 24 Aug 2020 12:41:03 +0000 (21:41 +0900)] 
BIGTOP-3389 (addendum). Extend timeout for installing R packages again. (#661)

22 months agoBIGTOP-3389. Extend timeout for installing protobuf and R packages. (#660)
Kengo Seki [Mon, 24 Aug 2020 01:35:19 +0000 (10:35 +0900)] 
BIGTOP-3389. Extend timeout for installing protobuf and R packages. (#660)

22 months agoBIGTOP-3386: Building GPDB fails on CentOS 8 (#659)
Kengo Seki [Sat, 22 Aug 2020 13:31:42 +0000 (22:31 +0900)] 
BIGTOP-3386: Building GPDB fails on CentOS 8 (#659)

23 months agoBIGTOP-3371. Unable to build Apache Ambari on RHEL8. (#656)
Kengo Seki [Thu, 30 Jul 2020 23:17:17 +0000 (08:17 +0900)] 
BIGTOP-3371. Unable to build Apache Ambari on RHEL8. (#656)

23 months agoBIGTOP-3380. Bump Tomcat to 8.5.57. (#655)
Masatake Iwasaki [Mon, 27 Jul 2020 23:14:29 +0000 (08:14 +0900)] 
BIGTOP-3380. Bump Tomcat to 8.5.57. (#655)

23 months agoBIGTOP-3377. Fix deployment failures on Fedora 31 (#652)
Kengo Seki [Sat, 25 Jul 2020 08:34:02 +0000 (17:34 +0900)] 
BIGTOP-3377. Fix deployment failures on Fedora 31 (#652)

23 months agoBIGTOP-3378. Fix Livy deployment failure on Debian and Ubuntu. (#653)
Kengo Seki [Fri, 24 Jul 2020 16:37:42 +0000 (01:37 +0900)] 
BIGTOP-3378. Fix Livy deployment failure on Debian and Ubuntu. (#653)

23 months agoBIGTOP-3379. Fix Spark build failure on CentOS 8. (#654)
Kengo Seki [Fri, 24 Jul 2020 12:59:05 +0000 (21:59 +0900)] 
BIGTOP-3379. Fix Spark build failure on CentOS 8. (#654)

23 months agoBIGTOP-3375. Bump Livy to 0.7.0 and build it with Python3. (#651)
Kengo Seki [Mon, 20 Jul 2020 04:36:09 +0000 (13:36 +0900)] 
BIGTOP-3375. Bump Livy to 0.7.0 and build it with Python3. (#651)

23 months agoBIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner. (...
Kengo Seki [Tue, 7 Jul 2020 14:46:35 +0000 (23:46 +0900)] 
BIGTOP-3343. Add Debian 10 and Ubuntu 18.04 support to the Docker provisioner. (#634)

23 months agoBIGTOP-3374. Add dependencies on the utilities provided by Bigtop to bigtop.bom....
Kengo Seki [Tue, 7 Jul 2020 05:55:02 +0000 (14:55 +0900)] 
BIGTOP-3374. Add dependencies on the utilities provided by Bigtop to bigtop.bom. (#650)

2 years agoBIGTOP-3355. [Puppet] Make GPG check for repos a configuration and default to true...
Kengo Seki [Sun, 5 Jul 2020 12:53:46 +0000 (21:53 +0900)] 
BIGTOP-3355. [Puppet] Make GPG check for repos a configuration and default to true (#649)

2 years agoBIGTOP-3319: Add Logstash as Bigtop component (#620)
Yuqi Gu [Sun, 5 Jul 2020 12:33:09 +0000 (20:33 +0800)] 
BIGTOP-3319: Add Logstash as Bigtop component (#620)

2 years agoBIGTOP-3372. Building Spark fails on Debian 9 and Ubuntu 16.04. (#648)
Kengo Seki [Fri, 3 Jul 2020 11:52:37 +0000 (20:52 +0900)] 
BIGTOP-3372. Building Spark fails on Debian 9 and Ubuntu 16.04. (#648)

2 years agoBIGTOP-3364: Leverage pip3/python3 to install setuptools
Yuqi Gu [Thu, 11 Jun 2020 09:01:26 +0000 (09:01 +0000)] 
BIGTOP-3364: Leverage pip3/python3 to install setuptools

setuptools could not be installed by pip/pip2 on Ubuntu-16.04/18.04 and centos-7.
From https://packaging.python.org/tutorials/installing-packages/#requirements-for-installing-packages,
it suggests to leverage python3/pip3 to install setuptools package.

Change-Id: I42b6eb483d3f0e66637e6fc6bbd1fa6eb0a4e60c
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3236: Zeppelin smoke tests implementation (#577)
Yuqi [Fri, 19 Jun 2020 01:30:14 +0000 (09:30 +0800)] 
BIGTOP-3236: Zeppelin smoke tests implementation (#577)

2 years agoBIGTOP-3363. Drop Apex, Hama and Tajo in the 1.5.0 release. (#647)
Masatake Iwasaki [Tue, 16 Jun 2020 15:47:01 +0000 (00:47 +0900)] 
BIGTOP-3363. Drop Apex, Hama and Tajo in the 1.5.0 release. (#647)

* dropped Apex.

* dropped Hama.

* dropped Tajo.

2 years agoBIGTOP-3369: RHEL8 does not have PowerTools repo but CodeReady Builder (#645)
masatana [Tue, 9 Jun 2020 05:19:10 +0000 (14:19 +0900)] 
BIGTOP-3369: RHEL8 does not have PowerTools repo but CodeReady Builder (#645)

2 years agoBIGTOP-3361: No libcrypt.so.1 in Fedora-31
Yuqi Gu [Fri, 29 May 2020 02:38:19 +0000 (02:38 +0000)] 
BIGTOP-3361: No libcrypt.so.1 in Fedora-31

libcrypt.so.1 is not available in Fedora-31 both for Arm64 (both in bigtop/puppet and bigtop/slaves)
and x86_64 (just in bigtop/puppet).
It would cause some dependency issues when some pacakges need to link it such as jffi.
Add libxcrypt-compat in Bigtop_toolchain to fix it.

Change-Id: Ia3b67c17ae710f4e1d06fe3f213c02a83e99b3df
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3362. Unable to install spark-python package on CentOS8 due to lack of python...
masatana [Thu, 4 Jun 2020 15:15:58 +0000 (00:15 +0900)] 
BIGTOP-3362. Unable to install spark-python package on CentOS8 due to lack of python package. (#644)

2 years agoBIGTOP-3359. Fix failure of installing Hadoop due to conflicting .build-id entry...
Masatake Iwasaki [Wed, 27 May 2020 05:15:03 +0000 (14:15 +0900)] 
BIGTOP-3359. Fix failure of installing Hadoop due to conflicting .build-id entry on CentOS 8. (#642)

2 years agoBIGTOP-3356: Add CentOS 8 support to the Docker provisioner (#640)
Kengo Seki [Wed, 27 May 2020 02:03:41 +0000 (11:03 +0900)] 
BIGTOP-3356: Add CentOS 8 support to the Docker provisioner (#640)

2 years agoBIGTOP-3358: get_roles() function should fail if a component is not in roles_map...
masatana [Mon, 25 May 2020 09:51:37 +0000 (18:51 +0900)] 
BIGTOP-3358: get_roles() function should fail if a component is not in roles_map (#641)

2 years agoBIGTOP-3327: Fix Zeppelin build issue after upgrading Spark-2.4.5 (#631)
Yuqi [Fri, 15 May 2020 14:53:07 +0000 (22:53 +0800)] 
BIGTOP-3327: Fix Zeppelin build issue after upgrading Spark-2.4.5 (#631)

When upgrading Spark-2.4.5, Zepplin-0.7.3 is not compatible with Spark-2.4.5.
The patch is to bump Zeppelin to 0.8.2 which is the latest and stable release,
and to fix the zeppelin build issues.

Change-Id: I1e4ff8a18f80bcd99896b68a37a7206002adfd0d
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3348. Fix Hive test failures caused by NoSuchMethodError of ApplicationId...
Masatake Iwasaki [Wed, 13 May 2020 15:21:55 +0000 (00:21 +0900)] 
BIGTOP-3348. Fix Hive test failures caused by NoSuchMethodError of ApplicationId.fromString. (#636)

2 years agoBIGTOP-3353: hive.hwi.war.file configuration is no longer needed (#638)
masatana [Wed, 6 May 2020 18:26:50 +0000 (03:26 +0900)] 
BIGTOP-3353: hive.hwi.war.file configuration is no longer needed (#638)

2 years agoBIGTOP-3347. Fix test failures of Hive ProxyFileSystem against Hadoop 2.10.0. (#633)
Masatake Iwasaki [Tue, 5 May 2020 07:01:58 +0000 (16:01 +0900)] 
BIGTOP-3347. Fix test failures of Hive ProxyFileSystem against Hadoop 2.10.0. (#633)

2 years agoBIGTOP-3352: Fix qfs build error on Ubuntu 18.04
Yuqi Gu [Wed, 29 Apr 2020 08:52:36 +0000 (08:52 +0000)] 
BIGTOP-3352: Fix qfs build error on Ubuntu 18.04

dh-python package missing.

Change-Id: I581885650361a6939dd8fd2c254f53302324ad91
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3351: Fix sqoop build error on Ubuntu 18.04
Yuqi Gu [Wed, 29 Apr 2020 08:08:54 +0000 (08:08 +0000)] 
BIGTOP-3351: Fix sqoop build error on Ubuntu 18.04

rsync package missing

Change-Id: I08fad6f4343abf4fc49d6d04c83b38ff7adb1a88
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3333. Fix Hive build after upgrading Spark to 2.4.5 (#622)
Kengo Seki [Mon, 27 Apr 2020 23:42:43 +0000 (08:42 +0900)] 
BIGTOP-3333. Fix Hive build after upgrading Spark to 2.4.5 (#622)

2 years agoBIGTOP-3337. Remove Node.js from the toolchain. (#626)
Kengo Seki [Thu, 23 Apr 2020 11:30:43 +0000 (20:30 +0900)] 
BIGTOP-3337. Remove Node.js from the toolchain. (#626)

2 years agoBIGTOP-3339. Fix Ambari build failure due to using old Bower. (#628)
Kengo Seki [Tue, 21 Apr 2020 05:15:58 +0000 (14:15 +0900)] 
BIGTOP-3339. Fix Ambari build failure due to using old Bower. (#628)

2 years agoBIGTOP-3341. Support for Openssl 1.1.1. (#630)
Masatake Iwasaki [Mon, 20 Apr 2020 04:20:31 +0000 (13:20 +0900)] 
BIGTOP-3341. Support for Openssl 1.1.1. (#630)

2 years agoBIGTOP-3340. Add Masatake Iwasaki to team list. (#629)
Masatake Iwasaki [Sat, 18 Apr 2020 00:52:04 +0000 (09:52 +0900)] 
BIGTOP-3340. Add Masatake Iwasaki to team list. (#629)

2 years agoBIGTOP-3338. Fix `./gradlew repo` to work with GPDB and QFS on Ubuntu 18.04. (#627)
Kengo Seki [Fri, 10 Apr 2020 18:02:27 +0000 (03:02 +0900)] 
BIGTOP-3338. Fix `./gradlew repo` to work with GPDB and QFS on Ubuntu 18.04. (#627)

2 years agoBIGTOP-3336. Fix Livy build on CentOS and Fedora (#625)
Kengo Seki [Fri, 10 Apr 2020 14:10:00 +0000 (23:10 +0900)] 
BIGTOP-3336. Fix Livy build on CentOS and Fedora (#625)

2 years agoBIGTOP-3335. Fix Mahout build after upgrading Spark to 2.4.5 (#624)
Kengo Seki [Fri, 10 Apr 2020 11:11:52 +0000 (20:11 +0900)] 
BIGTOP-3335. Fix Mahout build after upgrading Spark to 2.4.5 (#624)

2 years agoBIGTOP-3334. Fix Oozie build after upgrading Spark to 2.4.5 (#623)
Kengo Seki [Fri, 10 Apr 2020 06:16:03 +0000 (15:16 +0900)] 
BIGTOP-3334. Fix Oozie build after upgrading Spark to 2.4.5 (#623)

2 years agoBIGTOP-3331: Provisioner Deployment failed on Fedora (#619)
Yuqi [Thu, 2 Apr 2020 12:22:37 +0000 (20:22 +0800)] 
BIGTOP-3331: Provisioner Deployment failed on Fedora (#619)

'yum.conf' doesn't exist in Fedora-29/31 environment.
It causes format error when echo "gpgcheck=0" >> /etc/yum.conf
So the provisioner deployment failed in Fedora-29/31.

Change-Id: I798769bbb38cb2eb6fa9d178c2569cb63c22876c
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 years agoBIGTOP-3332: Add new provisioner yaml configuration for Fedora 31
Yuqi Gu [Mon, 30 Mar 2020 07:10:12 +0000 (07:10 +0000)] 
BIGTOP-3332: Add new provisioner yaml configuration for Fedora 31

Change-Id: I9d0fe4b890906cd9e24f6010543408881f22e5aa
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>