bigtop.git
3 years agoBIGTOP-3106: Update links in download page branch-1.3 release-1.3.0
Jun He [Tue, 27 Nov 2018 07:34:49 +0000 (07:34 +0000)] 
BIGTOP-3106: Update links in download page

Links in download page should:
1. be links to release artifacts, not links to directories.
2. link to the asc and sha files should refer to archives only for non-current
   releases
3. current release links should point to apache.org/dist

Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3103: Update download page to comply with Apache announcement requirements 407/head 421/head
Jun He [Sun, 25 Nov 2018 02:51:09 +0000 (10:51 +0800)] 
BIGTOP-3103: Update download page to comply with Apache announcement requirements

To comply with announcement requirement, the download page should contain links
to all current and archived releases along with links to KEYS, checksums, and
signatures for all releases.
Current download page only provides link to latest one.

Signed-off-by: Jun He <jun.he@linaro.org>
3 years agoUpdate provisioner docker images to 1.3.0 rel/1.3.0 release-1.3.0-RC2
Jun He [Fri, 5 Oct 2018 01:26:45 +0000 (01:26 +0000)] 
Update provisioner docker images to 1.3.0

Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3089: Update provision config files with changes in 1.3.0 release
Jun He [Wed, 3 Oct 2018 12:52:33 +0000 (20:52 +0800)] 
BIGTOP-3089: Update provision config files with changes in 1.3.0 release

v1.3.0 supports: centos-7, fedora-26, debian-9, ubuntu-16.04, opensuse-42.3
config files in provisioner/docker should be updated to sync with these changes
Now config files contain ARCH variable to automatically load arch-dependent
repo. And docker-hadoop.sh will load correct puppet image by detecting CPU
arch. This makes only one config file is needed for a certain distro acrossing
different architectures.

This closes #402

Change-Id: Ic5d66cb837e78908fcf19c2e55f819db28866a55
Signed-off-by: Jun He <jun.he@linaro.org>
3 years agoBIGTOP-3091: Set Bigtop repo to higher priority
Jun He [Wed, 3 Oct 2018 14:05:24 +0000 (22:05 +0800)] 
BIGTOP-3091: Set Bigtop repo to higher priority

Bigtop repo file in deploy/puppet doesn't have "priority" field.
An example is on Fedora-26, zookeeper is 3.4.9 in distro repo, while
conflicts with 3.4.6 in Bigtop stack. Similiar to BIGTOP-2796.

Change-Id: I2b1231a2cf62a30401784780c92b31ab810fe34b
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3090: provisioner failed on fedora-26 when deploying jdk
Jun He [Wed, 3 Oct 2018 13:56:48 +0000 (21:56 +0800)] 
BIGTOP-3090: provisioner failed on fedora-26 when deploying jdk

The puppet recipe of jdk intends to setup cacerts link with pki/java/cacerts
while jdk is not installed. This security failure will cause later bigtop
components installation failure.

Change-Id: Ib728cdf60db9feaddb3f65643ee2134158947254
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3088: provisioner failed to run puppet deployment on CentOS-7
Jun He [Wed, 3 Oct 2018 12:00:02 +0000 (20:00 +0800)] 
BIGTOP-3088: provisioner failed to run puppet deployment on CentOS-7

Deploy with puppet on centos-7 failed.
The root cause is puppet-stdlib (4.25) in distro is not compatible with
puppet <4. Pin stdlib to 4.12.0 should fix this problem.

Change-Id: I8ec557412b660cc51399da3c81014b47aa366d6a
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBigtop 1.3.0 release release-1.3.0-RC1
Jun He [Sun, 23 Sep 2018 23:25:08 +0000 (23:25 +0000)] 
Bigtop 1.3.0 release

Signed-off-by: Jun He <junhe@apache.org>
3 years agoUpdate bigtop site to 1.3.0
Jun He [Sun, 23 Sep 2018 23:23:12 +0000 (23:23 +0000)] 
Update bigtop site to 1.3.0

Signed-off-by: Jun He <junhe@apache.org>
3 years agoUpdate repo URLs to 1.3.0
Jun He [Sun, 23 Sep 2018 23:19:27 +0000 (23:19 +0000)] 
Update repo URLs to 1.3.0

Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3083: HBase and iginite-hadoop build failed on Fedora-26 due to BIGTOP-3081
Jun He [Thu, 20 Sep 2018 07:49:04 +0000 (07:49 +0000)] 
BIGTOP-3083: HBase and iginite-hadoop build failed on Fedora-26 due to BIGTOP-3081

Signed-off-by: Jun He <junhe@apache.org>
3 years agoAdd missing scala patch for flume
Jun He [Wed, 19 Sep 2018 23:41:52 +0000 (07:41 +0800)] 
Add missing scala patch for flume

The commit is to add patch to fix flume/kafka build issue which is
missed in BIGTOP-3082.

Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3082: Fix build failure with flume-1.8+kafka-0.10.2.2
Jun He [Wed, 19 Sep 2018 17:48:25 +0000 (01:48 +0800)] 
BIGTOP-3082: Fix build failure with flume-1.8+kafka-0.10.2.2

When upgrading flume to 1.8 and kafka to 0.10.2.2, the build of flume
is failed due to API changes and scala upgrade in kafka.
This patch rebased FLUME-2662/FLUME-3026 patches to flume-1.8, and for
the scala conflicts, used Anton's patch which can be found here:
https://lists.apache.org/thread.html/%3CCAJZrK_tN49GM+vCCiB+rjLL==82U=7nswoapwWmyJbVAS5F=QA@mail.gmail.com%3E

This closes #397

Change-Id: Idf8d00c22ef60d82783056fea556e9a6f517054a
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3081: Update HBase/Ignite-hadoop arch info
Jun He [Tue, 18 Sep 2018 06:57:19 +0000 (14:57 +0800)] 
BIGTOP-3081: Update HBase/Ignite-hadoop arch info

Both HBase and ignite-hadoop introduce arch dependency during build.
So the output packages should be arch dependent. The arch info in
package files should be updated to reflect these.

This closes #395

Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3075: Drop pig packaging
Yuqi Gu [Wed, 12 Sep 2018 08:10:47 +0000 (08:10 +0000)] 
BIGTOP-3075: Drop pig packaging

Drop pig packaging as it doesn't work with Hive-2.0+. See BIGTOP-2975

This closes #396

Change-Id: I7e6ca27979864f04e36595afbd5475e628e21b07
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
3 years agoBIGTOP-3074: Drop oozie packaging
Jun He [Wed, 29 Aug 2018 08:12:05 +0000 (08:12 +0000)] 
BIGTOP-3074: Drop oozie packaging

Change-Id: I92cb5256a7f51a04f0d859b6a812bc8fa0dfc669
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3062: Bump flink to 1.4.2
Yuqi Gu [Fri, 10 Aug 2018 09:17:47 +0000 (09:17 +0000)] 
BIGTOP-3062: Bump flink to 1.4.2

Bump flink to 1.4.2 for Bigtop 1.3 release

This closes #380

Change-Id: Icd527338d6663d7baa7e1a3690667f95c1e12320
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3076: QFS build failed on ppc64le
Jun He [Mon, 10 Sep 2018 03:19:14 +0000 (11:19 +0800)] 
BIGTOP-3076: QFS build failed on ppc64le

QFS build failed on ppc64le, this is caused by conflict definition of
gf_cpu_identify.

This closes #392

Change-Id: Ibc52b426a27c6ed27c71a0e8fabeaaa91577d000
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3078: ignite-shmem failed to build on ppc64le
Jun He [Mon, 10 Sep 2018 03:12:29 +0000 (11:12 +0800)] 
BIGTOP-3078: ignite-shmem failed to build on ppc64le

Ignite-shmem build failed on ppc64le. This is caused by ppc64le is
not supported by its config.guess.

This closes #391

Change-Id: Ife8b1afe185e086034c1fd25fbb20906c0dc15be
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-2101: ignite-hadoop contains an arch-dependent shared lib
Jun He [Mon, 13 Nov 2017 10:11:31 +0000 (18:11 +0800)] 
BIGTOP-2101: ignite-hadoop contains an arch-dependent shared lib

ignite-shmem contains x86 binary only. A build step is added to create
arch specific library for non-x86 platforms.

This closes #358

Change-Id: I9f4f03ec61250da21c13a915dc7efd5303e95d5b
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3077: OpenSuse-42.3 packages install error
Jun He [Tue, 4 Sep 2018 03:25:14 +0000 (11:25 +0800)] 
BIGTOP-3077: OpenSuse-42.3 packages install error

Update libapr1 to use distro version. Fixed packages dependency issue
on aarch64.

This closes #390

Change-Id: I043b701b4bd5b89eea7488e21ace939624528310
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3067: Bump Hive to 2.3.3
Jun He [Mon, 27 Aug 2018 03:22:21 +0000 (11:22 +0800)] 
BIGTOP-3067: Bump Hive to 2.3.3

Bump to 2.3.3 for v1.3 release.

This closes #386

Change-Id: I0a2d0ab3d58bcac1aac79db1ebeb1d089f6f435b
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3068: Bump Hama to 0.7.1
Yuqi Gu [Fri, 17 Aug 2018 06:08:48 +0000 (06:08 +0000)] 
BIGTOP-3068: Bump Hama to 0.7.1

Bump Hama to 0.7.1 for bigtop 1.3 release

This closes #388

Change-Id: I557f36500989e70c4120b43706a19459d0cb4993
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3055: Bump GPDB to 5.10.0
Yuqi Gu [Fri, 17 Aug 2018 06:44:20 +0000 (06:44 +0000)] 
BIGTOP-3055: Bump GPDB to 5.10.0

GPDB upstream has rolled out 5.10.0 release. Let's bump to this.

This closes #389

Change-Id: I1bcc96e31c4802d4a5cb87b47218b6cd6b3d4f3e
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3069: Bump crunch to 0.15.0
Jun He [Wed, 15 Aug 2018 07:01:59 +0000 (15:01 +0800)] 
BIGTOP-3069: Bump crunch to 0.15.0

Bump to latest 0.15.0 release.

This closes #385

Change-Id: I76921880550a06efc00d55d1cdcff69817578c8f
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3065: Bump Hadoop to 2.8.4
Jun He [Wed, 15 Aug 2018 07:07:42 +0000 (15:07 +0800)] 
BIGTOP-3065: Bump Hadoop to 2.8.4

Bump to latest release  on 2.8 branch

Change-Id: I58e719ef66174312fd005b0bb0d11711aee69ad0
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3064: Bump HBase to 1.3.2
Yuqi Gu [Fri, 17 Aug 2018 05:12:18 +0000 (05:12 +0000)] 
BIGTOP-3064: Bump HBase to 1.3.2

HBase 1.3.1 is released on Apr 7, 2017,
while has latest release on 1.3.x is 1.3.2, on Mar 19, 2018.
Bump Hbase to 1.3.2 for bigtop 1.3 release.

Change-Id: Iaca7a4918023c62b56fb6e6462cdd6b2c3a8302e
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3066: Bump tez to 0.9.1
Yuqi Gu [Fri, 17 Aug 2018 05:59:02 +0000 (05:59 +0000)] 
BIGTOP-3066: Bump tez to 0.9.1

Bump to 0.9.1 for 1.3 release.

Change-Id: I9ee682bdf17dde6f768d64538776643ae4d4dcbf
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3039. Change git browse link to Github
Evans Ye [Sat, 2 Jun 2018 08:47:46 +0000 (16:47 +0800)] 
BIGTOP-3039. Change git browse link to Github

3 years agoBIGTOP-3059: Bump Ambari to 2.6.1
Jun He [Wed, 15 Aug 2018 07:05:43 +0000 (15:05 +0800)] 
BIGTOP-3059: Bump Ambari to 2.6.1

Bump Ambari to 2.6.1 release.

This closes #379

Change-Id: I94e17d70bd78ec1d2ff59e914894ad8b0b3c5cae
Signed-off-by: Jun He <junhe@apache.org>
3 years agoChanging version to 1.3.0
Jun He [Thu, 16 Aug 2018 02:19:42 +0000 (02:19 +0000)] 
Changing version to 1.3.0

Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3061: Fix ClassNotFound issue for crunch build
Jun He [Wed, 8 Aug 2018 04:51:49 +0000 (12:51 +0800)] 
BIGTOP-3061: Fix ClassNotFound issue for crunch build

Pick patch from upstream.

This closes #378

Change-Id: I3c2a2bf717fa070b61b4c1788a52874361910dbc
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3060: juju bundle refresh (august 2018)
Kevin W Monroe [Sat, 5 May 2018 13:39:31 +0000 (13:39 +0000)] 
BIGTOP-3060: juju bundle refresh (august 2018)

Closes #377

3 years agoBIGTOP-3047: juju: nagios support for zookeeper
Kevin W Monroe [Thu, 2 Aug 2018 18:25:38 +0000 (13:25 -0500)] 
BIGTOP-3047: juju: nagios support for zookeeper

Closes #372

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
3 years agoBIGTOP-3054: Missing Spark archive caused Zeppelin build failure
Jun He [Tue, 24 Jul 2018 06:34:40 +0000 (14:34 +0800)] 
BIGTOP-3054: Missing Spark archive caused Zeppelin build failure

Zeppelin uses Spark-2.2.1 archive defined in bigtop.bom. This archive
is moved from www.apache.org to archive.apache.org.
So update zeppelin build script to point to Apache's archive site.

Change-Id: Ib637c36d3623fd856d50122875a0e65b1cc82723
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 years agoBIGTOP-3053: Tez failed to build due to bower version is deprecated
Jun He [Thu, 19 Jul 2018 02:46:19 +0000 (10:46 +0800)] 
BIGTOP-3053: Tez failed to build due to bower version is deprecated

Tez building failed due to the bower version is deprecated. Tez upstream uses
a workaround to upgrade it to 1.8.4.

This closes #375

Change-Id: Ib8ffa59fb4fdfc23599164fda7cbe8af30c44df7
Signed-off-by: Jun He <junhe@apache.org>
3 years agoBIGTOP-3052. Maven version is fixed in security verification code
Evans Ye [Sun, 8 Jul 2018 08:50:50 +0000 (16:50 +0800)] 
BIGTOP-3052. Maven version is fixed in security verification code

4 years agoBIGTOP-3048: Revert "BIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolc...
Evans Ye [Mon, 2 Jul 2018 17:06:58 +0000 (01:06 +0800)] 
BIGTOP-3048: Revert "BIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolchain"

This reverts commit 6781b266decf905c1229bf36a36af14dc988e31c.

4 years agoBIGTOP-3031: correct script to find lasted mvn version
Artem Eluferev [Mon, 25 Jun 2018 11:43:26 +0000 (14:43 +0300)] 
BIGTOP-3031: correct script to find lasted mvn version

This closes #369

Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3045: Remove insecure maven repository URL from gradle config
Olaf Flebbe [Sat, 9 Jun 2018 17:35:05 +0000 (19:35 +0200)] 
BIGTOP-3045: Remove insecure maven repository URL from gradle config

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-3035: Provisioner failed because init is missed in Debian-9
Jun He [Fri, 8 Jun 2018 06:26:11 +0000 (14:26 +0800)] 
BIGTOP-3035: Provisioner failed because init is missed in Debian-9

In provisioner/docker/docker-compose.yml, /sbin/init is used as entrypoint.
But this cmd is missed on Debian-9, which causes smoke test/provisioner
failure.

This closes #368

Change-Id: I4c3aac74f8299b23966ea8a5c30a1513c734f889
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3042: HDFS TestHDFSQuota doesn't match Hadoop's definition
Jun He [Thu, 7 Jun 2018 01:17:01 +0000 (09:17 +0800)] 
BIGTOP-3042: HDFS TestHDFSQuota doesn't match Hadoop's definition

The smoke test case of HDFS, TestHDFSQuota:testInputValues, asserts fail
on setSpaceQuota to 0, while this doesn't match Hadoop's definition for SpaceQuota.
This produces false failure, and should be removed from testcases.

This closes #366

Change-Id: I5e18a15e9b45a1057174717a37008b3c92bab0e9
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3040: Solr's default configsets doesn't work with smoke test cases
Jun He [Tue, 5 Jun 2018 04:57:59 +0000 (12:57 +0800)] 
BIGTOP-3040: Solr's default configsets doesn't work with smoke test cases

The default configsets used in solr is data_driven_schema_configs, which
doesn't work with solr smoke test, TestIndexing:testSearch. The reason is
in data_driven_schema_configs "name" is treated as StrField, not TextField,
so the words will not be tokenized, and lead to unexpected query result.
sample_techproducts_configs should be used.

This closes #364

Change-Id: I170c9530ed937410dcd99c975276c30777b24045
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-2818: Ambari downloads jdk
Jun He [Thu, 7 Jun 2018 03:17:50 +0000 (11:17 +0800)] 
BIGTOP-2818: Ambari downloads jdk

Ambari server uses jdk package from Hortonworks by default. This package
is not aligned with distro provided, and only x86 is supported.
Update Ambari server deploy script to use distro's JDK.

This closes #367

Change-Id: If062a9a4d689ed31276a944e68254b2fda9a264e
Signed-off-by: Jun He <jun.he@linaro.org>
4 years agoBIGTOP-3044: Fix docker build command for simple sh
Olaf Flebbe [Sat, 9 Jun 2018 17:19:33 +0000 (19:19 +0200)] 
BIGTOP-3044: Fix docker build command for simple sh

4 years agoBIGTOP-3043: Hide workaround for docker defect on aarch64 in build.sh
Olaf Flebbe [Fri, 8 Jun 2018 19:33:19 +0000 (21:33 +0200)] 
BIGTOP-3043: Hide workaround for docker defect on aarch64 in build.sh

4 years agoBIGTOP-3037: Download maven securely
Olaf Flebbe [Thu, 31 May 2018 21:16:09 +0000 (23:16 +0200)] 
BIGTOP-3037: Download maven securely

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-3041: Failed to init hadoop hdfs using init-hdfs.sh
Jun He [Tue, 5 Jun 2018 07:09:48 +0000 (15:09 +0800)] 
BIGTOP-3041: Failed to init hadoop hdfs using init-hdfs.sh

Failed to initialize hdfs using init-hdfs.sh script with 2.8.1.
The root cause is some jars are missed in CLASS_PATH

This closes #365

Change-Id: I6e3e1f42d437097ee122150aa94b41780346fb4f
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3027: Wrong Leveldbjni native binary for aarch64
Jun He [Wed, 9 May 2018 08:01:29 +0000 (16:01 +0800)] 
BIGTOP-3027: Wrong Leveldbjni native binary for aarch64

Leveldbjni used in Hadoop is v1.8, which only includes x86-64 native binaries.
It needs to build from source when creating hadoop packages on aarch64, like
what ppc does.

This closes #360

Change-Id: I4682e7885d9b5ac6d448d954ddffc78256fe0733
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3038: Add rollingupgrade option in hadoop-hdfs-namenode.svc
Sahil Aggarwal [Tue, 5 Jun 2018 13:56:21 +0000 (19:26 +0530)] 
BIGTOP-3038: Add rollingupgrade option in hadoop-hdfs-namenode.svc

Signed-off-by: Evans Ye <evansye@apache.org>
4 years agoBIGTOP-3036: Download gradle and groovy via TLS and do signature checking on ant BIGTOP-3036
Olaf Flebbe [Wed, 30 May 2018 18:47:43 +0000 (20:47 +0200)] 
BIGTOP-3036: Download gradle and groovy via TLS and do signature checking on ant

Thanks to brian m. carlson for reporting

4 years agoBIGTOP-3033: Spark build on OpenSUSE is failed
Jun He [Fri, 18 May 2018 03:08:49 +0000 (11:08 +0800)] 
BIGTOP-3033: Spark build on OpenSUSE is failed

Spark building on OpenSUSE is failed as pdflatex is missing.
Add package to install list.

This closes #359

Change-Id: Ib3914ab05fd37418a5aa798fa37b88cb54e4079e
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3030: Remove phantomjs dependecy in build
Jun He [Thu, 10 May 2018 09:29:32 +0000 (17:29 +0800)] 
BIGTOP-3030: Remove phantomjs dependecy in build

Upstream Ambari depends on phantomjs to build. And at present phantomjs provides
only x86* prebuilt binaries. This leads to build failure on non-x86 platforms.

As phantomjs is used for test, not runtime dep, it can be removed from build to
solve this failure.

This closes #357

Change-Id: I06b4fa48705024bb5de9a9167c337db546abbc36
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3031: Auto find latest maven dist
Jun He [Mon, 14 May 2018 05:56:04 +0000 (13:56 +0800)] 
BIGTOP-3031: Auto find latest maven dist

Similar to ant, maven dist binaries may get updated from time to time on
Apache mirrors. A better way is to auto find latest maven-3 binary release.
Used the same way in BIGTOP-3022.

This closes #356

Change-Id: I0384a656701f62bd7ef7fc40cd6ab85533ba49fc
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3024: Zeppelin build is failed after Spark is bumpped to 2.2.1
Jun He [Thu, 3 May 2018 03:06:14 +0000 (11:06 +0800)] 
BIGTOP-3024: Zeppelin build is failed after Spark is bumpped to 2.2.1

After Spark is bumpped to 2.2.1 by BIGTOP-2991, zeppelin build is failed.
The root cause is zeppelin now try to download spark-2.2.1.tgz and
spark-2.2.1-bin-without-hadoop.tgz from http://d3kbcqa49mib13.cloudfront.net.
This site currently doesn't offer spark 2.2.1 related packages.
So the download link need to be updated to apache site:
https://www.apache.org/dist/spark/

This closes #355

Change-Id: Icea4dc0ac6d65104c7a071e7b044161423dffec5
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3011: zookeeper autopurge options
José Pekkarinen [Fri, 2 Mar 2018 14:20:25 +0000 (16:20 +0200)] 
BIGTOP-3011: zookeeper autopurge options

Closes #346

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
4 years agoBIGTOP-3023: Bump qfs to 2.0.0
Jun He [Tue, 24 Apr 2018 06:11:34 +0000 (14:11 +0800)] 
BIGTOP-3023: Bump qfs to 2.0.0

Prior to v2.0.0, qfs doesn't build with openssl-1.1.
This makes bigtop failed on debian-9.
Bump to v2.0.0 should solve this problem.

Change-Id: I4665e6de206ee928d1d8fbf21694bb92ef11752c
Signed-off-by: Jun He <jun.he@linaro.org>
4 years agoBIGTOP-3021: Remove HUE from Bigtop
Olaf Flebbe [Tue, 10 Apr 2018 18:51:04 +0000 (20:51 +0200)] 
BIGTOP-3021: Remove HUE from Bigtop

4 years agoBIGTOP-2991: Bump up Spark version to 2.2.1
Jun He [Tue, 24 Apr 2018 04:59:30 +0000 (12:59 +0800)] 
BIGTOP-2991: Bump up Spark version to 2.2.1

This closes #353

Change-Id: I5e5b47db667206bd4a810b1015793d69c1223cde
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3022 Auto find of latest Ant
Anton Chevychalov [Tue, 17 Apr 2018 11:20:39 +0000 (14:20 +0300)] 
BIGTOP-3022 Auto find of latest Ant

That patch allow to derminate latest version of Ant
available on apache dist.

There is one restriction. We query main apache mirror instead
of nearest one because we can't be sure it use Apache with same
configuration as main.

Signed-off-by: Olaf Flebbe <oflebbe@apache.org>
4 years agoBIGTOP-3018:detect-javahome script has a typo in variable name
zhtisi [Fri, 30 Mar 2018 04:46:06 +0000 (12:46 +0800)] 
BIGTOP-3018:detect-javahome script has a typo in variable name

https://issues.apache.org/jira/browse/BIGTOP-3018

This closes #350

Signed-off-by: Jun He <jun.he@apache.org>
4 years agoBIGTOP-2723:Fix asciidoctor-maven-plugin for HBase build on AArch64
Yuqi Gu [Thu, 29 Mar 2018 08:10:55 +0000 (08:10 +0000)] 
BIGTOP-2723:Fix asciidoctor-maven-plugin for HBase build on AArch64

This closes #200

Change-Id: Ia7318eb115027ceabeff9952588e5ec364b00144
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <jun.he@apache.org>
4 years agoBIGTOP-3014: juju: use charm-env for shebangs
Kevin W Monroe [Sat, 10 Mar 2018 00:26:52 +0000 (18:26 -0600)] 
BIGTOP-3014: juju: use charm-env for shebangs

Fixes #349

4 years agoBIGTOP-3013: kafka charm: remove bigtop_version override
Kevin W Monroe [Fri, 9 Mar 2018 19:23:25 +0000 (13:23 -0600)] 
BIGTOP-3013: kafka charm: remove bigtop_version override

Fixes #347

4 years agoBIGTOP-3010: juju bundle refresh (feb 2018)
Kevin W Monroe [Wed, 18 Oct 2017 15:59:05 +0000 (15:59 +0000)] 
BIGTOP-3010: juju bundle refresh (feb 2018)

Closes #345

4 years agoBIGTOP-3005: zookeeper charm: support relation changes
José Pekkarinen [Fri, 23 Feb 2018 08:36:14 +0000 (10:36 +0200)] 
BIGTOP-3005: zookeeper charm: support relation changes

Closes #341

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
4 years agoBIGTOP-3007: hive pkg and charm opts for zookeeper
jamesbeedy [Tue, 27 Feb 2018 17:19:06 +0000 (09:19 -0800)] 
BIGTOP-3007: hive pkg and charm opts for zookeeper

Closes #344

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
4 years agoBIGTOP-3003: Ant-1.9.9 tarball is removed from Apache mirrors
Jun He [Mon, 26 Feb 2018 08:06:44 +0000 (16:06 +0800)] 
BIGTOP-3003: Ant-1.9.9 tarball is removed from Apache mirrors

As from 06-Feb, ant-1.9.10 is released and distributed
to Apache mirrors. Ant-1.9.9 is removed and can only
be found at Apache's archive. So bump ant to 1.9.10.

This closes #340

Change-Id: I8175684444b08a1607ee04589f5d252dac5588e2
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3004: Fix HBase build failure on Debian/Fedora
Jun He [Mon, 26 Feb 2018 07:28:42 +0000 (15:28 +0800)] 
BIGTOP-3004: Fix HBase build failure on Debian/Fedora

Debian/Fedora uses bash-4.4.12 which treats the extra
trailing backslash as error. HBase has patch for this
and should be backported.

Change-Id: I11353ec0fcdca6e04373c0dac34e0e23aa757f30
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
4 years agoBIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolchain
Akira Ajisaka [Tue, 27 Feb 2018 10:01:29 +0000 (19:01 +0900)] 
BIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolchain

This closes #343

Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3006: Add Jun He to team list
Jun He [Tue, 27 Feb 2018 07:57:04 +0000 (15:57 +0800)] 
BIGTOP-3006: Add Jun He to team list

Change-Id: I07e193bfad34d6fe393b02dd61270d6e0a6394bc
Signed-off-by: Jun He <junhe@apache.org>
4 years agoBIGTOP-3002: For Kafka, it should be possible to set the broker.id config and log...
Gregory Van Seghbroeck [Fri, 23 Feb 2018 20:07:47 +0000 (21:07 +0100)] 
BIGTOP-3002: For Kafka, it should be possible to set the broker.id config and log.dirs config.

Added 2 new parameters: broker_id, log_dirs. Both do not necessarily
need to be set. It is only when they are set , the parameters are used
in the server.properties file.

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2922: Drop arch specific bigtop-* images
Jun He [Tue, 14 Nov 2017 09:17:13 +0000 (17:17 +0800)] 
BIGTOP-2922: Drop arch specific bigtop-* images

Use unified Dockerfile for puppet/slaves images on different architectures.
Simplified the build process, and add gradle tasks for them.
Example: ./gradlew -POS=debian-9 -Pprefix=1.2.1 bigtop-puppet
         ./gradlew -POS=debian-9 -Pprefix=1.2.1 bigtop-slaves
This will create proper images with same image name as existings on different
HW architectures. This eases efforts for deploy and test.

Change-Id: I183452324faf54808a4e88981d604d8fe119b51a
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
4 years agoBIGTOP-2992: Building Pig with pre-arranged Forrest
Hari Krishna Dara [Sun, 4 Feb 2018 16:16:52 +0000 (21:46 +0530)] 
BIGTOP-2992: Building Pig with pre-arranged Forrest

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2989: Support a local dir with tomcat, when download can't work
Hari Krishna Dara [Sun, 4 Feb 2018 16:08:42 +0000 (21:38 +0530)] 
BIGTOP-2989: Support a local dir with tomcat, when download can't work

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2988: Incorrect use "dnf update" in puppetize.sh
Jun He [Thu, 1 Feb 2018 09:56:33 +0000 (17:56 +0800)] 
BIGTOP-2988: Incorrect use "dnf update" in puppetize.sh

The cmd "dnf -y update" is not correctly used as the
purpose here is to "synchronize the package index
files from their sources".
For this purpose the cmd should be "dnf check-update"

Change-Id: I9ed7b7b631f5fabc00a4ef679d30a68f5c68b539
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2912: Initialize dataset for solr smoke test
Jun He [Thu, 19 Oct 2017 08:45:40 +0000 (16:45 +0800)] 
BIGTOP-2912: Initialize dataset for solr smoke test

Change-Id: Ic55033d04f7433eaa6869b8df9c44170b1b7ec0f
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
4 years agoBIGTOP-2984. Rat check failed after BIGTOP-2698
Evans Ye [Wed, 31 Jan 2018 13:27:53 +0000 (21:27 +0800)] 
BIGTOP-2984. Rat check failed after BIGTOP-2698

4 years agoBIGTOP-2949. Add gradle task which leverage bigtop-ci/build.sh to build packages
Evans Ye [Tue, 28 Nov 2017 16:22:14 +0000 (00:22 +0800)] 
BIGTOP-2949. Add gradle task which leverage bigtop-ci/build.sh to build packages

4 years agoBIGTOP-2990: Upgrade Phoenix version to 4.13.1-HBase-1.3
Youngwoo Kim [Sat, 3 Feb 2018 03:53:09 +0000 (12:53 +0900)] 
BIGTOP-2990: Upgrade Phoenix version to 4.13.1-HBase-1.3

4 years agoBIGTOP-2894 Bump hbase to 1.3.1
Anton Chevychalov [Fri, 8 Sep 2017 16:52:14 +0000 (19:52 +0300)] 
BIGTOP-2894 Bump hbase to 1.3.1

There are a number of issues with licenses check when hbase
building with Hadoop 2.8.x

That patch bump hbase to latest release and add couple patches.

Signed-off-by: Youngwoo Kim <ywkim@apache.org>
4 years agoBIGTOP-2974: Add puppet code to deploy SparkR package
Youngwoo Kim [Wed, 27 Dec 2017 05:24:36 +0000 (14:24 +0900)] 
BIGTOP-2974: Add puppet code to deploy SparkR package

Signed-off-by: Evans Ye <evansye@apache.org>
4 years agoBIGTOP-2980: Hama does not build on DEB type systems
Jun He [Fri, 5 Jan 2018 10:05:53 +0000 (18:05 +0800)] 
BIGTOP-2980: Hama does not build on DEB type systems

Add source/format and modify "rules" file to comply
with Debian format "3.0 (quilt)".

Change-Id: I5aaaf89eb6f0a043bac45f4a4a610cbbdef1b91c
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2981: Packaging SparkR is broken due to BIGTOP-2959 335/head
Youngwoo Kim [Thu, 4 Jan 2018 01:28:41 +0000 (10:28 +0900)] 
BIGTOP-2981: Packaging SparkR is broken due to BIGTOP-2959

4 years agoBIGTOP-2979: JAVA_HOME inconsistent on non-x86 architectures BIGTOP-2979 412/head
Olaf Flebbe [Mon, 1 Jan 2018 18:18:07 +0000 (19:18 +0100)] 
BIGTOP-2979: JAVA_HOME inconsistent on non-x86 architectures

4 years agoBIGTOP-2978: Fix bigtop_toolchain for fedora: no epel-release necessary
Youngwoo Kim [Sat, 30 Dec 2017 06:21:27 +0000 (15:21 +0900)] 
BIGTOP-2978: Fix bigtop_toolchain for fedora: no epel-release necessary

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2976: zookeeper-rest missing dependency on lsb-base
Olaf Flebbe [Thu, 28 Dec 2017 15:07:10 +0000 (16:07 +0100)] 
BIGTOP-2976: zookeeper-rest missing dependency on lsb-base

4 years agoBIGTOP-2973: Hama packaging is broken with Hadoop 2.8+ 332/head
Youngwoo Kim [Wed, 27 Dec 2017 02:44:51 +0000 (11:44 +0900)] 
BIGTOP-2973: Hama packaging is broken with Hadoop 2.8+

4 years agoBIGTOP-2959: Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN
Youngwoo Kim [Tue, 26 Dec 2017 06:17:19 +0000 (15:17 +0900)] 
BIGTOP-2959: Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN

4 years agoBIGTOP-2970: Bump Zeppelin version to 0.7.3
Youngwoo Kim [Tue, 26 Dec 2017 16:45:40 +0000 (01:45 +0900)] 
BIGTOP-2970: Bump Zeppelin version to 0.7.3

4 years agoBIGTOP-2969: Bump up Spark version to 2.2.0
Youngwoo Kim [Tue, 26 Dec 2017 14:08:32 +0000 (23:08 +0900)] 
BIGTOP-2969: Bump up Spark version to 2.2.0

4 years agoBIGTOP-2698: Build/install protobuf from source for all platforms
sahil aggarwal [Tue, 14 Mar 2017 12:08:57 +0000 (17:38 +0530)] 
BIGTOP-2698: Build/install protobuf from source for all platforms
             Includes additional changes by Olaf Flebbe
Signed-off-by: Olaf Flebbe <of@oflebbe.de>
4 years agoBIGTOP-2809: Support R integration for Spark
Youngwoo Kim [Thu, 15 Jun 2017 01:59:21 +0000 (10:59 +0900)] 
BIGTOP-2809: Support R integration for Spark

4 years agoBIGTOP-2968: Bump Hive version to 2.3.2
YoungWoo Kim [Sat, 7 Nov 2015 15:07:16 +0000 (00:07 +0900)] 
BIGTOP-2968: Bump Hive version to 2.3.2

4 years agoBIGTOP-2811: Add R to toolchain for Spark and Zeppelin
Youngwoo Kim [Wed, 13 Dec 2017 08:24:24 +0000 (17:24 +0900)] 
BIGTOP-2811: Add R to toolchain for Spark and Zeppelin

4 years agoBIGTOP-2898 Remove h2 pig classifier in Phoenix
Anton Chevychalov [Mon, 2 Oct 2017 15:55:12 +0000 (18:55 +0300)] 
BIGTOP-2898 Remove h2 pig classifier in Phoenix

There is a regression in Pig relase 0.17
After PIG-4923 they drop h2 classifer.
There is only one jar now and it suports Hadoop 2

That commit also adds BIGTOP_PATCH macro to Phoenix

Signed-off-by: Youngwoo Kim <ywkim@apache.org>
4 years agoBIGTOP-2899 remove h2 pig classifier in Oozie code
Anton Chevychalov [Fri, 29 Sep 2017 16:09:10 +0000 (19:09 +0300)] 
BIGTOP-2899 remove h2 pig classifier in Oozie code

There is a regression in Pig relase 0.17
After PIG-4923 they drop h2 classifer.
There is only one jar now and it suports Hadoop 2

Signed-off-by: Youngwoo Kim <ywkim@apache.org>
4 years agoBIGTOP-2895 Bump Pig to 0.17.0 with Hadoop-2.8
Anton Chevychalov [Thu, 14 Sep 2017 12:01:06 +0000 (15:01 +0300)] 
BIGTOP-2895 Bump Pig to 0.17.0 with Hadoop-2.8

Bump Pig to 0.17.0.
Remove useless patches PIG-4676 and PIG-4728 it is already in 0.17 release.
Add patch PIG-5304 to build pig with Hadoop 2.8

Signed-off-by: Youngwoo Kim <ywkim@apache.org>
4 years agoBIGTOP-2893 Bump Hadoop to 2.8.1
Anton Chevychalov [Wed, 27 Sep 2017 11:48:38 +0000 (14:48 +0300)] 
BIGTOP-2893 Bump Hadoop to 2.8.1

Patches:
=======
HADOOP-12366 was ported to 2.8.1
HADOOP-11628 was removed because it was merged to 2.8

Build time issues fixed:
========================
Fuse_dfs target dir has been changed in 2.8
-------------------------------------------
cp: cannot stat 'hadoop-hdfs-project/hadoop-hdfs/target/native/main/native/fuse-dfs/fuse_dfs': No such file or directory

add package goal to mvn
----------------------
Without that goal tar is incomplete:

for dir in '${HADOOP_DIR}/{lib,}' '${HDFS_DIR}/{lib,}' '${YARN_DIR}/{lib,}' '${MAPREDUCE_DIR}/{lib,}'
'[' -e /code/build/hadoop/rpm/BUILDROOT/hadoop-2.8.1-243.el7.centos.x86_64/usr/lib/hadoop-mapreduce//hadoop-common-2.8.1.jar ']'
exit 1
error: Bad exit status from /var/tmp/rpm-tmp.5Etrah (%install)
    Bad exit status from /var/tmp/rpm-tmp.5Etrah (%install)

Signed-off-by: Youngwoo Kim <ywkim@apache.org>
4 years agoBIGTOP-2957: Upgrade YCSB to 0.12.0 316/head
Youngwoo Kim [Fri, 8 Dec 2017 02:46:29 +0000 (11:46 +0900)] 
BIGTOP-2957: Upgrade YCSB to 0.12.0