bigtop.git
18 hours agoBIGTOP-3136. [Provisioner] Local built repo should have higher priority (#445) master
Evans Ye [Fri, 18 Jan 2019 10:00:15 +0000 (18:00 +0800)] 
BIGTOP-3136. [Provisioner] Local built repo should have higher priority (#445)

18 hours agoBIGTOP-3130. Failed to run Alluxio smoke test (#446)
Evans Ye [Fri, 18 Jan 2019 09:59:47 +0000 (17:59 +0800)] 
BIGTOP-3130. Failed to run Alluxio smoke test (#446)

18 hours agoBIGTOP-3134. [Puppet] An upgrade to deploy Alluxio 1.8.1 (#443)
Evans Ye [Fri, 18 Jan 2019 09:59:36 +0000 (17:59 +0800)] 
BIGTOP-3134. [Puppet] An upgrade to deploy Alluxio 1.8.1 (#443)

18 hours agoBIGTOP-3132. Bump Alluxio up to 1.8.1 (#442)
Evans Ye [Fri, 18 Jan 2019 09:56:28 +0000 (17:56 +0800)] 
BIGTOP-3132. Bump Alluxio up to 1.8.1 (#442)

3 days agoBIGTOP-3130. Failed to run Alluxio smoke test 446/head
Evans Ye [Tue, 15 Jan 2019 08:19:18 +0000 (16:19 +0800)] 
BIGTOP-3130. Failed to run Alluxio smoke test

5 days agoBIGTOP-3133. [Puppet] Add a module to deploy bigtop-utils (#441)
Evans Ye [Sun, 13 Jan 2019 18:25:19 +0000 (02:25 +0800)] 
BIGTOP-3133. [Puppet] Add a module to deploy bigtop-utils (#441)

7 days agoBIGTOP-3131. Apex smoke test requires mvn to compile test jar (#440)
Evans Ye [Fri, 11 Jan 2019 06:37:37 +0000 (14:37 +0800)] 
BIGTOP-3131. Apex smoke test requires mvn to compile test jar (#440)

8 days agoBIGTOP-3129. Failed to run QFS smoke test (#439)
Evans Ye [Thu, 10 Jan 2019 06:29:16 +0000 (14:29 +0800)] 
BIGTOP-3129. Failed to run QFS smoke test (#439)

10 days agoBIGTOP-3125: [provisioner] sbin/init is missed in puppet:opensuse-42.3
Jun He [Fri, 4 Jan 2019 07:10:23 +0000 (15:10 +0800)] 
BIGTOP-3125: [provisioner] sbin/init is missed in puppet:opensuse-42.3

The docker image, bigtop/puppet:opensuse-42.3 doesn't have /sbin/init installed.
This caused docker provisioner failed.

Change-Id: I48cbd25893c4d0008789335261ffecbc34175a13
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
10 days agoMerge pull request #438 from oflebbe/apachemaster
Evans Ye [Tue, 8 Jan 2019 17:48:42 +0000 (01:48 +0800)] 
Merge pull request #438 from oflebbe/apachemaster

BIGTOP-3128: Switch to new gitbox repository

11 days agoBIGTOP-3128: Switch to new gitbox repository 438/head
Olaf Flebbe [Mon, 7 Jan 2019 19:36:36 +0000 (20:36 +0100)] 
BIGTOP-3128: Switch to new gitbox repository

11 days agoBIGTOP-3124. Remove package installation in provisioner/utils/smoke-tests.sh and...
Evans Ye [Fri, 4 Jan 2019 18:35:23 +0000 (02:35 +0800)] 
BIGTOP-3124. Remove package installation in provisioner/utils/smoke-tests.sh and polish the script

11 days agoBIGTOP-3126. [Puppet] Failed to deploy QFS due to Permission denied error at initaili...
Evans Ye [Fri, 4 Jan 2019 18:30:54 +0000 (02:30 +0800)] 
BIGTOP-3126. [Puppet] Failed to deploy QFS due to Permission denied error at initailization

2 weeks agoBIGTOP-3118. [Puppet] Failed to deploy GPDB
Evans Ye [Tue, 25 Dec 2018 17:14:02 +0000 (01:14 +0800)] 
BIGTOP-3118. [Puppet] Failed to deploy GPDB

2 weeks agoBIGTOP-3115. [Puppet] Alluxio master/worker can not startup successfully
Evans Ye [Mon, 24 Dec 2018 05:15:18 +0000 (13:15 +0800)] 
BIGTOP-3115. [Puppet] Alluxio master/worker can not startup successfully

3 weeks agoBIGTOP-3120: Annoying warning message from Ambari snippet
Olaf Flebbe [Thu, 27 Dec 2018 13:36:33 +0000 (14:36 +0100)] 
BIGTOP-3120: Annoying warning message from Ambari snippet

Signed-off-by: Evans Ye <evansye@apache.org>
3 weeks agoBIGTOP-3098: Update maven plugins for project pom
Olaf Flebbe [Tue, 6 Nov 2018 20:07:36 +0000 (21:07 +0100)] 
BIGTOP-3098: Update maven plugins for project pom

3 weeks agoBIGTOP-3117. [Puppet] Can't run spark commands for Spark on Yarn deployment
Evans Ye [Mon, 24 Dec 2018 15:24:15 +0000 (23:24 +0800)] 
BIGTOP-3117. [Puppet] Can't run spark commands for Spark on Yarn deployment

3 weeks agoBIGTOP-3114. [Test] HBase Importtsv smoke tests are failing
Evans Ye [Sun, 23 Dec 2018 02:05:49 +0000 (10:05 +0800)] 
BIGTOP-3114. [Test] HBase Importtsv smoke tests are failing

4 weeks agoBIGTOP-3110. [Puppet] Hive metastore can not startup successfully
Evans Ye [Thu, 13 Dec 2018 10:32:02 +0000 (18:32 +0800)] 
BIGTOP-3110. [Puppet] Hive metastore can not startup successfully

4 weeks agoBIGTOP-3113. Update previous releases links to the archive.apache.org in the download...
Evans Ye [Fri, 14 Dec 2018 09:07:45 +0000 (17:07 +0800)] 
BIGTOP-3113. Update previous releases links to the archive.apache.org in the download page

4 weeks agoBIGTOP-3112. Update release KEYS file from dist.apache.org to apache.org/dist
Evans Ye [Thu, 13 Dec 2018 15:15:25 +0000 (23:15 +0800)] 
BIGTOP-3112. Update release KEYS file from dist.apache.org to apache.org/dist

4 weeks agoBIGTOP-3108. Update version strings to 1.3.0 in the master
Evans Ye [Tue, 11 Dec 2018 07:58:32 +0000 (15:58 +0800)] 
BIGTOP-3108. Update version strings to 1.3.0 in the master

5 weeks agoBIGTOP-2892. The download page should provide instructions to check hashes and signatures
Evans Ye [Tue, 11 Dec 2018 09:16:56 +0000 (17:16 +0800)] 
BIGTOP-2892. The download page should provide instructions to check hashes and signatures

5 weeks agoBIGTOP-3086: Drop datafu packaging
Yuqi Gu [Mon, 8 Oct 2018 06:11:23 +0000 (06:11 +0000)] 
BIGTOP-3086: Drop datafu packaging

Since pig has been removed with BIGTOP-3075,
we should remove datafu as well, since datafu-pig depends on pig.

Change-Id: Ide8d8cb5e8223cf6307a48f7e31a606a02dcefdb
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
6 weeks agoBIGTOP-3107: Patch from upstream correcting the Maven surefire break from newer Java SDK 425/head
Arnaud Launay [Mon, 3 Dec 2018 11:00:48 +0000 (12:00 +0100)] 
BIGTOP-3107: Patch from upstream correcting the Maven surefire break from newer Java SDK

7 weeks agoBIGTOP-3106: Update links in download page
Jun He [Tue, 27 Nov 2018 07:34:49 +0000 (07:34 +0000)] 
BIGTOP-3106: Update links in download page

Links in download page should:
1. be links to release artifacts, not links to directories.
2. link to the asc and sha files should refer to archives only for non-current
   releases
3. current release links should point to apache.org/dist

Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
7 weeks agoBIGTOP-3105. Sync up site changes from branch 1.3 to master
Evans Ye [Mon, 26 Nov 2018 04:05:16 +0000 (12:05 +0800)] 
BIGTOP-3105. Sync up site changes from branch 1.3 to master

7 weeks agoBIGTOP-3102: docker-hadoop.sh WARNING: The scale command is deprecated
Guodong Xu [Fri, 23 Nov 2018 06:34:49 +0000 (14:34 +0800)] 
BIGTOP-3102: docker-hadoop.sh WARNING: The scale command is deprecated

When running docker-hadoop.sh to create a virtual Hadoop cluster on top
of Docker containers, there is a warning message:

WARNING: The scale command is deprecated. Use the up command with the
--scale flag instead.

Checking the shell file, the warning is spit out by docker-compose.
There is a suggestion from docker-compose official website to modify
it using 'up':

https://docs.docker.com/compose/reference/scale/

This patch made this update. And verified using Arm aarch64.

Signed-off-by: Guodong Xu <guodong.xu@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
2 months agoBIGTOP-3099: fix oozie building
Михайлов Виктор Владимирович [Thu, 15 Nov 2018 07:48:04 +0000 (10:48 +0300)] 
BIGTOP-3099: fix oozie building

Signed-off-by: Evans Ye <evansye@apache.org>
3 months agoBIGTOP-3089: Update provision config files with changes in 1.3.0 release
Jun He [Wed, 3 Oct 2018 12:52:33 +0000 (20:52 +0800)] 
BIGTOP-3089: Update provision config files with changes in 1.3.0 release

v1.3.0 supports: centos-7, fedora-26, debian-9, ubuntu-16.04, opensuse-42.3
config files in provisioner/docker should be updated to sync with these changes
Now config files contain ARCH variable to automatically load arch-dependent
repo. And docker-hadoop.sh will load correct puppet image by detecting CPU
arch. This makes only one config file is needed for a certain distro acrossing
different architectures.

This closes #402

Change-Id: Ic5d66cb837e78908fcf19c2e55f819db28866a55
Signed-off-by: Jun He <jun.he@linaro.org>
3 months agoBIGTOP-3091: Set Bigtop repo to higher priority
Jun He [Wed, 3 Oct 2018 14:05:24 +0000 (22:05 +0800)] 
BIGTOP-3091: Set Bigtop repo to higher priority

Bigtop repo file in deploy/puppet doesn't have "priority" field.
An example is on Fedora-26, zookeeper is 3.4.9 in distro repo, while
conflicts with 3.4.6 in Bigtop stack. Similiar to BIGTOP-2796.

Change-Id: I2b1231a2cf62a30401784780c92b31ab810fe34b
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 months agoBIGTOP-3090: provisioner failed on fedora-26 when deploying jdk
Jun He [Wed, 3 Oct 2018 13:56:48 +0000 (21:56 +0800)] 
BIGTOP-3090: provisioner failed on fedora-26 when deploying jdk

The puppet recipe of jdk intends to setup cacerts link with pki/java/cacerts
while jdk is not installed. This security failure will cause later bigtop
components installation failure.

Change-Id: Ib728cdf60db9feaddb3f65643ee2134158947254
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 months agoBIGTOP-3088: provisioner failed to run puppet deployment on CentOS-7
Jun He [Wed, 3 Oct 2018 12:00:02 +0000 (20:00 +0800)] 
BIGTOP-3088: provisioner failed to run puppet deployment on CentOS-7

Deploy with puppet on centos-7 failed.
The root cause is puppet-stdlib (4.25) in distro is not compatible with
puppet <4. Pin stdlib to 4.12.0 should fix this problem.

Change-Id: I8ec557412b660cc51399da3c81014b47aa366d6a
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
3 months agoBIGTOP-3087: Maven Deploy script breaks debian
Epikhin Mikhail [Mon, 1 Oct 2018 15:12:40 +0000 (18:12 +0300)] 
BIGTOP-3087: Maven Deploy script breaks debian

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
3 months agoBIGTOP-3083: HBase and iginite-hadoop build failed on Fedora-26 due to BIGTOP-3081
Jun He [Thu, 20 Sep 2018 07:49:04 +0000 (07:49 +0000)] 
BIGTOP-3083: HBase and iginite-hadoop build failed on Fedora-26 due to BIGTOP-3081

Signed-off-by: Jun He <junhe@apache.org>
3 months agoAdd missing scala patch for flume
Jun He [Wed, 19 Sep 2018 23:41:52 +0000 (07:41 +0800)] 
Add missing scala patch for flume

The commit is to add patch to fix flume/kafka build issue which is
missed in BIGTOP-3082.

Signed-off-by: Jun He <junhe@apache.org>
3 months agoBIGTOP-3082: Fix build failure with flume-1.8+kafka-0.10.2.2
Jun He [Wed, 19 Sep 2018 17:48:25 +0000 (01:48 +0800)] 
BIGTOP-3082: Fix build failure with flume-1.8+kafka-0.10.2.2

When upgrading flume to 1.8 and kafka to 0.10.2.2, the build of flume
is failed due to API changes and scala upgrade in kafka.
This patch rebased FLUME-2662/FLUME-3026 patches to flume-1.8, and for
the scala conflicts, used Anton's patch which can be found here:
https://lists.apache.org/thread.html/%3CCAJZrK_tN49GM+vCCiB+rjLL==82U=7nswoapwWmyJbVAS5F=QA@mail.gmail.com%3E

This closes #397

Change-Id: Idf8d00c22ef60d82783056fea556e9a6f517054a
Signed-off-by: Jun He <junhe@apache.org>
3 months agoBIGTOP-3081: Update HBase/Ignite-hadoop arch info
Jun He [Tue, 18 Sep 2018 06:57:19 +0000 (14:57 +0800)] 
BIGTOP-3081: Update HBase/Ignite-hadoop arch info

Both HBase and ignite-hadoop introduce arch dependency during build.
So the output packages should be arch dependent. The arch info in
package files should be updated to reflect these.

This closes #395

Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3075: Drop pig packaging
Yuqi Gu [Wed, 12 Sep 2018 08:10:47 +0000 (08:10 +0000)] 
BIGTOP-3075: Drop pig packaging

Drop pig packaging as it doesn't work with Hive-2.0+. See BIGTOP-2975

This closes #394

Change-Id: I7e6ca27979864f04e36595afbd5475e628e21b07
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
4 months agoAdd options to deploy Maven artifacts
Mike Grimes [Wed, 12 Sep 2018 04:33:17 +0000 (06:33 +0200)] 
Add options to deploy Maven artifacts

This adds options that can be passed to gradle to trigger specifically enabled builds to deploy maven artifacts, either
to their default remote repository, or to a user defined repo ID or URI:

- doMavenDeploy: flag to trigger deployment of maven artifacts
- mavenDeploySource: flag to add 'source:jars'
- mavenRepoId: repository ID, for getting credentials from Maven settings.xml
- mavenRepoUri: location of repository. Supports S3 URIs (see note)

These can all be passed to gradle using the -P flag.

S3 repo URIs are supported, via the org.springframework.build:aws-maven build extension.

Signed-off-by: Evans Ye <evansye@apache.org>
4 months agoBIGTOP-3062: Bump flink to 1.4.2
Yuqi Gu [Fri, 10 Aug 2018 09:17:47 +0000 (09:17 +0000)] 
BIGTOP-3062: Bump flink to 1.4.2

Bump flink to 1.4.2 for Bigtop 1.3 release

This closes #380

Change-Id: Icd527338d6663d7baa7e1a3690667f95c1e12320
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3077: OpenSuse-42.3 packages install error
Jun He [Tue, 4 Sep 2018 03:25:14 +0000 (11:25 +0800)] 
BIGTOP-3077: OpenSuse-42.3 packages install error

Update libapr1 to use distro version. Fixed packages dependency issue
on aarch64.

This closes #390

Change-Id: I043b701b4bd5b89eea7488e21ace939624528310
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3076: QFS build failed on ppc64le
Jun He [Mon, 10 Sep 2018 03:19:14 +0000 (11:19 +0800)] 
BIGTOP-3076: QFS build failed on ppc64le

QFS build failed on ppc64le, this is caused by conflict definition of
gf_cpu_identify.

This closes #392

Change-Id: Ibc52b426a27c6ed27c71a0e8fabeaaa91577d000
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3078: ignite-shmem failed to build on ppc64le
Jun He [Mon, 10 Sep 2018 03:12:29 +0000 (11:12 +0800)] 
BIGTOP-3078: ignite-shmem failed to build on ppc64le

Ignite-shmem build failed on ppc64le. This is caused by ppc64le is
not supported by its config.guess.

This closes #391

Change-Id: Ife8b1afe185e086034c1fd25fbb20906c0dc15be
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-2101: ignite-hadoop contains an arch-dependent shared lib
Jun He [Mon, 13 Nov 2017 10:11:31 +0000 (18:11 +0800)] 
BIGTOP-2101: ignite-hadoop contains an arch-dependent shared lib

ignite-shmem contains x86 binary only. A build step is added to create
arch specific library for non-x86 platforms.

This closes #358

Change-Id: I9f4f03ec61250da21c13a915dc7efd5303e95d5b
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3067: Bump Hive to 2.3.3
Jun He [Mon, 27 Aug 2018 03:22:21 +0000 (11:22 +0800)] 
BIGTOP-3067: Bump Hive to 2.3.3

Bump to 2.3.3 for v1.3 release.

This closes #386

Change-Id: I0a2d0ab3d58bcac1aac79db1ebeb1d089f6f435b
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3068: Bump Hama to 0.7.1
Yuqi Gu [Fri, 17 Aug 2018 06:08:48 +0000 (06:08 +0000)] 
BIGTOP-3068: Bump Hama to 0.7.1

Bump Hama to 0.7.1 for bigtop 1.3 release

This closes #388

Change-Id: I557f36500989e70c4120b43706a19459d0cb4993
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3055: Bump GPDB to 5.10.0
Yuqi Gu [Fri, 17 Aug 2018 06:44:20 +0000 (06:44 +0000)] 
BIGTOP-3055: Bump GPDB to 5.10.0

GPDB upstream has rolled out 5.10.0 release. Let's bump to this.

This closes #389

Change-Id: I1bcc96e31c4802d4a5cb87b47218b6cd6b3d4f3e
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3069: Bump crunch to 0.15.0
Jun He [Wed, 15 Aug 2018 07:01:59 +0000 (15:01 +0800)] 
BIGTOP-3069: Bump crunch to 0.15.0

Bump to latest 0.15.0 release.

This closes #385

Change-Id: I76921880550a06efc00d55d1cdcff69817578c8f
Signed-off-by: Jun He <junhe@apache.org>
4 months agoBIGTOP-3065: Bump Hadoop to 2.8.4
Jun He [Wed, 15 Aug 2018 07:07:42 +0000 (15:07 +0800)] 
BIGTOP-3065: Bump Hadoop to 2.8.4

Bump to latest release  on 2.8 branch

Change-Id: I58e719ef66174312fd005b0bb0d11711aee69ad0
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
4 months agoBIGTOP-3039. Change git browse link to Github
Evans Ye [Sat, 2 Jun 2018 08:47:46 +0000 (16:47 +0800)] 
BIGTOP-3039. Change git browse link to Github

4 months agoBIGTOP-3064: Bump HBase to 1.3.2
Yuqi Gu [Fri, 17 Aug 2018 05:12:18 +0000 (05:12 +0000)] 
BIGTOP-3064: Bump HBase to 1.3.2

HBase 1.3.1 is released on Apr 7, 2017,
while has latest release on 1.3.x is 1.3.2, on Mar 19, 2018.
Bump Hbase to 1.3.2 for bigtop 1.3 release.

Change-Id: Iaca7a4918023c62b56fb6e6462cdd6b2c3a8302e
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Evans Ye <evansye@apache.org>
4 months agoBIGTOP-3066: Bump tez to 0.9.1
Yuqi Gu [Fri, 17 Aug 2018 05:59:02 +0000 (05:59 +0000)] 
BIGTOP-3066: Bump tez to 0.9.1

Bump to 0.9.1 for 1.3 release.

Change-Id: I9ee682bdf17dde6f768d64538776643ae4d4dcbf
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Evans Ye <evansye@apache.org>
4 months agoBIGTOP-3059: Bump Ambari to 2.6.1
Jun He [Wed, 15 Aug 2018 07:05:43 +0000 (15:05 +0800)] 
BIGTOP-3059: Bump Ambari to 2.6.1

Bump Ambari to 2.6.1 release.

This closes #379

Change-Id: I94e17d70bd78ec1d2ff59e914894ad8b0b3c5cae
Signed-off-by: Jun He <junhe@apache.org>
5 months agoBumping up the version to 1.3.1-SNAPSHOT
Jun He [Thu, 16 Aug 2018 02:10:33 +0000 (02:10 +0000)] 
Bumping up the version to 1.3.1-SNAPSHOT

Signed-off-by: Jun He <junhe@apache.org>
5 months agoPreparing for release 1.3.0
Jun He [Thu, 16 Aug 2018 01:58:50 +0000 (01:58 +0000)] 
Preparing for release 1.3.0

Signed-off-by: Jun He <junhe@apache.org>
5 months agoBIGTOP-3061: Fix ClassNotFound issue for crunch build
Jun He [Wed, 8 Aug 2018 04:51:49 +0000 (12:51 +0800)] 
BIGTOP-3061: Fix ClassNotFound issue for crunch build

Pick patch from upstream.

This closes #378

Change-Id: I3c2a2bf717fa070b61b4c1788a52874361910dbc
Signed-off-by: Jun He <junhe@apache.org>
5 months agoBIGTOP-3060: juju bundle refresh (august 2018)
Kevin W Monroe [Sat, 5 May 2018 13:39:31 +0000 (13:39 +0000)] 
BIGTOP-3060: juju bundle refresh (august 2018)

Closes #377

5 months agoBIGTOP-3047: juju: nagios support for zookeeper
Kevin W Monroe [Thu, 2 Aug 2018 18:25:38 +0000 (13:25 -0500)] 
BIGTOP-3047: juju: nagios support for zookeeper

Closes #372

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
5 months agoBIGTOP-3054: Missing Spark archive caused Zeppelin build failure
Jun He [Tue, 24 Jul 2018 06:34:40 +0000 (14:34 +0800)] 
BIGTOP-3054: Missing Spark archive caused Zeppelin build failure

Zeppelin uses Spark-2.2.1 archive defined in bigtop.bom. This archive
is moved from www.apache.org to archive.apache.org.
So update zeppelin build script to point to Apache's archive site.

Change-Id: Ib637c36d3623fd856d50122875a0e65b1cc82723
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
6 months agoBIGTOP-3053: Tez failed to build due to bower version is deprecated
Jun He [Thu, 19 Jul 2018 02:46:19 +0000 (10:46 +0800)] 
BIGTOP-3053: Tez failed to build due to bower version is deprecated

Tez building failed due to the bower version is deprecated. Tez upstream uses
a workaround to upgrade it to 1.8.4.

This closes #375

Change-Id: Ib8ffa59fb4fdfc23599164fda7cbe8af30c44df7
Signed-off-by: Jun He <junhe@apache.org>
6 months agoBIGTOP-3052. Maven version is fixed in security verification code
Evans Ye [Sun, 8 Jul 2018 08:50:50 +0000 (16:50 +0800)] 
BIGTOP-3052. Maven version is fixed in security verification code

6 months agoBIGTOP-3048: Revert "BIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolc...
Evans Ye [Mon, 2 Jul 2018 17:06:58 +0000 (01:06 +0800)] 
BIGTOP-3048: Revert "BIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolchain"

This reverts commit 6781b266decf905c1229bf36a36af14dc988e31c.

6 months agoBIGTOP-3031: correct script to find lasted mvn version
Artem Eluferev [Mon, 25 Jun 2018 11:43:26 +0000 (14:43 +0300)] 
BIGTOP-3031: correct script to find lasted mvn version

This closes #369

Signed-off-by: Jun He <junhe@apache.org>
7 months agoBIGTOP-3045: Remove insecure maven repository URL from gradle config
Olaf Flebbe [Sat, 9 Jun 2018 17:35:05 +0000 (19:35 +0200)] 
BIGTOP-3045: Remove insecure maven repository URL from gradle config

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
7 months agoBIGTOP-3035: Provisioner failed because init is missed in Debian-9
Jun He [Fri, 8 Jun 2018 06:26:11 +0000 (14:26 +0800)] 
BIGTOP-3035: Provisioner failed because init is missed in Debian-9

In provisioner/docker/docker-compose.yml, /sbin/init is used as entrypoint.
But this cmd is missed on Debian-9, which causes smoke test/provisioner
failure.

This closes #368

Change-Id: I4c3aac74f8299b23966ea8a5c30a1513c734f889
Signed-off-by: Jun He <junhe@apache.org>
7 months agoBIGTOP-3042: HDFS TestHDFSQuota doesn't match Hadoop's definition
Jun He [Thu, 7 Jun 2018 01:17:01 +0000 (09:17 +0800)] 
BIGTOP-3042: HDFS TestHDFSQuota doesn't match Hadoop's definition

The smoke test case of HDFS, TestHDFSQuota:testInputValues, asserts fail
on setSpaceQuota to 0, while this doesn't match Hadoop's definition for SpaceQuota.
This produces false failure, and should be removed from testcases.

This closes #366

Change-Id: I5e18a15e9b45a1057174717a37008b3c92bab0e9
Signed-off-by: Jun He <junhe@apache.org>
7 months agoBIGTOP-3040: Solr's default configsets doesn't work with smoke test cases
Jun He [Tue, 5 Jun 2018 04:57:59 +0000 (12:57 +0800)] 
BIGTOP-3040: Solr's default configsets doesn't work with smoke test cases

The default configsets used in solr is data_driven_schema_configs, which
doesn't work with solr smoke test, TestIndexing:testSearch. The reason is
in data_driven_schema_configs "name" is treated as StrField, not TextField,
so the words will not be tokenized, and lead to unexpected query result.
sample_techproducts_configs should be used.

This closes #364

Change-Id: I170c9530ed937410dcd99c975276c30777b24045
Signed-off-by: Jun He <junhe@apache.org>
7 months agoBIGTOP-2818: Ambari downloads jdk
Jun He [Thu, 7 Jun 2018 03:17:50 +0000 (11:17 +0800)] 
BIGTOP-2818: Ambari downloads jdk

Ambari server uses jdk package from Hortonworks by default. This package
is not aligned with distro provided, and only x86 is supported.
Update Ambari server deploy script to use distro's JDK.

This closes #367

Change-Id: If062a9a4d689ed31276a944e68254b2fda9a264e
Signed-off-by: Jun He <jun.he@linaro.org>
7 months agoBIGTOP-3044: Fix docker build command for simple sh
Olaf Flebbe [Sat, 9 Jun 2018 17:19:33 +0000 (19:19 +0200)] 
BIGTOP-3044: Fix docker build command for simple sh

7 months agoBIGTOP-3043: Hide workaround for docker defect on aarch64 in build.sh
Olaf Flebbe [Fri, 8 Jun 2018 19:33:19 +0000 (21:33 +0200)] 
BIGTOP-3043: Hide workaround for docker defect on aarch64 in build.sh

7 months agoBIGTOP-3037: Download maven securely
Olaf Flebbe [Thu, 31 May 2018 21:16:09 +0000 (23:16 +0200)] 
BIGTOP-3037: Download maven securely

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
7 months agoBIGTOP-3041: Failed to init hadoop hdfs using init-hdfs.sh
Jun He [Tue, 5 Jun 2018 07:09:48 +0000 (15:09 +0800)] 
BIGTOP-3041: Failed to init hadoop hdfs using init-hdfs.sh

Failed to initialize hdfs using init-hdfs.sh script with 2.8.1.
The root cause is some jars are missed in CLASS_PATH

This closes #365

Change-Id: I6e3e1f42d437097ee122150aa94b41780346fb4f
Signed-off-by: Jun He <junhe@apache.org>
7 months agoBIGTOP-3027: Wrong Leveldbjni native binary for aarch64
Jun He [Wed, 9 May 2018 08:01:29 +0000 (16:01 +0800)] 
BIGTOP-3027: Wrong Leveldbjni native binary for aarch64

Leveldbjni used in Hadoop is v1.8, which only includes x86-64 native binaries.
It needs to build from source when creating hadoop packages on aarch64, like
what ppc does.

This closes #360

Change-Id: I4682e7885d9b5ac6d448d954ddffc78256fe0733
Signed-off-by: Jun He <junhe@apache.org>
7 months agoBIGTOP-3038: Add rollingupgrade option in hadoop-hdfs-namenode.svc
Sahil Aggarwal [Tue, 5 Jun 2018 13:56:21 +0000 (19:26 +0530)] 
BIGTOP-3038: Add rollingupgrade option in hadoop-hdfs-namenode.svc

Signed-off-by: Evans Ye <evansye@apache.org>
7 months agoBIGTOP-3036: Download gradle and groovy via TLS and do signature checking on ant BIGTOP-3036
Olaf Flebbe [Wed, 30 May 2018 18:47:43 +0000 (20:47 +0200)] 
BIGTOP-3036: Download gradle and groovy via TLS and do signature checking on ant

Thanks to brian m. carlson for reporting

7 months agoBIGTOP-3033: Spark build on OpenSUSE is failed
Jun He [Fri, 18 May 2018 03:08:49 +0000 (11:08 +0800)] 
BIGTOP-3033: Spark build on OpenSUSE is failed

Spark building on OpenSUSE is failed as pdflatex is missing.
Add package to install list.

This closes #359

Change-Id: Ib3914ab05fd37418a5aa798fa37b88cb54e4079e
Signed-off-by: Jun He <junhe@apache.org>
8 months agoBIGTOP-3030: Remove phantomjs dependecy in build
Jun He [Thu, 10 May 2018 09:29:32 +0000 (17:29 +0800)] 
BIGTOP-3030: Remove phantomjs dependecy in build

Upstream Ambari depends on phantomjs to build. And at present phantomjs provides
only x86* prebuilt binaries. This leads to build failure on non-x86 platforms.

As phantomjs is used for test, not runtime dep, it can be removed from build to
solve this failure.

This closes #357

Change-Id: I06b4fa48705024bb5de9a9167c337db546abbc36
Signed-off-by: Jun He <junhe@apache.org>
8 months agoBIGTOP-3031: Auto find latest maven dist
Jun He [Mon, 14 May 2018 05:56:04 +0000 (13:56 +0800)] 
BIGTOP-3031: Auto find latest maven dist

Similar to ant, maven dist binaries may get updated from time to time on
Apache mirrors. A better way is to auto find latest maven-3 binary release.
Used the same way in BIGTOP-3022.

This closes #356

Change-Id: I0384a656701f62bd7ef7fc40cd6ab85533ba49fc
Signed-off-by: Jun He <junhe@apache.org>
8 months agoBIGTOP-3024: Zeppelin build is failed after Spark is bumpped to 2.2.1
Jun He [Thu, 3 May 2018 03:06:14 +0000 (11:06 +0800)] 
BIGTOP-3024: Zeppelin build is failed after Spark is bumpped to 2.2.1

After Spark is bumpped to 2.2.1 by BIGTOP-2991, zeppelin build is failed.
The root cause is zeppelin now try to download spark-2.2.1.tgz and
spark-2.2.1-bin-without-hadoop.tgz from http://d3kbcqa49mib13.cloudfront.net.
This site currently doesn't offer spark 2.2.1 related packages.
So the download link need to be updated to apache site:
https://www.apache.org/dist/spark/

This closes #355

Change-Id: Icea4dc0ac6d65104c7a071e7b044161423dffec5
Signed-off-by: Jun He <junhe@apache.org>
8 months agoBIGTOP-3011: zookeeper autopurge options
José Pekkarinen [Fri, 2 Mar 2018 14:20:25 +0000 (16:20 +0200)] 
BIGTOP-3011: zookeeper autopurge options

Closes #346

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
8 months agoBIGTOP-3023: Bump qfs to 2.0.0
Jun He [Tue, 24 Apr 2018 06:11:34 +0000 (14:11 +0800)] 
BIGTOP-3023: Bump qfs to 2.0.0

Prior to v2.0.0, qfs doesn't build with openssl-1.1.
This makes bigtop failed on debian-9.
Bump to v2.0.0 should solve this problem.

Change-Id: I4665e6de206ee928d1d8fbf21694bb92ef11752c
Signed-off-by: Jun He <jun.he@linaro.org>
8 months agoBIGTOP-3021: Remove HUE from Bigtop
Olaf Flebbe [Tue, 10 Apr 2018 18:51:04 +0000 (20:51 +0200)] 
BIGTOP-3021: Remove HUE from Bigtop

8 months agoBIGTOP-2991: Bump up Spark version to 2.2.1
Jun He [Tue, 24 Apr 2018 04:59:30 +0000 (12:59 +0800)] 
BIGTOP-2991: Bump up Spark version to 2.2.1

This closes #353

Change-Id: I5e5b47db667206bd4a810b1015793d69c1223cde
Signed-off-by: Jun He <junhe@apache.org>
9 months agoBIGTOP-3022 Auto find of latest Ant
Anton Chevychalov [Tue, 17 Apr 2018 11:20:39 +0000 (14:20 +0300)] 
BIGTOP-3022 Auto find of latest Ant

That patch allow to derminate latest version of Ant
available on apache dist.

There is one restriction. We query main apache mirror instead
of nearest one because we can't be sure it use Apache with same
configuration as main.

Signed-off-by: Olaf Flebbe <oflebbe@apache.org>
9 months agoBIGTOP-3018:detect-javahome script has a typo in variable name
zhtisi [Fri, 30 Mar 2018 04:46:06 +0000 (12:46 +0800)] 
BIGTOP-3018:detect-javahome script has a typo in variable name

https://issues.apache.org/jira/browse/BIGTOP-3018

This closes #350

Signed-off-by: Jun He <jun.he@apache.org>
9 months agoBIGTOP-2723:Fix asciidoctor-maven-plugin for HBase build on AArch64
Yuqi Gu [Thu, 29 Mar 2018 08:10:55 +0000 (08:10 +0000)] 
BIGTOP-2723:Fix asciidoctor-maven-plugin for HBase build on AArch64

This closes #200

Change-Id: Ia7318eb115027ceabeff9952588e5ec364b00144
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: Jun He <jun.he@apache.org>
9 months agoBIGTOP-3014: juju: use charm-env for shebangs
Kevin W Monroe [Sat, 10 Mar 2018 00:26:52 +0000 (18:26 -0600)] 
BIGTOP-3014: juju: use charm-env for shebangs

Fixes #349

10 months agoBIGTOP-3013: kafka charm: remove bigtop_version override
Kevin W Monroe [Fri, 9 Mar 2018 19:23:25 +0000 (13:23 -0600)] 
BIGTOP-3013: kafka charm: remove bigtop_version override

Fixes #347

10 months agoBIGTOP-3010: juju bundle refresh (feb 2018)
Kevin W Monroe [Wed, 18 Oct 2017 15:59:05 +0000 (15:59 +0000)] 
BIGTOP-3010: juju bundle refresh (feb 2018)

Closes #345

10 months agoBIGTOP-3005: zookeeper charm: support relation changes
José Pekkarinen [Fri, 23 Feb 2018 08:36:14 +0000 (10:36 +0200)] 
BIGTOP-3005: zookeeper charm: support relation changes

Closes #341

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
10 months agoBIGTOP-3007: hive pkg and charm opts for zookeeper
jamesbeedy [Tue, 27 Feb 2018 17:19:06 +0000 (09:19 -0800)] 
BIGTOP-3007: hive pkg and charm opts for zookeeper

Closes #344

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>
10 months agoBIGTOP-3003: Ant-1.9.9 tarball is removed from Apache mirrors
Jun He [Mon, 26 Feb 2018 08:06:44 +0000 (16:06 +0800)] 
BIGTOP-3003: Ant-1.9.9 tarball is removed from Apache mirrors

As from 06-Feb, ant-1.9.10 is released and distributed
to Apache mirrors. Ant-1.9.9 is removed and can only
be found at Apache's archive. So bump ant to 1.9.10.

This closes #340

Change-Id: I8175684444b08a1607ee04589f5d252dac5588e2
Signed-off-by: Jun He <junhe@apache.org>
10 months agoBIGTOP-3004: Fix HBase build failure on Debian/Fedora
Jun He [Mon, 26 Feb 2018 07:28:42 +0000 (15:28 +0800)] 
BIGTOP-3004: Fix HBase build failure on Debian/Fedora

Debian/Fedora uses bash-4.4.12 which treats the extra
trailing backslash as error. HBase has patch for this
and should be backported.

Change-Id: I11353ec0fcdca6e04373c0dac34e0e23aa757f30
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
10 months agoBIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolchain
Akira Ajisaka [Tue, 27 Feb 2018 10:01:29 +0000 (19:01 +0900)] 
BIGTOP-3001. Change uid and gid for jenkins user in bigtop-toolchain

This closes #343

Signed-off-by: Jun He <junhe@apache.org>
10 months agoBIGTOP-3006: Add Jun He to team list
Jun He [Tue, 27 Feb 2018 07:57:04 +0000 (15:57 +0800)] 
BIGTOP-3006: Add Jun He to team list

Change-Id: I07e193bfad34d6fe393b02dd61270d6e0a6394bc
Signed-off-by: Jun He <junhe@apache.org>
10 months agoBIGTOP-3002: For Kafka, it should be possible to set the broker.id config and log...
Gregory Van Seghbroeck [Fri, 23 Feb 2018 20:07:47 +0000 (21:07 +0100)] 
BIGTOP-3002: For Kafka, it should be possible to set the broker.id config and log.dirs config.

Added 2 new parameters: broker_id, log_dirs. Both do not necessarily
need to be set. It is only when they are set , the parameters are used
in the server.properties file.

Signed-off-by: Olaf Flebbe <of@oflebbe.de>
11 months agoBIGTOP-2922: Drop arch specific bigtop-* images
Jun He [Tue, 14 Nov 2017 09:17:13 +0000 (17:17 +0800)] 
BIGTOP-2922: Drop arch specific bigtop-* images

Use unified Dockerfile for puppet/slaves images on different architectures.
Simplified the build process, and add gradle tasks for them.
Example: ./gradlew -POS=debian-9 -Pprefix=1.2.1 bigtop-puppet
         ./gradlew -POS=debian-9 -Pprefix=1.2.1 bigtop-slaves
This will create proper images with same image name as existings on different
HW architectures. This eases efforts for deploy and test.

Change-Id: I183452324faf54808a4e88981d604d8fe119b51a
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Evans Ye <evansye@apache.org>
11 months agoBIGTOP-2992: Building Pig with pre-arranged Forrest
Hari Krishna Dara [Sun, 4 Feb 2018 16:16:52 +0000 (21:46 +0530)] 
BIGTOP-2992: Building Pig with pre-arranged Forrest

Signed-off-by: Olaf Flebbe <of@oflebbe.de>