incubator-heron.git
2 months agoUpdated Netty to 4.1.76.Final nicknezis/netty-upgrade 3823/head
Nicholas Nezis [Fri, 29 Apr 2022 18:52:40 +0000 (18:52 +0000)] 
Updated Netty to 4.1.76.Final

2 months agoBump async from 2.6.3 to 2.6.4 in /website2/website (#3822)
dependabot[bot] [Fri, 29 Apr 2022 18:29:27 +0000 (14:29 -0400)] 
Bump async from 2.6.3 to 2.6.4 in /website2/website (#3822)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 months agoFix HealthManager (#3819)
choi se [Wed, 27 Apr 2022 18:25:47 +0000 (03:25 +0900)] 
Fix HealthManager (#3819)

Bug fixes in the HealthManager endpoint.

2 months agoSupport legacy API of HealthManger (#3818)
choi se [Thu, 14 Apr 2022 19:44:58 +0000 (04:44 +0900)] 
Support legacy API of HealthManger (#3818)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
2 months agoBump prismjs from 1.19.0 to 1.27.0 in /website2/website (#3815)
dependabot[bot] [Wed, 13 Apr 2022 22:37:03 +0000 (18:37 -0400)] 
Bump prismjs from 1.19.0 to 1.27.0 in /website2/website (#3815)

Bumps [prismjs](https://github.com/PrismJS/prism) from 1.19.0 to 1.27.0.
- [Release notes](https://github.com/PrismJS/prism/releases)
- [Changelog](https://github.com/PrismJS/prism/blob/master/CHANGELOG.md)
- [Commits](https://github.com/PrismJS/prism/compare/v1.19.0...v1.27.0)

---
updated-dependencies:
- dependency-name: prismjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 months agoRemove ByteSize for protobuf (#3816)
choi se [Tue, 12 Apr 2022 13:43:59 +0000 (22:43 +0900)] 
Remove ByteSize for protobuf (#3816)

Updates to  remove warning: 'ByteSize' is deprecated.

2 months agoBump url-parse from 1.4.7 to 1.5.10 in /website2/website (#3813)
dependabot[bot] [Tue, 12 Apr 2022 00:28:28 +0000 (20:28 -0400)] 
Bump url-parse from 1.4.7 to 1.5.10 in /website2/website (#3813)

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.4.7 to 1.5.10.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](https://github.com/unshiftio/url-parse/compare/1.4.7...1.5.10)

---
updated-dependencies:
- dependency-name: url-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 months agoUpdates to Heron UI and Tracker code to work with Python 3 updates (#3786)
Nicholas Nezis [Tue, 12 Apr 2022 00:26:51 +0000 (20:26 -0400)] 
Updates to Heron UI and Tracker code to work with Python 3 updates (#3786)

* Fix for missing physical plan in UI
* Fix the Heron UI timeline metrics
* Removed Tracker EnvelopeAPI Router which was not working
* Updated AckingTopology to not burn so much CPU
* Updated pylint due to build issue which resulted in lots of format updates

Co-authored-by: choi se <thinker0@gmail.com>
Co-authored-by: choi se <357785+thinker0@users.noreply.github.com>
Co-authored-by: Saad Ur Rahman <surahman@users.noreply.github.com>
Co-authored-by: Saad Ur Rahman <saadurrahman@apache.org>
2 months agoBump lodash from 4.17.15 to 4.17.21 in /website2/website (#3812)
dependabot[bot] [Mon, 11 Apr 2022 03:07:03 +0000 (23:07 -0400)] 
Bump lodash from 4.17.15 to 4.17.21 in /website2/website (#3812)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.15...4.17.21)

---
updated-dependencies:
- dependency-name: lodash
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 months agoReplace Helm chart's generic Bookkeeper platform property with specific properties...
Billy Romansky [Mon, 11 Apr 2022 01:09:47 +0000 (21:09 -0400)] 
Replace Helm chart's generic Bookkeeper platform property with specific properties (#3798)

2 months agoBump minimist from 1.2.5 to 1.2.6 in /website2/website (#3811)
dependabot[bot] [Mon, 11 Apr 2022 01:05:19 +0000 (21:05 -0400)] 
Bump minimist from 1.2.5 to 1.2.6 in /website2/website (#3811)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 months agoAdd SLF4JBridgeHandler (#3809)
choi se [Wed, 6 Apr 2022 03:38:34 +0000 (12:38 +0900)] 
Add SLF4JBridgeHandler (#3809)

3 months agoAdd logback.xml (#3807)
choi se [Sat, 2 Apr 2022 04:15:55 +0000 (13:15 +0900)] 
Add logback.xml (#3807)

3 months agoUpdated Bazel rules_pkg build dependency to 0.6.0 (#3806)
Nicholas Nezis [Tue, 29 Mar 2022 02:27:19 +0000 (22:27 -0400)] 
Updated Bazel rules_pkg build dependency to 0.6.0 (#3806)

3 months agoUpdate cppcheck to 2.7 (#3804)
Nicholas Nezis [Sat, 26 Mar 2022 19:20:21 +0000 (15:20 -0400)] 
Update cppcheck to 2.7 (#3804)

3 months agoUpdated to fix MacOS cppcheck RuntimeError (#3803)
Nicholas Nezis [Sat, 26 Mar 2022 19:13:38 +0000 (15:13 -0400)] 
Updated to fix MacOS cppcheck RuntimeError (#3803)

3 months agoRemoving prevously missed refs to removed files (#3805)
Nicholas Nezis [Sat, 26 Mar 2022 19:12:30 +0000 (15:12 -0400)] 
Removing prevously missed refs to removed files (#3805)

3 months agoAdded Bazel Platform auto discovery (and lots of other related cleanup) (#3779)
Nicholas Nezis [Fri, 25 Mar 2022 20:55:40 +0000 (16:55 -0400)] 
Added Bazel Platform auto discovery (and lots of other related cleanup) (#3779)

* Added Bazel platforms dependency

* Updated BUILD files to use platform info

* Updates to remove platform setting in the scripts

* Removing TravisCI special config.

* Remove outdated Applatix CI scripts

* Remove outdated Docker bazel.rc

* Removed Mesos/Marathon from Vagrantfile init.sh

* travis build.sh cleanup

* Some Dockerfile cleanup

* Removing OS detect script

* Remove referenced to deleted bazelrc from the various Dockerfiles

* Added comment to give hint about need to upgrade Docker version for anyone trying to build Ubuntu 22.04 image

* Create Dockerfile.dist.ubuntu22.04 (#3800)

Co-authored-by: Windham Wong <windhamwong@nva-hk.com>
Co-authored-by: Saad Ur Rahman <surahman@users.noreply.github.com>
3 months agoUpgrade protobuf to 3.16.1 CVE-2021-22569 (#3797)
choi se [Fri, 25 Mar 2022 17:42:11 +0000 (02:42 +0900)] 
Upgrade protobuf to 3.16.1 CVE-2021-22569 (#3797)

* Upgrade protobuf to 3.16.1 to address CVE-2021-22569

- protobuf < 3.16.1 contains DoS vulnerability CVE-2021-22569, https://nvd.nist.gov/vuln/detail/CVE-2021-22569.

* Fix include

(cherry picked from commit 52ed112dc14bc831afe25c1e3a66a96d9d4e7f06)

* Update protobuf-3.16.0 of Python
Update protobuf-java-3.16.1 of Java

* typo conflit

3 months agoUpdate build-docker.sh (#3799)
Windham Wong [Fri, 25 Mar 2022 15:54:30 +0000 (23:54 +0800)] 
Update build-docker.sh (#3799)

Removing the debug message on build-docker.sh

3 months agoUpdate to Guice 5.1.0 to support JDK17 (#3792)
choi se [Thu, 24 Mar 2022 05:33:32 +0000 (14:33 +0900)] 
Update to Guice 5.1.0 to support JDK17 (#3792)

3 months agoRemove fixed young generation heap space of instance (#3789)
choi se [Mon, 21 Mar 2022 07:00:22 +0000 (16:00 +0900)] 
Remove fixed young generation heap space of instance (#3789)

3 months agoChange Log4J to SLF4J for Java Logging (#3791)
choi se [Mon, 21 Mar 2022 06:17:11 +0000 (15:17 +0900)] 
Change Log4J to SLF4J for Java Logging (#3791)

3 months agoRemove unnecessary Bookkeeper format init container from Helm chart (#3795)
Nicholas Nezis [Sat, 19 Mar 2022 23:49:21 +0000 (19:49 -0400)] 
Remove unnecessary Bookkeeper format init container from Helm chart (#3795)

Co-authored-by: Billy Romansky <wromansky@protonmail.com>
3 months agoRemove use of unsecure git protocol in build dependency (#3794) 3793/head
choi se [Fri, 18 Mar 2022 15:37:50 +0000 (00:37 +0900)] 
Remove use of unsecure git protocol in build dependency (#3794)

3 months agoFix conflict in Rocky 8 Dockerfile (#3787)
choi se [Sun, 6 Mar 2022 14:01:30 +0000 (23:01 +0900)] 
Fix conflict in Rocky 8 Dockerfile (#3787)

4 months agoAdd Rocky 8 container option (#3773)
choi se [Sun, 6 Mar 2022 04:32:02 +0000 (13:32 +0900)] 
Add Rocky 8 container option (#3773)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
4 months agoAdd back legacy CentOS 7 with newer gcc and Python (#3772)
choi se [Sat, 5 Mar 2022 18:22:35 +0000 (03:22 +0900)] 
Add back legacy CentOS 7 with newer gcc and Python (#3772)

4 months agoRemoving old Tornado asynchronous annotations from Heron shell (#3781)
Nicholas Nezis [Wed, 2 Mar 2022 04:32:06 +0000 (23:32 -0500)] 
Removing old Tornado asynchronous annotations from Heron shell (#3781)

4 months agoAdded missing packing plan handler in heron-tracker (#3783)
Nicholas Nezis [Mon, 28 Feb 2022 08:45:09 +0000 (03:45 -0500)] 
Added missing packing plan handler in heron-tracker (#3783)

4 months agoUpdate to fix Heron UI Jinja2 issue (#3777)
Nicholas Nezis [Sat, 26 Feb 2022 19:54:11 +0000 (14:54 -0500)] 
Update to fix Heron UI Jinja2 issue (#3777)

4 months agoFixing a type which broke the helm chart (#3775)
Nicholas Nezis [Sat, 26 Feb 2022 07:20:51 +0000 (02:20 -0500)] 
Fixing a type which broke the helm chart (#3775)

5 months agoUpgrade Python to 3.8 as a minimum. Also simplified Tracker implementation. (#3646) 0.20.5-incubating-rc1-20220214
Oliver Bristow [Tue, 25 Jan 2022 16:37:11 +0000 (16:37 +0000)] 
Upgrade Python to 3.8 as a minimum. Also simplified Tracker implementation. (#3646)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
Co-authored-by: Saad Ur Rahman <saadurrahman@apache.org>
Co-authored-by: Nicholas Nezis nanezis <nanezis@ip-10-113-78-208.evoforge.org>
5 months agoFixed a segfault issue on macOS. Deleting item invalidated the iterator. (#3767)
Nicholas Nezis [Mon, 24 Jan 2022 19:25:58 +0000 (14:25 -0500)] 
Fixed a segfault issue on macOS. Deleting item invalidated the iterator. (#3767)

5 months agoUpdated to Google Test 1.11.0 which now directly supports Bazel (#3765)
Nicholas Nezis [Mon, 24 Jan 2022 13:18:27 +0000 (08:18 -0500)] 
Updated to Google Test 1.11.0 which now directly supports Bazel (#3765)

5 months agoUpgrade to Bazel 4.2.2 (#3764)
Nicholas Nezis [Mon, 24 Jan 2022 13:17:21 +0000 (08:17 -0500)] 
Upgrade to Bazel 4.2.2 (#3764)

5 months agoUbuntu 22.04 Support (#3761)
Saad Ur Rahman [Sun, 23 Jan 2022 16:46:30 +0000 (11:46 -0500)] 
Ubuntu 22.04 Support (#3761)

- Added a Ubuntu 22.04 Docker Container.
- Update to support compiling and linking of <libunwind 1.5.0> with the C++14 standard. g++ seems to be defaulting to C++17 which is not compatible with <libunwind 1.5.0>'s dynamic exception specifications.

5 months agolibunwind 1.5.0 Upgrade (#3760)
Saad Ur Rahman [Fri, 21 Jan 2022 23:24:37 +0000 (18:24 -0500)] 
libunwind 1.5.0 Upgrade (#3760)

This library is used by C++ components to maintain a call history. Upgrading to newer Linux kernels necessitates this update.

5 months agoHeronPy 0.20.5 release prep cleanup (#3759)
Nicholas Nezis [Tue, 18 Jan 2022 02:56:44 +0000 (21:56 -0500)] 
HeronPy 0.20.5 release prep cleanup (#3759)

5 months agoHelm chart cleanup (#3758)
Nicholas Nezis [Mon, 17 Jan 2022 16:50:38 +0000 (11:50 -0500)] 
Helm chart cleanup (#3758)

* Upgrade to Helm 3.7.2
* Add disclaimer files
* Updated Chart.yaml to v2 apiVersion (Helm 3)

5 months agoUpdated Dhalion to 0.2.6 (#3757)
Nicholas Nezis [Wed, 5 Jan 2022 04:38:05 +0000 (23:38 -0500)] 
Updated Dhalion to 0.2.6 (#3757)

5 months agoUpdated Netty to 4.1.72 (#3755)
Nicholas Nezis [Tue, 4 Jan 2022 22:38:51 +0000 (17:38 -0500)] 
Updated Netty to 4.1.72 (#3755)

5 months agoUpdated Kubernetes client library to 14.0.0 (#3754)
Nicholas Nezis [Tue, 4 Jan 2022 21:35:45 +0000 (16:35 -0500)] 
Updated Kubernetes client library to 14.0.0 (#3754)

* Updated Kubernetes client library to 14.0.0
* Removed explicit dependency on `snakeyml`. Instead the proper version will be calculated as a transitive dependency.

5 months agoUpdated External JVM Rules plugin to 4.2 (#3753)
Nicholas Nezis [Tue, 4 Jan 2022 18:13:17 +0000 (13:13 -0500)] 
Updated External JVM Rules plugin to 4.2 (#3753)

6 months agoAdded Docker to Vagrant VM init script (#3756)
Nicholas Nezis [Tue, 4 Jan 2022 13:46:32 +0000 (08:46 -0500)] 
Added Docker to Vagrant VM init script (#3756)

* Added docker to the Vagrant VM init script

* Updated the subnet to match the Vagrant allowed range

6 months agoAdded missing Javadocs to Kubernetes V1Controller (#3752)
Saad Ur Rahman [Mon, 3 Jan 2022 15:11:51 +0000 (10:11 -0500)] 
Added missing Javadocs to Kubernetes V1Controller (#3752)

6 months ago[Heron-3723] Add Kubernetes support for Empty Dir, Host Path, and NFS via CLI (#3747) 0.20.5-incubating-rc1
Saad Ur Rahman [Sun, 19 Dec 2021 06:03:32 +0000 (01:03 -0500)] 
[Heron-3723] Add Kubernetes support for Empty Dir, Host Path, and NFS via CLI (#3747)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
6 months agoUpdated to fix missing anticrlf dependency in Heron shell process. (#3749)
Nicholas Nezis [Sun, 19 Dec 2021 05:54:36 +0000 (00:54 -0500)] 
Updated to fix missing anticrlf dependency in Heron shell process. (#3749)

6 months agoadding Saad (#3748)
Josh Fischer [Wed, 15 Dec 2021 17:23:28 +0000 (11:23 -0600)] 
adding Saad (#3748)

6 months agoFix dependency (#3746)
choi se [Fri, 10 Dec 2021 11:05:01 +0000 (20:05 +0900)] 
Fix dependency (#3746)

6 months agoBump kazoo library from 2.7.0 to 2.8.0 to fix a compatibility issue of Python 3.8...
Windham Wong [Mon, 6 Dec 2021 05:37:25 +0000 (13:37 +0800)] 
Bump kazoo library from 2.7.0 to 2.8.0 to fix a compatibility issue of Python 3.8 (#3743)

6 months ago[Heron-3724] Separate the Manager and Executors. (#3741)
Saad Ur Rahman [Sun, 5 Dec 2021 17:10:39 +0000 (12:10 -0500)] 
[Heron-3724] Separate the Manager and Executors. (#3741)

All of the changes below are pertaining to the Kubernetes Scheduler.

* Topology's Executors and Manager separated into individual StatefulSets.

* Added the ability to specify individual Pod Templates for a topology's Executors and Manager via the CLI.

* Added the ability to specify Persistent Volume Claims for a topology's Executors and Manager via the CLI.

* Added the ability to specify resource Requests and Limits for a topology's Executors and Manager via the CLI.

7 months ago[Heron-3723] Add support for Persistent Volumes for stateful storage (#3725)
Saad Ur Rahman [Tue, 30 Nov 2021 05:20:34 +0000 (00:20 -0500)] 
[Heron-3723] Add support for Persistent Volumes for stateful storage (#3725)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
Co-authored-by: Josh Fischer <josh@joshfischer.io>
Co-authored-by: zhangshaoning <32099766+zhangshaoning1@users.noreply.github.com>
Co-authored-by: Huijun Wu <huijunwu@users.noreply.github.com>
Co-authored-by: Huijun Wu <huijunw@twitter.com>
7 months agoAdded ability to restart a topology on Kubernetes (#3740)
Saad Ur Rahman [Sat, 20 Nov 2021 02:25:02 +0000 (21:25 -0500)] 
Added ability to restart a topology on Kubernetes (#3740)

7 months agoProtect from arbitrary json file reading and writing in integration_test http_server...
Huijun Wu [Wed, 17 Nov 2021 05:56:51 +0000 (21:56 -0800)] 
Protect from arbitrary json file reading and writing in integration_test http_server (#3739)

7 months agoFixed CI Pipeline Failures Involving the <stmgr_unittest> (#3735)
Saad Ur Rahman [Mon, 15 Nov 2021 06:37:45 +0000 (01:37 -0500)] 
Fixed CI Pipeline Failures Involving the <stmgr_unittest> (#3735)

7 months agoBump Netty to 4.1.70 (#3729)
Nicholas Nezis [Sat, 13 Nov 2021 16:59:24 +0000 (11:59 -0500)] 
Bump Netty to 4.1.70 (#3729)

7 months agoUpdated deprecated policy/v1beta1 to policy/v1 PodDisruptionBudget API (#3737)
Nicholas Nezis [Sat, 13 Nov 2021 16:58:57 +0000 (11:58 -0500)] 
Updated deprecated policy/v1beta1 to policy/v1 PodDisruptionBudget API (#3737)

7 months agoFix get_heron_tracker_dir (#3734)
choi se [Sat, 13 Nov 2021 03:52:13 +0000 (12:52 +0900)] 
Fix get_heron_tracker_dir (#3734)

7 months agoFix `get_heron_dir` which is causing Python processes to fail (#3731)
choi se [Fri, 12 Nov 2021 19:45:50 +0000 (04:45 +0900)] 
Fix `get_heron_dir` which is causing Python processes to fail (#3731)

7 months agoHelm: Adding option to switch the Kubernetes services between NodePort and ClusterIP...
Windham Wong [Thu, 11 Nov 2021 04:28:52 +0000 (12:28 +0800)] 
Helm: Adding option to switch the Kubernetes services between NodePort and ClusterIP (#3721)

* Adding option to switch the apiserver service between NodePort and ClusterIP

* Now support custom NodePort port number

* Better value.yaml settings for nodeport config

* Some cleanup and allows K8s to auto-pick nodeport

* Fixing typo

Co-authored-by: Windham Wong <windham.wong@stormeye.io>
Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
7 months agoUpdate pip-2020-resolver of PexBuilder to fix darwin building (#3727)
choi se [Thu, 11 Nov 2021 04:13:07 +0000 (13:13 +0900)] 
Update pip-2020-resolver of PexBuilder to fix darwin building (#3727)

7 months agoUpdated glog and gflags libraries (#3728)
choi se [Mon, 8 Nov 2021 23:46:22 +0000 (08:46 +0900)] 
Updated glog and gflags libraries (#3728)

8 months ago[Heron 3707] ConfigMap Pod Template Support Documentation (#3717)
Saad Ur Rahman [Tue, 2 Nov 2021 13:41:31 +0000 (09:41 -0400)] 
[Heron 3707]  ConfigMap Pod Template Support Documentation (#3717)

* [Docs] Initial document.

* [Docs] <ConfigMaps> section.

* [Docs] generating and registering a ConfigMap.

* [Docs] linking into the sidebar.

* [Docs] updated for the CONFIG-MAP-NAME.POD-TEMPLATE-NAME.

* [Docs] updates with output from <minkube kubectl> and for commands.

* [Docs] updates on how to disable and remove topologies in the event of an error.

* [Docs] added items overwritten by Heron in Pod Templates.

* [Docs] minor typos and additions.

* [Docs] customisation of Heron's executor and support for auxiliary containers.

* [Docs] fixed dangling references within the Scheduler and cleaned up the State Manager in the event of a failed topology submission.

* [Docs] fixed description of Tolerations.

* [Docs] added info on Toleration merging.

* [Docs] updates for namespace and removal of deprecated taints.

* [Docs] Heron config values for Limits take precedence.

8 months ago[HERON-3707] ConfigMap Pod Template Support (#3710)
Saad Ur Rahman [Tue, 2 Nov 2021 13:40:39 +0000 (09:40 -0400)] 
[HERON-3707] ConfigMap Pod Template Support (#3710)

* [Kubernetes] setup basic mount info for Pod ConfigMap.

* [Kubernetes] updated function signature to handle Pod Template ConfigMap name.

* [kubernetes] extracting Pod Template ConfigMap name from <Config>.

* [kubernetes] checking for Pod Template ConfigMap and appropriately adding to Stateful Set.

* [kubernetes] Java Style lint fix.

* [Tests] Kubernetes Controller tests for Pod Template ConfigMap.

* [Tests] Kubernetes Constants and Context tests for Pod Template ConfigMap.

* [Tests] Kubernetes V1Controller test suite stubbed.

* [Tests] Java style lint fixed.

* [Tests] Kubernetes V1Controller Pod Template ConfigMap volume mount.

* [Kubernetes] cleaned up to begin work on <loadPodFromTemplate>.

* [Kubernetes] created <loadPodFromTemplate>.

* [Tests] Begun mock test setup of <loadPodFromTemplate>.

* [Kubernetes] style check/linting fix.

* [Kubernetes] refactoring <V1Controller> and <KubernetesController>.

* [Kubernetes] added description to failed to locate exception.

* [Tests] <loadPodFromTemplate> Pod Template checks.

* [Kubernetes] check for no ConfigMaps set.

* [Tests] working on mocking null list of V1ConfigMapList.

* [Kubernetes] refactoring <loadPodFromTemplate>

Adding checks for null pointers. Default constructed V1 objects tend to have uninitialised fields set to null by default. Extracting <getConfigMaps> to method to support mocking.

* [Tests] Stubbed <getConfigMaps> and testing <loadPodFromTemplate>.

* [Kubernetes] <loadPodFromTemplate> adjusted to get <V1PodTemplateSpec> from <V1PodTemplate>.

* [Kubernetes] check for empty Pod Template in ConfigMap.

* [Tests] Valid Pod Template test.

* [Tests] Invalid Pod Template.

* [Tests] refactored test data to their respective tests.

* [Kubernetes] refactored <loadPodFromTemplate> for readability.

* [Kubernetes] params for <getConfigMaps> tweaked.

Judging from <release-11.0.0/kubernetes/src/main/java/io/kubernetes/client/openapi/apis/CoreV1Api.java> "optional" means the field can be set to <null>.

* [Kubernetes] <getPodTemplateLocation> extracting ConfigMap and Pod Template names.

* [Tests] <getPodTemplateLocation> for correct and incorrect parsing.

* [Kubernetes] <getPodTemplateLocation> catching empty names.

* [Tests] <getPodTemplateLocation> separated tests for ConfigMap and Pod Template names.

* [Kubernetes] updated <loadPodFromTemplate> to use ConfigMap and Pod Template names.

* [Tests] updated tests for <loadPodFromTemplate> to use ConfigMap and Pod Template names.

* [Kubernetes] added INFO logging to <loadPodFromTemplate> for the deployed Pod Template.

* [Kubernetes] Bug fixes in error messages for <loadPodFromTemplate>.

* [Kubernetes] bug fix and test for missing delimiter in <getPodTemplateLocation>.

* [Kubernetes] <getConfigMaps> namespace access updated.

* [Kubernetes] <configureRBAC> basic logic.

TODO: get API key for K8s.

* [Tests] cleaned up <V1Controller> tests.

* [Kubernetes] <configureRBAC> more detailed error log.

* [Kubernetes] <configureRBAC> role configurations.

* [Kubernetes] refactored <loadFromPodTemplate>.

* [Tests] switched to <ConfigMapBuilder> in <V1ControllerTest>.

* [Kubernetes] switched to <V1RoleBuilder> in <configureRBAC>.

* [Kubernetes] made <loadPodFromTemplate> protected.

Removed illegal reflection access to avoid support issues with newer testing frameworks.

* [Kubernetes] removed <configureRBAC>.

RBAC must be configured using Role/ClusterRole and RoleBinding/ClusterRoleBinding to the Heron <heron-apiserver> ServiceAccount.

* [Kubernetes] <getPodTemplateLocation> error message passed up.

* [Kubernetes] refactored <getPodSpec> to <finalizePodSpec>.

Heron should have the final say on the Pod Spec. This is as much a point of security as an operational one.

* [Kubernetes] Added boot flag to disable Pod Templates.

* [Tests] testing to validate boot flag for disabled Pod Templates.

* [Kubernetes] Wiring in boot flag to disable Pod Templates in  <loadPodFromTemplate>.

* [Tests] disabled Pod Templates output validation.

* [Kubernetes] Added class scoped variable <isPodTemplateDisabled>.

* [Kubernetes] <getContainer> modified to utilise supplied executor container.

* [Kubernetes] <getContainer> <V1EnvVar>s.

Environment variables merged with Heron defaults taking precedence.

* [Kubernetes] <getContainer> Limits.

Resource Limits merged with Heron defaults taking precedence.

* [Kubernetes] disabled Pod Templates will return error when attempting to submit.

* [Kubernetes] <API Server> configs.

Updated RBAC API version and added a commented flag command to disable to Pod Templates.

* [Kubernetes] <configureExecutorContainer>

Refactored <getContainer> to <configureExecutorContainer>. Permitting additional containers for side-car purposes.

* [Kubernetes] <configureExecutorContainer>

Switched to <TreeSet> with custom comparator for <V1EnvVar> name. <V1EnvVar>'s comparator performs a complete element wise comparison.

* [Kubernetes] <configureExecutorContainer>

Merged executor container ports with ports provided in Pod Template. Heron defaults take precedence.

* [Kubernetes] <mountVolumeIfPresent>

Merge volume mounts with those provided in Pod Template. Heron defaults take precedence.

* [Kubernetes] <V1Controller>

general cleanup of new code and comments.

* [Kubernetes] <mountVolumeIfPresent>

Error check for malformed Pod Template.

* [Kubernetes] <configureContainerPorts>

Refactored <getContainerPorts> and moved port merge with error handling to it.

* [Kubernetes] <configureExecutorContainer>

Removed a redundant <limit> put into the HashMap.

* [Kubernetes] <addVolumesIfPresent>

Merging Pod Template volume configs with Heron defaults. Heron values take precedence.

* [Kubernetes] <configureExecutorContainer>

Allow user values for CPU and MEMORY limits to override those provided by Heron.

* [Scheduler-Core] <LaunchRunner> handling <submit> errors better.

Some Schedulers, such as K8s, throw exceptions instead of returning false when <submit> fails. This leaves the Topology Manager with dangling references. An addition RPC call to the Scheduler is required to completely clear the state.

* [Kubernetes] <V1Controller>

General cleanup in tests and class.

* [Kubernetes] code review changes.

Code review from @nwangtw.
<KubernetesContext.getPodTemplateConfigMapDisabled> switched to <equalsIgnoreCase>.

* [Scheduler-Core] code review changes.

Code review from @nwangtw.
<LaunchRunner> error message assembly improved.
<LaunchRunner> added <FINE> level logging for failure to clear failed topology launch from Scheduler.

* [Tests] <configureContainerPorts>.

* [Kubernetes] <API Server> configs.

Code review from @nwangtw, @nicknezis.
Updated command to disable Pod Templates to <false> by default.

* [Kubernetes] <configureContainerEnvVars>

Logic for merging environment variables extracted to a method for testing.

* [Tests] <configureContainerEnvVars>.

* [Kubernetes] <configureExecutorContainer>

Wired in <configureContainerEnvVars> and removed old code.

* Update for Helm chart

* Updated version to match the other k8s ClusterRoleBindings

* [Kubernetes] <configureContainerResources>

Logic to configure container's resources extracted to method to facilitate testing.

* [Kubernetes] <configureExecutorContainer>

Removed old logic and wired <configureContainerResources> into <configureExecutorContainer>.

* [Tests] <testConfigureContainerPorts>.

Added a test for debugging ports.

* [Kubernetes] <addVolumesIfPresent>.

Exposed for testing.

* [Tests] <addVolumesIfPresent>.

Testing on a <hostPath> volume but will generalise across others.

* [Kubernetes] <mountVolumeIfPresent>.

Exposed for testing.

* [Tests] <mountVolumeIfPresent>.

Tested by setting a Volume Mount in the Config and then a custom Volume Mount in the container.

* [Tests] <addVolumesIfPresent>.

Cleaned up tests.

* [Tests] <mountVolumesIfPresent>.

Testing for when no Volume Mounts should be set.

* [Tests] <addVolumesIfPresent>.

Testing for when no Volumes should be set.

* Attempt to fix Travis CI build

* [Tests] <configureContainerEnvVars> <configureContainerPorts>.

Extracted logic to generate executor environment variables, ports, and debugging ports. This is to resolve production-testing code inconsistencies which may arise.

* [Tests] <V1ControllerTests>

General cleanup and simplification of test suite.

* Travis fix take 3

* Travis CI fix

* [Kubernetes] <V1ControllerUtils>

Added nested utility class to improve code maintainability.

<mergeListDedupe> will merge two input lists by keeping all values in one and deduplicating the second list.

* [Tests] <mergeListDedupe>.

Full battery of tests null lists, merged lists, and thrown errors.

* [Kubernetes] <V1Controller>.

Switched to using <mergeListsDedupe> to improve code maintainability.
Effects:
<addVolumesIfPresent>
<configureContainerEnvVars>
<configureContainerPorts>
<mountVolumeIfPresent>

* [Kubernetes] <V1Controller> cleaned up unneeded returns when using setter methods.

* [Kubernetes] <V1Controller>.

Merging Pod Specification Tolerations and deduplicating on the <V1Tolerations::key>.

* [Tests] <configureTolerations>.

Test for a null, empty, and merging of Toleration lists.

* [Kubernetes] <configurePodSpec>.

Wired in <configureTolerations>

* [Tests] cleaning up code.

* [Kubernetes] <configurePodSpec>.

Added check for multiple executor container specs in Pod Template. Will throw error if detected.

* [Tests] <V1Controller> general cleanup.

* [Kubernetes] Constants

Updated tolerations to remove deprecated taints.

* [Kubernetes] <V1Controller>

<getConfigMap> retrieving a single named ConfigMap in a specific namespace.
<loadPodFromTemplate> logic updated to handle a single ConfigMap.

* [Tests] <V1Controller>

Fixed and cleaned up tests after switching to <readNamespacedConfigMap>.

* [Kubernetes] <V1Controller>

Error message cleanup.

* [Tests] <V1Controller>

Test description cleanup.

* [Kubernetes] <KubernetesUtils>

Javadoc cleanup.

* [Tests] <KubernetesUtils>

Test description cleanup.

* [Tests] <V1Controller>

<configureContainerResources> Heron values take precedence for limits.

* [Kubernetes] <V1Controller>

<configureContainerResources> Heron values take precedence for limits.

* Add support for reading configmap

* Removed deprecated k8s tolerations

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
9 months agoupdatelogging (#3718)
Huijun Wu [Thu, 30 Sep 2021 18:32:07 +0000 (11:32 -0700)] 
updatelogging (#3718)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
9 months agoFix heron-downloader cannot load the jar package correctly (#3709)
zhangshaoning [Fri, 24 Sep 2021 11:25:01 +0000 (19:25 +0800)] 
Fix heron-downloader cannot load the jar package correctly (#3709)

1. Install heron in the linux system to the /usr/lib/heron directory
2. Create symbolic links of several executable files in heron/bin to the /usr/bin directory
3. Try to call the heron* program separately, and the heron-downloader program fails to load the jar package correctly, and other programs are normal

9 months agomirror link change (#3716)
Josh Fischer [Fri, 17 Sep 2021 14:58:37 +0000 (09:58 -0500)] 
mirror link change (#3716)

9 months agopointing older releases to be retrieved from archive.a.o (#3714)
Josh Fischer [Fri, 17 Sep 2021 11:12:19 +0000 (06:12 -0500)] 
pointing older releases to be retrieved from archive.a.o (#3714)

* pointing older releases to be retrieved from archive.a.o

* typo

9 months ago[docs] Added instructions for setting up Intellij IDEA with the Bazel plugin. (#3712)
Saad Ur Rahman [Sun, 12 Sep 2021 19:53:35 +0000 (15:53 -0400)] 
[docs] Added instructions for setting up Intellij IDEA with the Bazel plugin. (#3712)

10 months agoUpdated to allow for external Helm version when building on a non-versioned branch... s3-uploader
Nicholas Nezis [Thu, 2 Sep 2021 03:27:38 +0000 (23:27 -0400)] 
Updated to allow for external Helm version when building on a non-versioned branch (#3708)

10 months agofix label
Josh Fischer [Sat, 21 Aug 2021 21:34:06 +0000 (16:34 -0500)] 
fix label

10 months agoupdate labels
Josh Fischer [Sat, 21 Aug 2021 21:33:06 +0000 (16:33 -0500)] 
update labels

10 months agoUpdate asf yaml (#3706)
Josh Fischer [Sat, 21 Aug 2021 21:24:15 +0000 (16:24 -0500)] 
Update asf yaml (#3706)

* update asf.yaml

10 months agoBumped Bazel to 4.1.0 (#3703)
Nicholas Nezis [Sat, 21 Aug 2021 21:10:59 +0000 (17:10 -0400)] 
Bumped Bazel to 4.1.0 (#3703)

11 months agoFixed Kubernetes scheduler to provide Java remote debug ports (#3704)
Nicholas Nezis [Mon, 2 Aug 2021 03:17:54 +0000 (23:17 -0400)] 
Fixed Kubernetes scheduler to provide Java remote debug ports (#3704)

11 months agoUpdated Kryo to 5.2.0 (#3705)
Nicholas Nezis [Sun, 1 Aug 2021 20:41:28 +0000 (16:41 -0400)] 
Updated Kryo to 5.2.0 (#3705)

11 months agoAdd ability to add Kubernetes Secrets and SecretKeyRefs (#3702)
Nicholas Nezis [Sun, 11 Jul 2021 02:25:04 +0000 (22:25 -0400)] 
Add ability to add Kubernetes Secrets and SecretKeyRefs (#3702)

11 months agoAdd support for dynamic kubernetes labels on pod and service (#3701)
Nicholas Nezis [Fri, 9 Jul 2021 01:00:44 +0000 (21:00 -0400)] 
Add support for dynamic kubernetes labels on pod and service (#3701)

11 months agoAdded support for adding Kubernetes annotations to the topology pod and service ...
Nicholas Nezis [Thu, 8 Jul 2021 13:00:41 +0000 (09:00 -0400)] 
Added support for adding Kubernetes annotations to the topology pod and service (#3699)

12 months agoAdding missing heron s3 region into kubernetes config (#3697)
Bouryu [Thu, 17 Jun 2021 20:44:15 +0000 (04:44 +0800)] 
Adding missing heron s3 region into kubernetes config (#3697)

Co-authored-by: Windham Wong <windhamwong@nva-hk.com>
12 months agoFix extra.links of heron-tracker (#3696) 0.20.4-incubating-rc1-line
choi se [Tue, 15 Jun 2021 05:14:25 +0000 (14:14 +0900)] 
Fix extra.links of heron-tracker (#3696)

13 months ago0.20.4 site changes (#3695)
Josh Fischer [Wed, 2 Jun 2021 01:12:16 +0000 (20:12 -0500)] 
0.20.4 site changes (#3695)

* links

* initial changes

* adding versioned docs; updating heron:bazel matrix

* fix property

* updating urls; making install docs more clear

* clean up versioned getting started docs

* remove duplicated text

* adding simple docker instructions

* clean up more urls, adding release article for 0.20.4-incubating

* clean up more urls, adding release article for 0.20.4-incubating

* fix typo

* grammar

* attempt to remove noindex

* fix maven command

* adding missed commands in versioned docs

13 months agoUpdate Bookkeeper to 4.13.0 and Zookeeper to 3.6.3 (#3692) 0.20.4-incubating-rc1-20210601
Nicholas Nezis [Mon, 24 May 2021 00:21:16 +0000 (20:21 -0400)] 
Update Bookkeeper to 4.13.0 and Zookeeper to 3.6.3 (#3692)

* Update to Bookkeeper and Zookeeper

* Committed this by accident

* Updating Guava to fix submit failure

* Updating BK image versions

13 months agolinks (#3694)
Josh Fischer [Sun, 23 May 2021 13:46:21 +0000 (08:46 -0500)] 
links (#3694)

14 months agoAdding disclaimers to release artifacts (#3689) ubuntu20.04-rpi-arm64 0.20.4-incubating-rc1
Josh Fischer [Wed, 28 Apr 2021 21:08:37 +0000 (16:08 -0500)] 
Adding disclaimers to release artifacts (#3689)

14 months agoUse the proper featured Zookeeper config generation script (#3688)
Nicholas Nezis [Sun, 18 Apr 2021 06:18:12 +0000 (02:18 -0400)] 
Use the proper featured Zookeeper config generation script (#3688)

* Updates to make Zookeeper configurable through the Helm chart logic

14 months agoupdatedocandlink (#3685) ubuntu20.04-arm64
bed debug [Tue, 6 Apr 2021 19:12:25 +0000 (12:12 -0700)] 
updatedocandlink (#3685)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
15 months agoUpdated to Kryo 5.0.4 (#3683)
Nicholas Nezis [Fri, 19 Mar 2021 01:46:39 +0000 (21:46 -0400)] 
Updated to Kryo 5.0.4 (#3683)

15 months agoFixing incorrect article title (#3682)
Josh Fischer [Mon, 15 Mar 2021 04:18:38 +0000 (23:18 -0500)] 
Fixing incorrect article title (#3682)

15 months agoGrammar fix (#3681)
fredster33 [Wed, 10 Mar 2021 02:34:46 +0000 (18:34 -0800)] 
Grammar fix (#3681)

15 months agoRemove cppunit-dev dependency (#3680)
bed debug [Tue, 9 Mar 2021 08:47:27 +0000 (00:47 -0800)] 
Remove cppunit-dev dependency (#3680)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
16 months agoDecrease the minimum requirement of RAM of single instance to 16MB in Round Robin...
Windham Wong [Sat, 27 Feb 2021 20:21:18 +0000 (04:21 +0800)] 
Decrease the minimum requirement of RAM of single instance to 16MB in Round Robin algorithm (#3678)

Co-authored-by: Windham Wong <windham.wong@stormeye.io>
16 months agoRemoving unused Bookkeeper hostPort that causes port conflict (#3677)
Nicholas Nezis [Fri, 19 Feb 2021 03:05:05 +0000 (22:05 -0500)] 
Removing unused Bookkeeper hostPort that causes port conflict (#3677)

16 months agoSpecifying explicit transitive dependency for uvloop due to Python version conflict...
Nicholas Nezis [Thu, 18 Feb 2021 23:25:20 +0000 (18:25 -0500)] 
Specifying explicit transitive dependency for uvloop due to Python version conflict (#3675)

Adding a temporary fix until uvicorn fix is released

17 months agoFixed website style issue (#3673)
Josh Fischer [Wed, 3 Feb 2021 05:28:04 +0000 (23:28 -0600)] 
Fixed website style issue (#3673)

17 months agoJoshfischer/add download links (#3671)
Josh Fischer [Mon, 1 Feb 2021 14:54:48 +0000 (08:54 -0600)] 
Joshfischer/add download links (#3671)

* initial addition of heron download page.  Still work to do.

* adding start to release page; adding updates for versioned docs

* adding versioned docs

17 months agoReverting Glog back to 0.3.5 (#3672)
Nicholas Nezis [Mon, 1 Feb 2021 13:40:59 +0000 (08:40 -0500)] 
Reverting Glog back to 0.3.5 (#3672)

17 months agoUse Build Kit docker feature to shrink the Docker image size (#3669)
Nicholas Nezis [Wed, 27 Jan 2021 04:52:37 +0000 (23:52 -0500)] 
Use Build Kit docker feature to shrink the Docker image size (#3669)

17 months agoChanged heartbeat log statement to debug (#3667)
Nicholas Nezis [Fri, 22 Jan 2021 05:51:47 +0000 (00:51 -0500)] 
Changed heartbeat log statement to debug (#3667)