incubator-heron.git
6 months ago[Heron-3723] Add Kubernetes support for Empty Dir, Host Path, and NFS via CLI (#3747) 0.20.5-incubating-rc1
Saad Ur Rahman [Sun, 19 Dec 2021 06:03:32 +0000 (01:03 -0500)] 
[Heron-3723] Add Kubernetes support for Empty Dir, Host Path, and NFS via CLI (#3747)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
6 months agoUpdated to fix missing anticrlf dependency in Heron shell process. (#3749)
Nicholas Nezis [Sun, 19 Dec 2021 05:54:36 +0000 (00:54 -0500)] 
Updated to fix missing anticrlf dependency in Heron shell process. (#3749)

6 months agoadding Saad (#3748)
Josh Fischer [Wed, 15 Dec 2021 17:23:28 +0000 (11:23 -0600)] 
adding Saad (#3748)

6 months agoFix dependency (#3746)
choi se [Fri, 10 Dec 2021 11:05:01 +0000 (20:05 +0900)] 
Fix dependency (#3746)

6 months agoBump kazoo library from 2.7.0 to 2.8.0 to fix a compatibility issue of Python 3.8...
Windham Wong [Mon, 6 Dec 2021 05:37:25 +0000 (13:37 +0800)] 
Bump kazoo library from 2.7.0 to 2.8.0 to fix a compatibility issue of Python 3.8 (#3743)

6 months ago[Heron-3724] Separate the Manager and Executors. (#3741)
Saad Ur Rahman [Sun, 5 Dec 2021 17:10:39 +0000 (12:10 -0500)] 
[Heron-3724] Separate the Manager and Executors. (#3741)

All of the changes below are pertaining to the Kubernetes Scheduler.

* Topology's Executors and Manager separated into individual StatefulSets.

* Added the ability to specify individual Pod Templates for a topology's Executors and Manager via the CLI.

* Added the ability to specify Persistent Volume Claims for a topology's Executors and Manager via the CLI.

* Added the ability to specify resource Requests and Limits for a topology's Executors and Manager via the CLI.

7 months ago[Heron-3723] Add support for Persistent Volumes for stateful storage (#3725)
Saad Ur Rahman [Tue, 30 Nov 2021 05:20:34 +0000 (00:20 -0500)] 
[Heron-3723] Add support for Persistent Volumes for stateful storage (#3725)

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
Co-authored-by: Josh Fischer <josh@joshfischer.io>
Co-authored-by: zhangshaoning <32099766+zhangshaoning1@users.noreply.github.com>
Co-authored-by: Huijun Wu <huijunwu@users.noreply.github.com>
Co-authored-by: Huijun Wu <huijunw@twitter.com>
7 months agoAdded ability to restart a topology on Kubernetes (#3740)
Saad Ur Rahman [Sat, 20 Nov 2021 02:25:02 +0000 (21:25 -0500)] 
Added ability to restart a topology on Kubernetes (#3740)

7 months agoProtect from arbitrary json file reading and writing in integration_test http_server...
Huijun Wu [Wed, 17 Nov 2021 05:56:51 +0000 (21:56 -0800)] 
Protect from arbitrary json file reading and writing in integration_test http_server (#3739)

7 months agoFixed CI Pipeline Failures Involving the <stmgr_unittest> (#3735)
Saad Ur Rahman [Mon, 15 Nov 2021 06:37:45 +0000 (01:37 -0500)] 
Fixed CI Pipeline Failures Involving the <stmgr_unittest> (#3735)

7 months agoBump Netty to 4.1.70 (#3729)
Nicholas Nezis [Sat, 13 Nov 2021 16:59:24 +0000 (11:59 -0500)] 
Bump Netty to 4.1.70 (#3729)

7 months agoUpdated deprecated policy/v1beta1 to policy/v1 PodDisruptionBudget API (#3737)
Nicholas Nezis [Sat, 13 Nov 2021 16:58:57 +0000 (11:58 -0500)] 
Updated deprecated policy/v1beta1 to policy/v1 PodDisruptionBudget API (#3737)

7 months agoFix get_heron_tracker_dir (#3734)
choi se [Sat, 13 Nov 2021 03:52:13 +0000 (12:52 +0900)] 
Fix get_heron_tracker_dir (#3734)

7 months agoFix `get_heron_dir` which is causing Python processes to fail (#3731)
choi se [Fri, 12 Nov 2021 19:45:50 +0000 (04:45 +0900)] 
Fix `get_heron_dir` which is causing Python processes to fail (#3731)

7 months agoHelm: Adding option to switch the Kubernetes services between NodePort and ClusterIP...
Windham Wong [Thu, 11 Nov 2021 04:28:52 +0000 (12:28 +0800)] 
Helm: Adding option to switch the Kubernetes services between NodePort and ClusterIP (#3721)

* Adding option to switch the apiserver service between NodePort and ClusterIP

* Now support custom NodePort port number

* Better value.yaml settings for nodeport config

* Some cleanup and allows K8s to auto-pick nodeport

* Fixing typo

Co-authored-by: Windham Wong <windham.wong@stormeye.io>
Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
7 months agoUpdate pip-2020-resolver of PexBuilder to fix darwin building (#3727)
choi se [Thu, 11 Nov 2021 04:13:07 +0000 (13:13 +0900)] 
Update pip-2020-resolver of PexBuilder to fix darwin building (#3727)

7 months agoUpdated glog and gflags libraries (#3728)
choi se [Mon, 8 Nov 2021 23:46:22 +0000 (08:46 +0900)] 
Updated glog and gflags libraries (#3728)

8 months ago[Heron 3707] ConfigMap Pod Template Support Documentation (#3717)
Saad Ur Rahman [Tue, 2 Nov 2021 13:41:31 +0000 (09:41 -0400)] 
[Heron 3707]  ConfigMap Pod Template Support Documentation (#3717)

* [Docs] Initial document.

* [Docs] <ConfigMaps> section.

* [Docs] generating and registering a ConfigMap.

* [Docs] linking into the sidebar.

* [Docs] updated for the CONFIG-MAP-NAME.POD-TEMPLATE-NAME.

* [Docs] updates with output from <minkube kubectl> and for commands.

* [Docs] updates on how to disable and remove topologies in the event of an error.

* [Docs] added items overwritten by Heron in Pod Templates.

* [Docs] minor typos and additions.

* [Docs] customisation of Heron's executor and support for auxiliary containers.

* [Docs] fixed dangling references within the Scheduler and cleaned up the State Manager in the event of a failed topology submission.

* [Docs] fixed description of Tolerations.

* [Docs] added info on Toleration merging.

* [Docs] updates for namespace and removal of deprecated taints.

* [Docs] Heron config values for Limits take precedence.

8 months ago[HERON-3707] ConfigMap Pod Template Support (#3710)
Saad Ur Rahman [Tue, 2 Nov 2021 13:40:39 +0000 (09:40 -0400)] 
[HERON-3707] ConfigMap Pod Template Support (#3710)

* [Kubernetes] setup basic mount info for Pod ConfigMap.

* [Kubernetes] updated function signature to handle Pod Template ConfigMap name.

* [kubernetes] extracting Pod Template ConfigMap name from <Config>.

* [kubernetes] checking for Pod Template ConfigMap and appropriately adding to Stateful Set.

* [kubernetes] Java Style lint fix.

* [Tests] Kubernetes Controller tests for Pod Template ConfigMap.

* [Tests] Kubernetes Constants and Context tests for Pod Template ConfigMap.

* [Tests] Kubernetes V1Controller test suite stubbed.

* [Tests] Java style lint fixed.

* [Tests] Kubernetes V1Controller Pod Template ConfigMap volume mount.

* [Kubernetes] cleaned up to begin work on <loadPodFromTemplate>.

* [Kubernetes] created <loadPodFromTemplate>.

* [Tests] Begun mock test setup of <loadPodFromTemplate>.

* [Kubernetes] style check/linting fix.

* [Kubernetes] refactoring <V1Controller> and <KubernetesController>.

* [Kubernetes] added description to failed to locate exception.

* [Tests] <loadPodFromTemplate> Pod Template checks.

* [Kubernetes] check for no ConfigMaps set.

* [Tests] working on mocking null list of V1ConfigMapList.

* [Kubernetes] refactoring <loadPodFromTemplate>

Adding checks for null pointers. Default constructed V1 objects tend to have uninitialised fields set to null by default. Extracting <getConfigMaps> to method to support mocking.

* [Tests] Stubbed <getConfigMaps> and testing <loadPodFromTemplate>.

* [Kubernetes] <loadPodFromTemplate> adjusted to get <V1PodTemplateSpec> from <V1PodTemplate>.

* [Kubernetes] check for empty Pod Template in ConfigMap.

* [Tests] Valid Pod Template test.

* [Tests] Invalid Pod Template.

* [Tests] refactored test data to their respective tests.

* [Kubernetes] refactored <loadPodFromTemplate> for readability.

* [Kubernetes] params for <getConfigMaps> tweaked.

Judging from <release-11.0.0/kubernetes/src/main/java/io/kubernetes/client/openapi/apis/CoreV1Api.java> "optional" means the field can be set to <null>.

* [Kubernetes] <getPodTemplateLocation> extracting ConfigMap and Pod Template names.

* [Tests] <getPodTemplateLocation> for correct and incorrect parsing.

* [Kubernetes] <getPodTemplateLocation> catching empty names.

* [Tests] <getPodTemplateLocation> separated tests for ConfigMap and Pod Template names.

* [Kubernetes] updated <loadPodFromTemplate> to use ConfigMap and Pod Template names.

* [Tests] updated tests for <loadPodFromTemplate> to use ConfigMap and Pod Template names.

* [Kubernetes] added INFO logging to <loadPodFromTemplate> for the deployed Pod Template.

* [Kubernetes] Bug fixes in error messages for <loadPodFromTemplate>.

* [Kubernetes] bug fix and test for missing delimiter in <getPodTemplateLocation>.

* [Kubernetes] <getConfigMaps> namespace access updated.

* [Kubernetes] <configureRBAC> basic logic.

TODO: get API key for K8s.

* [Tests] cleaned up <V1Controller> tests.

* [Kubernetes] <configureRBAC> more detailed error log.

* [Kubernetes] <configureRBAC> role configurations.

* [Kubernetes] refactored <loadFromPodTemplate>.

* [Tests] switched to <ConfigMapBuilder> in <V1ControllerTest>.

* [Kubernetes] switched to <V1RoleBuilder> in <configureRBAC>.

* [Kubernetes] made <loadPodFromTemplate> protected.

Removed illegal reflection access to avoid support issues with newer testing frameworks.

* [Kubernetes] removed <configureRBAC>.

RBAC must be configured using Role/ClusterRole and RoleBinding/ClusterRoleBinding to the Heron <heron-apiserver> ServiceAccount.

* [Kubernetes] <getPodTemplateLocation> error message passed up.

* [Kubernetes] refactored <getPodSpec> to <finalizePodSpec>.

Heron should have the final say on the Pod Spec. This is as much a point of security as an operational one.

* [Kubernetes] Added boot flag to disable Pod Templates.

* [Tests] testing to validate boot flag for disabled Pod Templates.

* [Kubernetes] Wiring in boot flag to disable Pod Templates in  <loadPodFromTemplate>.

* [Tests] disabled Pod Templates output validation.

* [Kubernetes] Added class scoped variable <isPodTemplateDisabled>.

* [Kubernetes] <getContainer> modified to utilise supplied executor container.

* [Kubernetes] <getContainer> <V1EnvVar>s.

Environment variables merged with Heron defaults taking precedence.

* [Kubernetes] <getContainer> Limits.

Resource Limits merged with Heron defaults taking precedence.

* [Kubernetes] disabled Pod Templates will return error when attempting to submit.

* [Kubernetes] <API Server> configs.

Updated RBAC API version and added a commented flag command to disable to Pod Templates.

* [Kubernetes] <configureExecutorContainer>

Refactored <getContainer> to <configureExecutorContainer>. Permitting additional containers for side-car purposes.

* [Kubernetes] <configureExecutorContainer>

Switched to <TreeSet> with custom comparator for <V1EnvVar> name. <V1EnvVar>'s comparator performs a complete element wise comparison.

* [Kubernetes] <configureExecutorContainer>

Merged executor container ports with ports provided in Pod Template. Heron defaults take precedence.

* [Kubernetes] <mountVolumeIfPresent>

Merge volume mounts with those provided in Pod Template. Heron defaults take precedence.

* [Kubernetes] <V1Controller>

general cleanup of new code and comments.

* [Kubernetes] <mountVolumeIfPresent>

Error check for malformed Pod Template.

* [Kubernetes] <configureContainerPorts>

Refactored <getContainerPorts> and moved port merge with error handling to it.

* [Kubernetes] <configureExecutorContainer>

Removed a redundant <limit> put into the HashMap.

* [Kubernetes] <addVolumesIfPresent>

Merging Pod Template volume configs with Heron defaults. Heron values take precedence.

* [Kubernetes] <configureExecutorContainer>

Allow user values for CPU and MEMORY limits to override those provided by Heron.

* [Scheduler-Core] <LaunchRunner> handling <submit> errors better.

Some Schedulers, such as K8s, throw exceptions instead of returning false when <submit> fails. This leaves the Topology Manager with dangling references. An addition RPC call to the Scheduler is required to completely clear the state.

* [Kubernetes] <V1Controller>

General cleanup in tests and class.

* [Kubernetes] code review changes.

Code review from @nwangtw.
<KubernetesContext.getPodTemplateConfigMapDisabled> switched to <equalsIgnoreCase>.

* [Scheduler-Core] code review changes.

Code review from @nwangtw.
<LaunchRunner> error message assembly improved.
<LaunchRunner> added <FINE> level logging for failure to clear failed topology launch from Scheduler.

* [Tests] <configureContainerPorts>.

* [Kubernetes] <API Server> configs.

Code review from @nwangtw, @nicknezis.
Updated command to disable Pod Templates to <false> by default.

* [Kubernetes] <configureContainerEnvVars>

Logic for merging environment variables extracted to a method for testing.

* [Tests] <configureContainerEnvVars>.

* [Kubernetes] <configureExecutorContainer>

Wired in <configureContainerEnvVars> and removed old code.

* Update for Helm chart

* Updated version to match the other k8s ClusterRoleBindings

* [Kubernetes] <configureContainerResources>

Logic to configure container's resources extracted to method to facilitate testing.

* [Kubernetes] <configureExecutorContainer>

Removed old logic and wired <configureContainerResources> into <configureExecutorContainer>.

* [Tests] <testConfigureContainerPorts>.

Added a test for debugging ports.

* [Kubernetes] <addVolumesIfPresent>.

Exposed for testing.

* [Tests] <addVolumesIfPresent>.

Testing on a <hostPath> volume but will generalise across others.

* [Kubernetes] <mountVolumeIfPresent>.

Exposed for testing.

* [Tests] <mountVolumeIfPresent>.

Tested by setting a Volume Mount in the Config and then a custom Volume Mount in the container.

* [Tests] <addVolumesIfPresent>.

Cleaned up tests.

* [Tests] <mountVolumesIfPresent>.

Testing for when no Volume Mounts should be set.

* [Tests] <addVolumesIfPresent>.

Testing for when no Volumes should be set.

* Attempt to fix Travis CI build

* [Tests] <configureContainerEnvVars> <configureContainerPorts>.

Extracted logic to generate executor environment variables, ports, and debugging ports. This is to resolve production-testing code inconsistencies which may arise.

* [Tests] <V1ControllerTests>

General cleanup and simplification of test suite.

* Travis fix take 3

* Travis CI fix

* [Kubernetes] <V1ControllerUtils>

Added nested utility class to improve code maintainability.

<mergeListDedupe> will merge two input lists by keeping all values in one and deduplicating the second list.

* [Tests] <mergeListDedupe>.

Full battery of tests null lists, merged lists, and thrown errors.

* [Kubernetes] <V1Controller>.

Switched to using <mergeListsDedupe> to improve code maintainability.
Effects:
<addVolumesIfPresent>
<configureContainerEnvVars>
<configureContainerPorts>
<mountVolumeIfPresent>

* [Kubernetes] <V1Controller> cleaned up unneeded returns when using setter methods.

* [Kubernetes] <V1Controller>.

Merging Pod Specification Tolerations and deduplicating on the <V1Tolerations::key>.

* [Tests] <configureTolerations>.

Test for a null, empty, and merging of Toleration lists.

* [Kubernetes] <configurePodSpec>.

Wired in <configureTolerations>

* [Tests] cleaning up code.

* [Kubernetes] <configurePodSpec>.

Added check for multiple executor container specs in Pod Template. Will throw error if detected.

* [Tests] <V1Controller> general cleanup.

* [Kubernetes] Constants

Updated tolerations to remove deprecated taints.

* [Kubernetes] <V1Controller>

<getConfigMap> retrieving a single named ConfigMap in a specific namespace.
<loadPodFromTemplate> logic updated to handle a single ConfigMap.

* [Tests] <V1Controller>

Fixed and cleaned up tests after switching to <readNamespacedConfigMap>.

* [Kubernetes] <V1Controller>

Error message cleanup.

* [Tests] <V1Controller>

Test description cleanup.

* [Kubernetes] <KubernetesUtils>

Javadoc cleanup.

* [Tests] <KubernetesUtils>

Test description cleanup.

* [Tests] <V1Controller>

<configureContainerResources> Heron values take precedence for limits.

* [Kubernetes] <V1Controller>

<configureContainerResources> Heron values take precedence for limits.

* Add support for reading configmap

* Removed deprecated k8s tolerations

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
9 months agoupdatelogging (#3718)
Huijun Wu [Thu, 30 Sep 2021 18:32:07 +0000 (11:32 -0700)] 
updatelogging (#3718)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
9 months agoFix heron-downloader cannot load the jar package correctly (#3709)
zhangshaoning [Fri, 24 Sep 2021 11:25:01 +0000 (19:25 +0800)] 
Fix heron-downloader cannot load the jar package correctly (#3709)

1. Install heron in the linux system to the /usr/lib/heron directory
2. Create symbolic links of several executable files in heron/bin to the /usr/bin directory
3. Try to call the heron* program separately, and the heron-downloader program fails to load the jar package correctly, and other programs are normal

9 months agomirror link change (#3716)
Josh Fischer [Fri, 17 Sep 2021 14:58:37 +0000 (09:58 -0500)] 
mirror link change (#3716)

9 months agopointing older releases to be retrieved from archive.a.o (#3714)
Josh Fischer [Fri, 17 Sep 2021 11:12:19 +0000 (06:12 -0500)] 
pointing older releases to be retrieved from archive.a.o (#3714)

* pointing older releases to be retrieved from archive.a.o

* typo

9 months ago[docs] Added instructions for setting up Intellij IDEA with the Bazel plugin. (#3712)
Saad Ur Rahman [Sun, 12 Sep 2021 19:53:35 +0000 (15:53 -0400)] 
[docs] Added instructions for setting up Intellij IDEA with the Bazel plugin. (#3712)

10 months agoUpdated to allow for external Helm version when building on a non-versioned branch... s3-uploader
Nicholas Nezis [Thu, 2 Sep 2021 03:27:38 +0000 (23:27 -0400)] 
Updated to allow for external Helm version when building on a non-versioned branch (#3708)

10 months agofix label
Josh Fischer [Sat, 21 Aug 2021 21:34:06 +0000 (16:34 -0500)] 
fix label

10 months agoupdate labels
Josh Fischer [Sat, 21 Aug 2021 21:33:06 +0000 (16:33 -0500)] 
update labels

10 months agoUpdate asf yaml (#3706)
Josh Fischer [Sat, 21 Aug 2021 21:24:15 +0000 (16:24 -0500)] 
Update asf yaml (#3706)

* update asf.yaml

10 months agoBumped Bazel to 4.1.0 (#3703)
Nicholas Nezis [Sat, 21 Aug 2021 21:10:59 +0000 (17:10 -0400)] 
Bumped Bazel to 4.1.0 (#3703)

11 months agoFixed Kubernetes scheduler to provide Java remote debug ports (#3704)
Nicholas Nezis [Mon, 2 Aug 2021 03:17:54 +0000 (23:17 -0400)] 
Fixed Kubernetes scheduler to provide Java remote debug ports (#3704)

11 months agoUpdated Kryo to 5.2.0 (#3705)
Nicholas Nezis [Sun, 1 Aug 2021 20:41:28 +0000 (16:41 -0400)] 
Updated Kryo to 5.2.0 (#3705)

11 months agoAdd ability to add Kubernetes Secrets and SecretKeyRefs (#3702)
Nicholas Nezis [Sun, 11 Jul 2021 02:25:04 +0000 (22:25 -0400)] 
Add ability to add Kubernetes Secrets and SecretKeyRefs (#3702)

11 months agoAdd support for dynamic kubernetes labels on pod and service (#3701)
Nicholas Nezis [Fri, 9 Jul 2021 01:00:44 +0000 (21:00 -0400)] 
Add support for dynamic kubernetes labels on pod and service (#3701)

11 months agoAdded support for adding Kubernetes annotations to the topology pod and service ...
Nicholas Nezis [Thu, 8 Jul 2021 13:00:41 +0000 (09:00 -0400)] 
Added support for adding Kubernetes annotations to the topology pod and service (#3699)

12 months agoAdding missing heron s3 region into kubernetes config (#3697)
Bouryu [Thu, 17 Jun 2021 20:44:15 +0000 (04:44 +0800)] 
Adding missing heron s3 region into kubernetes config (#3697)

Co-authored-by: Windham Wong <windhamwong@nva-hk.com>
12 months agoFix extra.links of heron-tracker (#3696) 0.20.4-incubating-rc1-line
choi se [Tue, 15 Jun 2021 05:14:25 +0000 (14:14 +0900)] 
Fix extra.links of heron-tracker (#3696)

13 months ago0.20.4 site changes (#3695)
Josh Fischer [Wed, 2 Jun 2021 01:12:16 +0000 (20:12 -0500)] 
0.20.4 site changes (#3695)

* links

* initial changes

* adding versioned docs; updating heron:bazel matrix

* fix property

* updating urls; making install docs more clear

* clean up versioned getting started docs

* remove duplicated text

* adding simple docker instructions

* clean up more urls, adding release article for 0.20.4-incubating

* clean up more urls, adding release article for 0.20.4-incubating

* fix typo

* grammar

* attempt to remove noindex

* fix maven command

* adding missed commands in versioned docs

13 months agoUpdate Bookkeeper to 4.13.0 and Zookeeper to 3.6.3 (#3692) 0.20.4-incubating-rc1-20210601
Nicholas Nezis [Mon, 24 May 2021 00:21:16 +0000 (20:21 -0400)] 
Update Bookkeeper to 4.13.0 and Zookeeper to 3.6.3 (#3692)

* Update to Bookkeeper and Zookeeper

* Committed this by accident

* Updating Guava to fix submit failure

* Updating BK image versions

13 months agolinks (#3694)
Josh Fischer [Sun, 23 May 2021 13:46:21 +0000 (08:46 -0500)] 
links (#3694)

14 months agoAdding disclaimers to release artifacts (#3689) ubuntu20.04-rpi-arm64 0.20.4-incubating-rc1
Josh Fischer [Wed, 28 Apr 2021 21:08:37 +0000 (16:08 -0500)] 
Adding disclaimers to release artifacts (#3689)

14 months agoUse the proper featured Zookeeper config generation script (#3688)
Nicholas Nezis [Sun, 18 Apr 2021 06:18:12 +0000 (02:18 -0400)] 
Use the proper featured Zookeeper config generation script (#3688)

* Updates to make Zookeeper configurable through the Helm chart logic

14 months agoupdatedocandlink (#3685) ubuntu20.04-arm64
bed debug [Tue, 6 Apr 2021 19:12:25 +0000 (12:12 -0700)] 
updatedocandlink (#3685)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
15 months agoUpdated to Kryo 5.0.4 (#3683)
Nicholas Nezis [Fri, 19 Mar 2021 01:46:39 +0000 (21:46 -0400)] 
Updated to Kryo 5.0.4 (#3683)

15 months agoFixing incorrect article title (#3682)
Josh Fischer [Mon, 15 Mar 2021 04:18:38 +0000 (23:18 -0500)] 
Fixing incorrect article title (#3682)

15 months agoGrammar fix (#3681)
fredster33 [Wed, 10 Mar 2021 02:34:46 +0000 (18:34 -0800)] 
Grammar fix (#3681)

15 months agoRemove cppunit-dev dependency (#3680)
bed debug [Tue, 9 Mar 2021 08:47:27 +0000 (00:47 -0800)] 
Remove cppunit-dev dependency (#3680)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
16 months agoDecrease the minimum requirement of RAM of single instance to 16MB in Round Robin...
Windham Wong [Sat, 27 Feb 2021 20:21:18 +0000 (04:21 +0800)] 
Decrease the minimum requirement of RAM of single instance to 16MB in Round Robin algorithm (#3678)

Co-authored-by: Windham Wong <windham.wong@stormeye.io>
16 months agoRemoving unused Bookkeeper hostPort that causes port conflict (#3677)
Nicholas Nezis [Fri, 19 Feb 2021 03:05:05 +0000 (22:05 -0500)] 
Removing unused Bookkeeper hostPort that causes port conflict (#3677)

16 months agoSpecifying explicit transitive dependency for uvloop due to Python version conflict...
Nicholas Nezis [Thu, 18 Feb 2021 23:25:20 +0000 (18:25 -0500)] 
Specifying explicit transitive dependency for uvloop due to Python version conflict (#3675)

Adding a temporary fix until uvicorn fix is released

16 months agoFixed website style issue (#3673)
Josh Fischer [Wed, 3 Feb 2021 05:28:04 +0000 (23:28 -0600)] 
Fixed website style issue (#3673)

17 months agoJoshfischer/add download links (#3671)
Josh Fischer [Mon, 1 Feb 2021 14:54:48 +0000 (08:54 -0600)] 
Joshfischer/add download links (#3671)

* initial addition of heron download page.  Still work to do.

* adding start to release page; adding updates for versioned docs

* adding versioned docs

17 months agoReverting Glog back to 0.3.5 (#3672)
Nicholas Nezis [Mon, 1 Feb 2021 13:40:59 +0000 (08:40 -0500)] 
Reverting Glog back to 0.3.5 (#3672)

17 months agoUse Build Kit docker feature to shrink the Docker image size (#3669)
Nicholas Nezis [Wed, 27 Jan 2021 04:52:37 +0000 (23:52 -0500)] 
Use Build Kit docker feature to shrink the Docker image size (#3669)

17 months agoChanged heartbeat log statement to debug (#3667)
Nicholas Nezis [Fri, 22 Jan 2021 05:51:47 +0000 (00:51 -0500)] 
Changed heartbeat log statement to debug (#3667)

17 months agoRemoving defaulted HeapDumpOnOutOfMemory (#3665)
Nicholas Nezis [Fri, 22 Jan 2021 05:49:05 +0000 (00:49 -0500)] 
Removing defaulted HeapDumpOnOutOfMemory (#3665)

17 months agoAdded CLI submit config toggle for verbose GC logging (#3663)
Nicholas Nezis [Thu, 21 Jan 2021 05:33:27 +0000 (00:33 -0500)] 
Added CLI submit config toggle for verbose GC logging (#3663)

17 months agoAdd sum/max/min reducers in Scala (#3133)
Ning Wang [Wed, 20 Jan 2021 08:42:03 +0000 (00:42 -0800)] 
Add sum/max/min reducers in Scala (#3133)

17 months agoUpdated K8s Scheduler to set Resource Limits and Requests (#3664)
Nicholas Nezis [Mon, 18 Jan 2021 17:26:57 +0000 (12:26 -0500)] 
Updated K8s Scheduler to set Resource Limits and Requests (#3664)

What was previously used for K8s request is now used as a container limit.
There is an optional setting to either set or not set the K8s request to the same value.

17 months agoexplicit unreleased code (#3658)
Josh Fischer [Tue, 12 Jan 2021 01:18:58 +0000 (19:18 -0600)] 
explicit unreleased code (#3658)

17 months agoAllow localhost graceful shutdown (#3650)
choi se [Mon, 11 Jan 2021 22:14:25 +0000 (07:14 +0900)] 
Allow localhost graceful shutdown (#3650)

* Allow localhost graceful shutdown

* Update heron/shell/src/python/handlers/killexecutorhandler.py

Co-authored-by: Oliver Bristow <evilumbrella+github@gmail.com>
Co-authored-by: Oliver Bristow <evilumbrella+github@gmail.com>
17 months agoGLog updated to 0.4.0. Also some C++ code cleanup to remove warnings (#3655)
Nicholas Nezis [Thu, 7 Jan 2021 05:46:05 +0000 (00:46 -0500)] 
GLog updated to 0.4.0. Also some C++ code cleanup to remove warnings (#3655)

17 months agoUpdated Kryo to 5.0.3 (#3657)
Nicholas Nezis [Wed, 6 Jan 2021 19:20:25 +0000 (14:20 -0500)] 
Updated Kryo to 5.0.3 (#3657)

17 months agoupdate notice year (#3656)
Josh Fischer [Wed, 6 Jan 2021 00:43:07 +0000 (18:43 -0600)] 
update notice year (#3656)

17 months agoImproving Kubernetes scheduler logic (#3653)
Nicholas Nezis [Mon, 4 Jan 2021 01:59:50 +0000 (20:59 -0500)] 
Improving Kubernetes scheduler logic (#3653)

* Added support for HTTP_NOT_FOUND response code
* Updated to use try-with-resources logic for Response cleanup
* More cleanup. Now throwing TopologyRuntimeManagementException in more places

18 months agoBumping Bazel to 3.7.2 (#3654)
Nicholas Nezis [Thu, 31 Dec 2020 07:14:40 +0000 (02:14 -0500)] 
Bumping Bazel to 3.7.2 (#3654)

18 months agoAdd Prometheus metrics rules (#3651)
choi se [Mon, 21 Dec 2020 04:05:59 +0000 (13:05 +0900)] 
Add Prometheus metrics rules (#3651)

18 months agoUpdated Heron UI with Apache compliant font (#3648) release/0.20.3-incubating-rc9 0.20.3-incubating-rc9
Josh Fischer [Wed, 16 Dec 2020 01:02:10 +0000 (19:02 -0600)] 
Updated Heron UI with Apache compliant font (#3648)

18 months agoSupport javadoc of jdk11 (#3649)
choi se [Sat, 12 Dec 2020 22:57:51 +0000 (07:57 +0900)] 
Support javadoc of jdk11 (#3649)

Co-authored-by: thinker0 <thinker0@linecorp.com>
18 months agofix year in notice (#3647)
Josh Fischer [Wed, 9 Dec 2020 12:17:05 +0000 (06:17 -0600)] 
fix year in notice (#3647)

18 months agoFix vagrant init (#3643)
Oliver Bristow [Thu, 3 Dec 2020 13:43:36 +0000 (13:43 +0000)] 
Fix vagrant init (#3643)

19 months agoAdd license to shell.snip.html (#3641)
Nicholas Nezis [Wed, 2 Dec 2020 16:51:27 +0000 (11:51 -0500)] 
Add license to shell.snip.html (#3641)

* Converting comments to jinja2 format

19 months agoTypo fix (#3637) release/0.20.3-incubating-rc8 0.20.3-incubating-rc8
Nicholas Nezis [Fri, 27 Nov 2020 19:17:37 +0000 (14:17 -0500)] 
Typo fix (#3637)

19 months agoFix for large int that Helm was changing to scientific notation (#3635)
Nicholas Nezis [Mon, 23 Nov 2020 23:18:54 +0000 (18:18 -0500)] 
Fix for large int that Helm was changing to scientific notation (#3635)

19 months agoFixes to heron-ui arg name changes (#3634)
Nicholas Nezis [Mon, 23 Nov 2020 23:18:04 +0000 (18:18 -0500)] 
Fixes to heron-ui arg name changes (#3634)

19 months agocorrect reference to values in yaml file (#3632)
Josh Fischer [Fri, 20 Nov 2020 13:15:27 +0000 (07:15 -0600)] 
correct reference to values in yaml file (#3632)

19 months agoApache Rat additional excludes rules added (#3631)
Josh Fischer [Wed, 18 Nov 2020 08:39:56 +0000 (02:39 -0600)] 
Apache Rat additional excludes rules added (#3631)

* adding website2 folder and travis toolchain bazel rule to .rat_excludes

* adding missed directory for generated site docs

19 months agoUpgrade to Bazel 3.7 (#3630)
Nicholas Nezis [Mon, 16 Nov 2020 16:43:54 +0000 (11:43 -0500)] 
Upgrade to Bazel 3.7 (#3630)

19 months agoUpdate travis-ci.com in README (#3629)
bed debug [Mon, 16 Nov 2020 01:28:32 +0000 (17:28 -0800)] 
Update travis-ci.com in README (#3629)

19 months agoClean heron-ui backend (#3597)
Oliver Bristow [Wed, 11 Nov 2020 07:14:04 +0000 (07:14 +0000)] 
Clean heron-ui backend (#3597)

* Clean heron-ui backend

 * replace tornado server with FastAPI+Jinja2+Uvicorn
 * factor out torado from heron.tools.common
 * add type annotations
 * expose port 8889 of Vagrant VM for heron-ui
 * pin to latest 4.x version of Tornado
 * visual changes to container pages

This should only be an internal refactor.

Async querying of the tracker was dropped, but could be reintroduced
using [aiohttp](https://docs.aiohttp.org/en/stable/) if speed becomes
an issue for requests that aggregate tracker data.

* Remove duplicate tracker process

19 months agoRename "topology master" to "topology manager" (#3624)
Jim Rath [Fri, 6 Nov 2020 05:41:44 +0000 (00:41 -0500)] 
Rename "topology master" to "topology manager" (#3624)

19 months agoUse click instead of argparse in py instance (#3598)
Oliver Bristow [Tue, 3 Nov 2020 15:33:38 +0000 (15:33 +0000)] 
Use click instead of argparse in py instance (#3598)

* Use click instead of argparse in py instance
* Improve heron-python-instance option name consistency
* Rename instance/st_heron_instance.py to instance.py to avoid [this kind of runtime
warning](https://stackoverflow.com/questions/43393764/python-3-6-project-structure-leads-to-runtimewarning).

19 months agoPurge heron-admin (#3585)
Oliver Bristow [Tue, 3 Nov 2020 15:12:20 +0000 (15:12 +0000)] 
Purge heron-admin (#3585)

* remove heron-admin binary from release
 * remove nomad binary from release
 * remove "standalone" cluster config from release and documentation

19 months agoUpdating to Bazel 3.5.0 (#3627)
Nicholas Nezis [Tue, 3 Nov 2020 05:35:04 +0000 (00:35 -0500)] 
Updating to Bazel 3.5.0 (#3627)

19 months agoAdded more Helm chart Bookkeeper config items (#3628)
Nicholas Nezis [Tue, 3 Nov 2020 05:32:39 +0000 (00:32 -0500)] 
Added more Helm chart Bookkeeper config items (#3628)

20 months agoclean up site docs (#3626)
Josh Fischer [Sat, 31 Oct 2020 14:29:39 +0000 (09:29 -0500)] 
clean up site docs (#3626)

* clean up

* Update website2/docs/topology-development-streamlet-api.md

Co-authored-by: Oliver Bristow <evilumbrella+github@gmail.com>
* clean up duplicates

* clean up

* last version clean up

* commit before reset

* adding version to link

Co-authored-by: Oliver Bristow <evilumbrella+github@gmail.com>
20 months agoClean tracker top level code (#3602)
Oliver Bristow [Sat, 31 Oct 2020 00:39:27 +0000 (00:39 +0000)] 
Clean tracker top level code (#3602)

 * Allow arbitrary whitespace in queries
 * DRY up Multiply/Divide/Subtract
 * use snake_case
 * code simplification

20 months agoAdding python3-venv to Debian image (#3625)
Josh Fischer [Fri, 30 Oct 2020 18:57:51 +0000 (13:57 -0500)] 
Adding python3-venv to Debian image (#3625)

20 months agoUse click instead of argparse in tracker (#3599)
Oliver Bristow [Fri, 30 Oct 2020 18:56:26 +0000 (18:56 +0000)] 
Use click instead of argparse in tracker (#3599)

20 months agoUpdated Kryo to 5.0.0 (#3621)
Nicholas Nezis [Mon, 19 Oct 2020 01:08:13 +0000 (21:08 -0400)] 
Updated Kryo to 5.0.0 (#3621)

20 months agoFix documentation CI build process (#3620)
bed debug [Fri, 16 Oct 2020 06:59:35 +0000 (23:59 -0700)] 
Fix documentation CI build process (#3620)

Co-authored-by: Huijun Wu <huijunw@twitter.com>
21 months agoAdded ability to disable base_url in Helm chart (#3619)
Nicholas Nezis [Wed, 23 Sep 2020 22:55:24 +0000 (18:55 -0400)] 
Added ability to disable base_url in Helm chart (#3619)

21 months agoStop using "cluster admin" role and restrict to k8s namespace (#3618)
Scott [Mon, 21 Sep 2020 04:00:55 +0000 (00:00 -0400)] 
Stop using "cluster admin" role and restrict to k8s namespace (#3618)

21 months agoAdding missing ENV variables for Centos 7 image (#3617)
Nicholas Nezis [Mon, 14 Sep 2020 21:14:44 +0000 (17:14 -0400)] 
Adding missing ENV variables for Centos 7 image (#3617)

21 months agoFixing typo in Helm chart template
Nicholas Nezis [Sat, 12 Sep 2020 07:25:48 +0000 (03:25 -0400)] 
Fixing typo in Helm chart template

21 months agoFix Helm chart to support clustered Zookeepers (#3611)
Nicholas Nezis [Tue, 8 Sep 2020 02:49:57 +0000 (22:49 -0400)] 
Fix Helm chart to support clustered Zookeepers (#3611)

22 months agoUpdating Kryo to 5.0.0-RC9 (#3593) roundrobin-improvement
Nicholas Nezis [Mon, 24 Aug 2020 19:20:25 +0000 (15:20 -0400)] 
Updating Kryo to 5.0.0-RC9 (#3593)

22 months agoDockerfiles Python 3 cleanup (#3601)
Nicholas Nezis [Fri, 21 Aug 2020 21:21:17 +0000 (17:21 -0400)] 
Dockerfiles Python 3 cleanup (#3601)

22 months agoUpdated to fix Kubernetes scale issue (#3607)
Nicholas Nezis [Fri, 21 Aug 2020 19:14:51 +0000 (15:14 -0400)] 
Updated to fix Kubernetes scale issue (#3607)

22 months agoUnvendor (#3603)
Oliver Bristow [Fri, 14 Aug 2020 19:03:37 +0000 (20:03 +0100)] 
Unvendor (#3603)

* Unvendor kashmir

* Unvendor cpplint

* Fix cpplint issues

22 months agoAdding Bazel version check (#3605)
Josh Fischer [Fri, 14 Aug 2020 17:45:01 +0000 (12:45 -0500)] 
Adding Bazel version check (#3605)