6 years agoFALCON-1969 Provide server-side error details on CLI, if any
yzheng-hortonworks [Mon, 23 May 2016 19:22:00 +0000 (12:22 -0700)] 
FALCON-1969 Provide server-side error details on CLI, if any

Here is a sample CLI output for comparison.

_[ambari-qasandbox falcon-0.10-SNAPSHOT]$ bin/falcon extension -submit -extensionName hdfs-mirroring -file /tmp/falcon-xml/hdfs-mirror-para.txt
Hadoop is installed, adding hadoop classpath to falcon classpath
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
ERROR: Internal Server Error;_

_[ambari-qasandbox falcon-0.10-SNAPSHOT]$ bin/falcon extension -submit -extensionName hdfs-mirroring -file /tmp/falcon-xml/hdfs-mirror-para.txt
Hadoop is installed, adding hadoop classpath to falcon classpath
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
ERROR: Internal Server Error;**Missing extension property: jobName**_

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>"

Closes #148 from yzheng-hortonworks/FALCON-1969 and squashes the following commits:

2956e89 [yzheng-hortonworks] show error details on CLI
7a76b4a [yzheng-hortonworks] FALCON-1969 Provide server-side error details on CLI, if any

6 years agoFALCON-1963 Falcon CLI should provide detailed hints if the user's command is invalid
yzheng-hortonworks [Mon, 23 May 2016 19:19:31 +0000 (12:19 -0700)] 
FALCON-1963 Falcon CLI should provide detailed hints if the user's command is invalid

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>"

Closes #147 from yzheng-hortonworks/FALCON-1963

6 years agoFALCON-1937 Cluster update documentation
bvellanki [Fri, 20 May 2016 22:14:29 +0000 (15:14 -0700)] 
FALCON-1937 Cluster update documentation

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Ying Zheng <yzheng@hortonworks.com>"

Closes #153 from bvellanki/FALCON-1937

6 years agoFALCON-1962 Extesnion related bugs
Sowmya Ramesh [Fri, 20 May 2016 20:27:26 +0000 (13:27 -0700)] 
FALCON-1962 Extesnion related bugs

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Venkat <n.r.v@live.com>"

Closes #155 from sowmyaramesh/master

6 years agoFALCON-1972 Handling cases when Extension service or extension.store.uri is not set...
Sowmya Ramesh [Fri, 20 May 2016 17:06:50 +0000 (10:06 -0700)] 
FALCON-1972 Handling cases when Extension service or extension.store.uri is not set in startup properties"

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>"

Closes #152 from sowmyaramesh/FALCON-1972

6 years agoFALCON-1974 Cluster update : Allow superuser to update bundle/coord of dependent...
bvellanki [Fri, 20 May 2016 16:37:51 +0000 (09:37 -0700)] 
FALCON-1974 Cluster update : Allow superuser to update bundle/coord of dependent entities

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Ying Zheng  <yzheng@hortonworks.com>"

Closes #151 from bvellanki/FALCON-1974

6 years agoFALCON-1975 Getting NoSuchMethodError when calling isNoneEmpty
yzheng-hortonworks [Fri, 20 May 2016 15:18:50 +0000 (20:48 +0530)] 
FALCON-1975 Getting NoSuchMethodError when calling isNoneEmpty

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: Balu <bvellanki@hortonworks.com>, Venkat <n.r.v@live.com>, Praveen <adlakha.praveen@gmail.com>, Peeyush <peeyushb@apache.org>

Closes #154 from yzheng-hortonworks/FALCON-1975

6 years agoFALCON-1973 Falcon build failure due to checkstyle issue
Peeyush [Thu, 19 May 2016 12:33:23 +0000 (18:03 +0530)] 
FALCON-1973 Falcon build failure due to checkstyle issue

Please review the following pull request in which Falcon build failure is happening with checkstyle issue from the FALCON-1964 patch.

Author: peeyush b <pbishnoi@hortonworks.com>

Reviewers: Pavan <pavan.kolamuri@gmail.com>

Closes #149 from peeyushb/FALCON-1973

6 years agoFALCON-1853 : Add spark process workflow builder
peeyush b [Thu, 19 May 2016 05:19:13 +0000 (10:49 +0530)] 
FALCON-1853 : Add spark process workflow builder

Following is the pull request that will perform the integration to execute the Spark application through Falcon. Please review.

XSD changes for integration are already committed through pull request #89 .

Author: peeyush b <pbishnoi@hortonworks.com>

Reviewers: "Pavan Kumar Kolamuri <pavan.kolamuri@gmail.com> , Venkat Ranganathan <vranganathan@hortonworks.com>"

Closes #118 from peeyushb/FALCON-1853

6 years agoFALCON-1961 Should return error if an extension job doesn't exist for delete/suspend...
Ying Zheng [Wed, 18 May 2016 22:45:41 +0000 (15:45 -0700)] 
FALCON-1961 Should return error if an extension job doesn't exist for delete/suspend/resume/schedule operations

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Sowmya Ramesh <sowmya_kr@apache.org>, Balu Vellanki <balu@apache.org>"

Closes #144 from yzheng-hortonworks/FALCON-1961

6 years agoFALCON-1964 Should delete temporary JKS file after IT tests for credential provider...
yzheng-hortonworks [Wed, 18 May 2016 20:37:56 +0000 (13:37 -0700)] 
FALCON-1964 Should delete temporary JKS file after IT tests for credential provider alias

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Sowmya Ramesh <sramesh@hortonworks.com>, Balu Vellanki <balu@apache.org>"

Closes #146 from yzheng-hortonworks/FALCON-1964

6 years agoFALCON-1943 Extension API/CLI fails when authorization is enabled
Sowmya Ramesh [Tue, 17 May 2016 21:40:32 +0000 (14:40 -0700)] 
FALCON-1943 Extension API/CLI fails when authorization is enabled

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>, Ying Zheng <yzheng@hortonworks.com>"

Closes #145 from sowmyaramesh/FALCON-1943

6 years agoFALCON-1956 Graphite Plugin for monitoring
Praveen Adlakha [Tue, 17 May 2016 11:45:04 +0000 (17:15 +0530)] 
FALCON-1956 Graphite Plugin for monitoring

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: "Pavan Kumar Kolamuri <pavan.kolamuri@gmail.com>, Srikanth Sundarrajan <sriksun@hotmail.com>"

Closes #141 from PraveenAdlakha/codahale

6 years agoFALCON-1957 Documentation on credential provider alias for passwords in startup prope...
yzheng-hortonworks [Mon, 16 May 2016 22:05:45 +0000 (15:05 -0700)] 
FALCON-1957 Documentation on credential provider alias for passwords in startup properties

Also restructured the overview of Falcon security features.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>, Balu Vellanki <balu@apache.org>"

Closes #142 from yzheng-hortonworks/FALCON-1957

6 years agoFALCON-1954 Document enabling Oozie JMS for Falcon
Venkatesan Ramachandran [Fri, 13 May 2016 22:47:13 +0000 (15:47 -0700)] 
FALCON-1954 Document enabling Oozie JMS for Falcon

Steps to enable Oozie JMS notification for Falcon.

Author: Venkatesan Ramachandran <vramachandran@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>, Balu Vellanki <balu@apache.org>"

Closes #137 from vramachan/FALCON-1948.OozieForFalcon

6 years agoFALCON-1934 Document Safemode in Falcon Server
bvellanki [Fri, 13 May 2016 22:43:59 +0000 (15:43 -0700)] 
FALCON-1934 Document Safemode in Falcon Server

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Ying Zheng <yzheng@hortonworks.com>, Venkatesan Ramachandran <vramachandran@hortonworks.com>"

Closes #140 from bvellanki/FALCON-1934

6 years agoFALCON-1919 Provide user the option to store sensitive information with Hadoop creden...
yzheng-hortonworks [Fri, 13 May 2016 22:22:07 +0000 (15:22 -0700)] 
FALCON-1919 Provide user the option to store sensitive information with Hadoop credential provider

Tested alias properties being resolved correctly with Hadoop credential provider.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>, Venkat Ranganathan <venkat@hortonworks.com>"

Closes #138 from yzheng-hortonworks/FALCON-1919

6 years agoFALCON-1908 Document HDFS snapshot based mirroring extension
bvellanki [Fri, 13 May 2016 16:09:41 +0000 (09:09 -0700)] 
FALCON-1908 Document HDFS snapshot based mirroring extension

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Ying Zheng <yzheng@hortonworks.com>"

Closes #139 from bvellanki/FALCON-1908

6 years agoFALCON-1905 Extension Job Management: IT tests for REST APIs and CLIs
yzheng-hortonworks [Thu, 12 May 2016 18:34:37 +0000 (11:34 -0700)] 
FALCON-1905 Extension Job Management: IT tests for REST APIs and CLIs

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: " Sowmya<sramesh@hortonworks.com>,Balu Vellanki <balu@apache.org>"

Closes #129 from yzheng-hortonworks/FALCON-1905

6 years agoFALCON-1926 Filter out effectively non-falcon related JMS messages …
Venkatesan Ramachandran [Thu, 12 May 2016 01:09:16 +0000 (18:09 -0700)] 
FALCON-1926 Filter out effectively non-falcon related JMS messages …

* Falcon to retrieve Oozie JMS notifications that belong to Falcon generated Workflows .. ignoring notifications generated by other WF directly submitted to Oozie.

Author: Venkatesan Ramachandran <vramachandran@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>, Ying Zheng <yzheng@hortonworks.com>"

Closes #119 from vramachan/FALCON-1923.OpsCoord

6 years agoFALCON-1953 Build fails when profiles hivedr and test-patch are used together
bvellanki [Thu, 12 May 2016 00:04:25 +0000 (17:04 -0700)] 
FALCON-1953 Build fails when profiles hivedr and test-patch are used together

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>, Venkatesan <me.venkatr@gmail.com>"

Closes #136 from bvellanki/FALCON-1953

6 years agoFALCON-1946 Compile time error in build when using JDK1.8
bvellanki [Wed, 11 May 2016 15:55:38 +0000 (08:55 -0700)] 
FALCON-1946 Compile time error in build when using JDK1.8

Author: bvellanki <bvellanki@hortonworks.com>

Closes #135 from bvellanki/FALCON-1946

6 years agoFALCON-1904 Extension Job Management: documentation for REST APIs and CLIs
yzheng-hortonworks [Wed, 11 May 2016 04:36:32 +0000 (21:36 -0700)] 
FALCON-1904 Extension Job Management: documentation for REST APIs and CLIs

Added documentation for all REST APIs and CLIs for extension job management.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>, Sowmya Ramesh <sramesh@hortonworks.com>"

Closes #131 from yzheng-hortonworks/FALCON-1904

6 years agoFALCON-1942 Allow Falcon server and client classpath to be customizable
Venkat Ranganathan [Tue, 10 May 2016 21:51:05 +0000 (14:51 -0700)] 
FALCON-1942 Allow Falcon server and client classpath to be customizable

Allow server and client classpath to be customizable

Author: Venkat Ranganathan <venkat@hortonworks.com>

Reviewers: "Ying Zheng <yzheng@hortonworks.com>, Balu Vellanki <balu@apache.org>"

Closes #134 from vrangan/FALCON-1942

6 years agoFALCON-1941 HiveDR fails with NN-HA enabled on both the source and target clusters
Venkat Ranganathan [Tue, 10 May 2016 21:50:02 +0000 (14:50 -0700)] 
FALCON-1941 HiveDR fails with NN-HA enabled on both the source and target clusters

Fix for HiveDR to work NN-HA enabled clusters

Author: Venkat Ranganathan <venkat@hortonworks.com>

Reviewers: "Ying Zheng <yzheng@hortonworks.com>, Balu Vellanki <balu@apache.org>"

Closes #133 from vrangan/FALCON-1941

6 years agoFALCON-1940 Fix misc build issues in Falcon
bvellanki [Tue, 10 May 2016 20:29:38 +0000 (13:29 -0700)] 
FALCON-1940 Fix misc build issues in Falcon

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>"

Closes #132 from bvellanki/FALCON-1940

6 years agoFALCON-1085 Support Cluster entity updates in Falcon Server
bvellanki [Tue, 10 May 2016 20:27:33 +0000 (13:27 -0700)] 
FALCON-1085 Support Cluster entity updates in Falcon Server

Added basic documentation, https://issues.apache.org/jira/browse/FALCON-1937 will contain detailed documentation.

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>, yzheng-hortonworks <yzheng@hortonworks.com>, peeyush b <pbishnoi@hortonworks.com>"

Closes #127 from bvellanki/FALCON-1085

6 years agoFALCON-1939 Avoid creating multiple falcon*.tar.gz during falcon build
bvellanki [Mon, 9 May 2016 23:31:29 +0000 (16:31 -0700)] 
FALCON-1939 Avoid creating multiple falcon*.tar.gz during falcon build

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: Venkat Ranganathan <venkat@hortonworks.com>

Closes #130 from bvellanki/FALCON-1939

6 years agoFALCON-1932 Extension CLI should support common options
yzheng-hortonworks [Thu, 5 May 2016 22:55:41 +0000 (15:55 -0700)] 
FALCON-1932 Extension CLI should support common options

Tested CLI command with common options.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>, Sowmya Ramesh <sowmya_kr@apache.org>"

Closes #124 from yzheng-hortonworks/FALCON-1932

6 years agoFALCON-1936 Extensions related files are not available in distro packaging
Sowmya Ramesh [Thu, 5 May 2016 18:25:30 +0000 (11:25 -0700)] 
FALCON-1936 Extensions related files are not available in distro packaging

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>"

Closes #125 from sowmyaramesh/FALCON-1936_1

6 years agoFALCON-1106 Documentation for extensions
Sowmya Ramesh [Tue, 3 May 2016 21:45:40 +0000 (14:45 -0700)] 
FALCON-1106 Documentation for extensions

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>", Ying Zheng <yzheng@hortonworks.com>"

Closes #120 from sowmyaramesh/FALCON-1106

6 years agoFALCON-1767 Improve Falcon retention policy documentation
["Sowmya Ramesh [Tue, 3 May 2016 21:21:31 +0000 (14:21 -0700)] 
FALCON-1767 Improve Falcon retention policy documentation

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>"

Closes #121 from sowmyaramesh/FALCON-1767

6 years agoFALCON-1623 Implement safemode in Falcon Server
bvellanki [Tue, 3 May 2016 18:23:43 +0000 (11:23 -0700)] 
FALCON-1623 Implement safemode in Falcon Server

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "yzheng-hortonworks <yzheng@hortonworks.com>, Venkat Ranganathan <venkat@hortonworks.com>, Sowmya Ramesh <sowmya_kr@apache.org>"

Closes #116 from bvellanki/FALCON-1623

6 years agoFALCON-1928 Updating default startup.properties to add FalconJPAService
Pallavi Rao [Thu, 28 Apr 2016 10:42:07 +0000 (16:12 +0530)] 
FALCON-1928 Updating default startup.properties to add FalconJPAService

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: @sandeepSamudrala, @peeyushb

Closes #115 from pallavi-rao/1928

6 years ago89
Peeyush Bishnoi [Wed, 27 Apr 2016 16:02:27 +0000 (21:32 +0530)] 

Pull request is available that will extend the cluster and process XSD to include the spark execution engine.

Author: peeyush b <pbishnoi@hortonworks.com>

Reviewers: Pallavi <pallavi@inmobi.com>, Pavan <pavan.kolamuri@gmail.com>, Venkat <n.r.v@live.com>, Venkatesan <me.venkatr@gmail.com>

Closes #89 from peeyushb/FALCON-1817 and squashes the following commits:

64d8c35 [peeyush b] making name as required parameter
b26af4e [peeyush b] removed line
8056916 [peeyush b] Incorporating review comments

6 years agoFALCON-1909 Catalog instance triage action fails with null pointer exception
bvellanki [Wed, 27 Apr 2016 07:03:34 +0000 (00:03 -0700)] 
FALCON-1909 Catalog instance triage action fails with null pointer exception


Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: " Praveen Adlakha <adlakha.praveen@gmail.com>, yzheng-hortonworks <yzheng@hortonworks.com>, Venkat Ranganathan <venkat@hortonworks.com>"

Closes #117 from bvellanki/FALCON-1909

6 years agoFALCON-1899 Create examples artifact module in Falcon
Peeyush Bishnoi [Sat, 23 Apr 2016 04:53:44 +0000 (10:23 +0530)] 
FALCON-1899 Create examples artifact module in Falcon

Author: peeyush b <pbishnoi@hortonworks.com>

Reviewers: Balu <bvellanki@hortonworks.com>, Venkatesan <me.venkatr@gmail.com>, Venkat Ranganathan <n.r.v@live.com>

Closes #100 from peeyushb/FALCON-1899 and squashes the following commits:

0824204 [peeyush b] changed examples path in assembly-standalone.xml
cdcb3cd [peeyush b] removed extra line
26bc4f8 [peeyush b] FALCON-1899 : Create examples artifact module in Falcon

6 years agoFALCON-1924 Falcon Coordinator rerun return old workflow id
Praveen Adlakha [Fri, 22 Apr 2016 16:27:00 +0000 (21:57 +0530)] 
FALCON-1924 Falcon Coordinator rerun return old workflow id

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Pallavi<pallavi.rao@inmobi.com>, Sandeep<sandysmdl@gmail.com>, Peeyush<peeyushb@apache.org>

Closes #113 from PraveenAdlakha/1924

6 years agoFALCON-1897 Extension Job Management: CLI support
yzheng-hortonworks [Fri, 22 Apr 2016 16:04:48 +0000 (09:04 -0700)] 
FALCON-1897 Extension Job Management: CLI support

Note: There is a bug to run Falcon CLI after the code refactoring. sowmyaramesh has the patch to fix this and will send a pull request. Tested all the CLI methods with her fix patch.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>"

Closes #103 from yzheng-hortonworks/FALCON-1897

6 years agoFALCON-1925 Add hadoop classpath to falcon client classpath in falcon-config.sh
Venkat Ranganathan [Fri, 22 Apr 2016 14:30:30 +0000 (20:00 +0530)] 
FALCON-1925 Add hadoop classpath to falcon client classpath in falcon-config.sh

This is for recipe submission in a secure cluster

Author: Venkat Ranganathan <venkat@hortonworks.com>
Author: Sowmya Ramesh <sowmya_kr@apache.org>
Author: Praveen Adlakha <adlakha.praveen@gmail.com>
Author: bvellanki <bvellanki@hortonworks.com>
Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: Peeyush<peeyushb@apache.org>, Sandeep<sandysmdl@gmail.com>

Closes #112 from vrangan/master and squashes the following commits:

66907d8 [Venkat Ranganathan] Merge remote-tracking branch 'upstream/master'
4025644 [Venkat Ranganathan] FALCON-1925: Add hadoop classpath to falcon client classpath in falcon-config.sh
608efeb [bvellanki] FALCON-1861 Support HDFS Snapshot based replication in Falcon
9f7fa4b [Sowmya Ramesh] FALCON-1105 REST API support for Server side extensions artifact repository management
ac6dd92 [yzheng-hortonworks] FALCON-1790 CLI support for instance search
4c50d8d [Praveen Adlakha] FALCON-1749 Instance status does not show instances if entity is dele…
157a4f7 [Venkat Ranganathan] Fix for allowing RM principal to be specified in cluster entity
5edcc8f [Venkat Ranganathan] Merge remote-tracking branch 'upstream/master'
bc5a458 [Venkat Ranganathan] Falcon webUI returns 413 (Full head - Request entity too large) error when TLS is enabled in a secure cluster with AD integration

6 years agoFALCON-1916 Allow RM principal to be specified in Cluster entity
Venkat Ranganathan [Thu, 21 Apr 2016 23:50:44 +0000 (16:50 -0700)] 
FALCON-1916 Allow RM principal to be specified in Cluster entity

When we have clusters setup with individual security domains with  cross realm trusts, the RM principal needs to be specified explicitly to disambiguate the realm of the resource manager of the corresponding cluster entities.

This patch allows RM principal to be set and processed from the cluster properties

Author: Venkat Ranganathan <venkat@hortonworks.com>

Reviewers: "Venkatesan Ramachandran <vramachandran@hortonworks.com>"

Closes #111 from vrangan/master

6 years agoFALCON-1861 Support HDFS Snapshot based replication in Falcon
bvellanki [Thu, 21 Apr 2016 23:48:16 +0000 (16:48 -0700)] 
FALCON-1861 Support HDFS Snapshot based replication in Falcon

Documentation will be added in Jira FALCON-1908

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Sowmya <sramesh@hortonworks.com>, sandeepSamudrala <sandysmdl@gmail.com>, Ying Zheng <yzheng@hortonworks.com>, Venkat Ranganathan <venkat@hortonworks.com>"

Closes #105 from bvellanki/master

6 years agoFALCON-1105 REST API support for Server side extensions artifact repository management
Sowmya Ramesh [Thu, 21 Apr 2016 22:18:43 +0000 (15:18 -0700)] 
FALCON-1105 REST API support for Server side extensions artifact repository management

REST API support for Server side extensions artifact repository management.
Fixed the warnings in the file.

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Balu<bvellanki@hortonworks.com>, Ying Zheng<yzheng@hortonworks.com>"

Closes #102 from sowmyaramesh/FALCON-1105

6 years agoFALCON-1790 CLI support for instance search
yzheng-hortonworks [Thu, 21 Apr 2016 22:06:05 +0000 (15:06 -0700)] 
FALCON-1790 CLI support for instance search

Tested CLI support. Also added documentation for both REST api and CLI.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Praveen Adlakha <adlakha.praveen@gmail.com>, Balu Vellanki <balu@apache.org>"

Closes #110 from yzheng-hortonworks/FALCON-1790

6 years agoFALCON-1749 Instance status does not show instances if entity is dele…
Praveen Adlakha [Thu, 21 Apr 2016 10:09:36 +0000 (15:39 +0530)] 
FALCON-1749 Instance status does not show instances if entity is dele…

…ted from one of the colos

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Pallavi Rao <pallavi.rao@inmobi.com>

Closes #108 from PraveenAdlakha/1749 and squashes the following commits:

ef50c74 [Praveen Adlakha] minor fix
0cedbb8 [Praveen Adlakha] Merge branch '1749' of github.com:PraveenAdlakha/falcon into 1749
7e25db6 [Praveen Adlakha] FALCON-1749 Instance status does not show instances if entity is deleted from one of the colos

6 years agoFALCON-1335 Backend support of instance search of a group of entities
yzheng-hortonworks [Wed, 20 Apr 2016 18:49:07 +0000 (11:49 -0700)] 
FALCON-1335 Backend support of instance search of a group of entities

Tested REST API: api/instance/search; Added unit test

p.s. Accidentally deleted the previous branch. Previous reviews: https://github.com/apache/falcon/pull/93

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Sowmya <sramesh@hortonworks.com>, Peeyush<peeyushb@apache.org>, Balu <balu@apache.org>"

Closes #106 from yzheng-hortonworks/FALCON-1335

6 years agoFALCON-1917 Fix build failure
Sowmya Ramesh [Wed, 20 Apr 2016 18:42:11 +0000 (11:42 -0700)] 
FALCON-1917 Fix build failure

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: Ying Zheng<yzheng@hortonworks.com>, Peeyush Bishnoi<peeyushb@apache.org>, Venkat Ranganathan <venkat@hortonworks.com>, Praveen Adlakha <adlakha.praveen@gmail.com>

Closes #109 from sowmyaramesh/FALCON-1917

6 years agoFALCON-1802 EL Expressions support in Native Scheduler
pavankumar526 [Tue, 19 Apr 2016 06:33:57 +0000 (12:03 +0530)] 
FALCON-1802 EL Expressions support in Native Scheduler

Author: pavankumar526 <pavan.kolamuri@gmail.com>

Reviewers: "sandeepSamudrala <sandysmdl@gmail.com>, Ajay Yadava <ajayyadava@apache.org>"

Closes #57 from pavankumar526/FALCON-1802

6 years agoFALCON-1907 Package new CLI module
Sowmya Ramesh [Mon, 18 Apr 2016 11:42:00 +0000 (17:12 +0530)] 
FALCON-1907 Package new CLI module

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: Sandeep<sandysmdl@gmail.com>, Sowmya<sramesh@hortonworks.com>, Peeyush<peeyushb@apache.org>

Closes #104 from sowmyaramesh/FALCON-1907

6 years agoFALCON-1888 Falcon JMS notification details and example
Venkatesan Ramachandran [Mon, 18 Apr 2016 08:45:39 +0000 (14:15 +0530)] 
FALCON-1888 Falcon JMS notification details and example

Updates the operability document to enable Falcon user notifications and example code to read and Map Message from ActiveMQ.

Author: Venkatesan Ramachandran <vramachandran@hortonworks.com>

Reviewers: Sowmya<sramesh@hortonworks.com>, Peeyush<peeyushb@apache.org>

Closes #101 from vramachan/master and squashes the following commits:

3429177 [Venkatesan Ramachandran] FALCON-1888 : Falcon JMS system and user notification details and sample code
766c130 [Venkatesan Ramachandran] FALCON-1888 : Falcon JMS system and user notification details and sample code
a1fe26e [Venkatesan Ramachandran] git rebase from upstream master to ramachan fork master
3cfc4e0 [Venkatesan Ramachandran] FALCON-1888 : Falcon JMS details and usage
fac28b1 [Venkatesan Ramachandran] Merge
afa5173 [Venkatesan Ramachandran] FALCON-1838:Export instances are not added graph db for lineage tracking - remove call to findVertex(), refactor method name
a886958 [Venkatesan Ramachandran] FALCON-1838:Export instances are not added graph db for lineage tracking

6 years agoFALCON-1789 Extension Job Management: REST API
Ying Zheng [Fri, 15 Apr 2016 18:46:33 +0000 (11:46 -0700)] 
FALCON-1789 Extension Job Management: REST API

Tested REST APIs. Will add IT tests (FALCON-1905) and documentations (FALCON-1904) in separate pull requests.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu <balu@apache.org>, Sowmya <sowmya_kr@apache.org>"

Closes #98 from yzheng-hortonworks/FALCON-1789

6 years agoFALCON-1902 Command line support for server side extension artifacts…
Sowmya Ramesh [Thu, 14 Apr 2016 23:44:57 +0000 (16:44 -0700)] 
FALCON-1902 Command line support for server side extension artifacts…

… repository management

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: Balu<bvellanki@hortonworks.com>, Ying Zheng<yzheng@hortonworks.com>, Peeyush Bishnoi<peeyushb@apache.org>

Closes #99 from sowmyaramesh/FALCON-1902 and squashes the following commits:

60dc1ac [Sowmya Ramesh] FALCON-1902: Incorporated review comments
642f0ae [Sowmya Ramesh] FALCON-1902: Incorpoarted review comments
d27e2db [Sowmya Ramesh] FALCON-1902: Command line support for server side extension artifacts repository management

6 years agoFALCON-1901: Fix intermittent IT test failures caused by extension co…
Sowmya Ramesh [Thu, 14 Apr 2016 01:12:11 +0000 (18:12 -0700)] 
FALCON-1901: Fix intermittent IT test failures caused by extension co…

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: Balu<bvellanki@hortonworks.com>, Ying Zheng<yzheng@hortonworks.com>

Closes #97 from sowmyaramesh/FALCON-1901

6 years agoFALCON-1895 Refactoring of FalconCLI and FalconClient
Praveen Adlakha [Wed, 13 Apr 2016 15:39:37 +0000 (21:09 +0530)] 
FALCON-1895 Refactoring of FalconCLI and FalconClient

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Peeyush Bishnoi<peeyushb@apache.org>, Balu<bvellanki@hortonworks.com>, Sowmya<sramesh@hortonworks.com>

Closes #94 from PraveenAdlakha/1895 and squashes the following commits:

00a65a2 [Praveen Adlakha] merge conflicts resolved
d069466 [Praveen Adlakha] sync from upstream
4e79efc [Praveen Adlakha] extra space removed
d8bc106 [Praveen Adlakha] FALCON-1895 Refactoring of FalconCLI and FalconClient

6 years agoFALCON-1898 Build fails on assembly:assembly when using maven-assembly-plugin version 2.6
bvellanki [Wed, 13 Apr 2016 02:15:47 +0000 (19:15 -0700)] 
FALCON-1898 Build fails on assembly:assembly when using maven-assembly-plugin version 2.6

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Venkat Ranganathan <venkat@hortonworks.com>, Venkatesan Ramachandran <vramachandran@hortonworks.com>"

Closes #96 from bvellanki/master

6 years agoFALCON-1107 Move trusted extensions processing to server side
Sowmya Ramesh [Tue, 12 Apr 2016 23:05:48 +0000 (16:05 -0700)] 
FALCON-1107 Move trusted extensions processing to server side

Ignore any documentation issues as it will be addressed in https://issues.apache.org/jira/browse/FALCON-1106. Thanks!

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>, Venkat Ranganathan <venkat@hortonworks.com>"

Closes #92 from sowmyaramesh/FALCON-1107

6 years agoFALCON-1892 Remove client side Recipe logic
Sowmya Ramesh [Tue, 12 Apr 2016 23:04:39 +0000 (16:04 -0700)] 
FALCON-1892 Remove client side Recipe logic

Please use this pull request to review. Ignore any documentation issues as it will be addressed in  https://issues.apache.org/jira/browse/FALCON-1106. Thanks!

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: "Balu Vellanki <balu@apache.org>, Venkat Ranganathan <venkat@hortonworks.com>"

Closes #91 from sowmyaramesh/FALCON-1892

6 years agoFALCON-1896 Failure in Falcon build in distro module
Praveen Adlakha [Tue, 12 Apr 2016 16:00:57 +0000 (09:00 -0700)] 
FALCON-1896 Failure in Falcon build in distro module

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: "Balu Vellanki <balu@apache.org>"

Closes #95 from PraveenAdlakha/1896

6 years agoFALCON-1885 SLA monitoring API throws ResultNotFoundException
Praveen Adlakha [Tue, 12 Apr 2016 14:44:19 +0000 (20:14 +0530)] 
FALCON-1885 SLA monitoring API throws ResultNotFoundException

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Ajay Yadava <ajayyadava@apache.org>

Closes #88 from PraveenAdlakha/1885

6 years agoFALCON-1882 Instance status api not working via prism
Praveen Adlakha [Tue, 12 Apr 2016 14:41:18 +0000 (20:11 +0530)] 
FALCON-1882 Instance status api not working via prism

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Ajay Yadava<ajayyadava@apache.org>, Pavan Kumar Kolamuri

Closes #87 from PraveenAdlakha/1882

6 years agoFALCON-1880 To support TDE encryption, add --skipcrccheck to distcp options for HiveDR.
bvellanki [Wed, 6 Apr 2016 22:16:36 +0000 (15:16 -0700)] 
FALCON-1880 To support TDE encryption, add --skipcrccheck to distcp options for HiveDR.

Add --skipcrccheck to distcp options for HiveDR.

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: "Peeyush Bishnoi <peeyushb@apache.org>,  Ajay Yadava <ajaynsit@gmail.com>, Venkatesan Ramachandran <Me.venkatr@gmail.com>"

Closes #82 from bvellanki/master

6 years agoFALCON-1881 Database export does not require fields tag in entity specification
Venkatesan Ramachandran [Wed, 6 Apr 2016 20:39:42 +0000 (13:39 -0700)] 
FALCON-1881 Database export does not require fields tag in entity specification

Database export does not require fields tag in entity specification

Author: Venkatesan Ramachandran <vramachandran@hortonworks.com>

Reviewers: "Peeyush Bishnoi <peeyushb@apache.org>, Pallavi Rao <pallavi.rao@inmobi.com>, Balu Vellanki <bvellanki@hortonworks.com>"

Closes #83 from vramachan/FALCON-1881.20160331-1.ExportFieldsList

6 years agoFALCON-1886 Feed sla monitoring does not work across restarts.
Ajay Yadava [Wed, 6 Apr 2016 10:25:18 +0000 (15:55 +0530)] 
FALCON-1886 Feed sla monitoring does not work across restarts.

Removed redundant fields.
Added unit tests.

Author: Ajay Yadava <ajaynsit@gmail.com>

Reviewers: Praveen Adlakha

Closes #86 from ajayyadava/1886

6 years agoFALCON-1845 Retries fix when entity was deleted during instance rerun
pavankumar526 [Wed, 6 Apr 2016 08:41:49 +0000 (14:11 +0530)] 
FALCON-1845 Retries fix when entity was deleted during instance rerun

Author: pavankumar526 <pavan.kolamuri@gmail.com>

Reviewers: Ajay Yadava <ajayyadava@apache.org>, Sandeep Samudrala, Pallavi Rao

Closes #62 from pavankumar526/FALCON-1845

6 years agoFALCON-1855 Falcon regression build fails with checkstyle issues
Murali Ramasami [Wed, 6 Apr 2016 08:39:15 +0000 (14:09 +0530)] 
FALCON-1855 Falcon regression build fails with checkstyle issues

Falcon checkstyle is now moved to parent pom because of which regression is failing. Fixing the path

Author: Murali Ramasami <mramasami@Murali-Ramasami-MBP.local>

Reviewers: "Pragya <mittal.pragya23@gmail.com>, Praveen Adlakha <adlakha.praveen@gmail.com>"

Closes #84 from muraliramasami/FALCON-1855

6 years agoFALCON-1883 Falcon regression build fails with minor checkstyle issues
Murali Ramasami [Wed, 6 Apr 2016 08:35:18 +0000 (14:05 +0530)] 
FALCON-1883 Falcon regression build fails with minor checkstyle issues

Fixing the checkstyle issue observed during building the regression trunk

Author: Murali Ramasami <mramasami@Murali-Ramasami-MBP.local>

Reviewers: "Pragya <mittal.pragya23@gmail.com>, Praveen Adlakha <adlakha.praveen@gmail.com>"

Closes #85 from muraliramasami/FALCON-1883

6 years agoFALCON-1866 Bug in JDBCStateStore
Praveen Adlakha [Wed, 6 Apr 2016 08:15:12 +0000 (13:45 +0530)] 
FALCON-1866 Bug in JDBCStateStore

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Ajay Yadava <ajayyadava@apache.org>, Pavan Kumar Kolamuri

Closes #80 from PraveenAdlakha/1866

6 years agoFALCON-1856 Fix ProcessInstanceStatusTest
Pragya [Mon, 4 Apr 2016 10:42:45 +0000 (16:12 +0530)] 
FALCON-1856 Fix ProcessInstanceStatusTest

Author: Pragya <mittal.pragya23@gmail.com>

Reviewers: Murali Ramasami <murali.msse@gmail.com>

Closes #68 from pragya-mittal/ProcessInstanceStatusTest

6 years agoFALCON-1854 : Fixing PrismProcessScheduleTest and NoOutputProcessTest
Murali Ramasami [Mon, 4 Apr 2016 10:21:38 +0000 (15:51 +0530)] 
FALCON-1854 : Fixing PrismProcessScheduleTest and NoOutputProcessTest

Fixing PrismProcessScheduleTest and NoOutputProcessTest

Author: Murali Ramasami <mramasami@hortonworks.com>

Reviewers: Pragya <mittal.pragya23@gmail.com>

Closes #67 from muraliramasami/master

6 years agoFalcon webUI returns 413
Venkat Ranganathan [Thu, 31 Mar 2016 21:50:56 +0000 (14:50 -0700)] 
Falcon webUI returns 413

 (Full head - Request entity too large) error when TLS is enabled in a secure cluster with AD integration)

Fix is to duplicate the SocketConnector change to increase header and request buffer for Jetty based on startup properties

Author: Venkat Ranganathan <venkat@hortonworks.com>

Reviewers: "Ajay Yadava <ajayyadava@apache.org>"

Closes #81 from vrangan/master

6 years agoFALCON-1867 Hardcoded query names in JDBCStateStore
Praveen Adlakha [Tue, 29 Mar 2016 14:19:45 +0000 (19:49 +0530)] 
FALCON-1867 Hardcoded query names in JDBCStateStore

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Ajay Yadava <ajayyadava@apache.org>

Closes #79 from PraveenAdlakha/1867

6 years agoFALCON-1865 Persist Feed sla data to database
Praveen Adlakha [Mon, 28 Mar 2016 13:38:56 +0000 (19:08 +0530)] 
FALCON-1865 Persist Feed sla data to database

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Ajay Yadava <ajayyadava@apache.org>

Closes #77 from PraveenAdlakha/feed_alert

6 years agoFALCON-1874 Import and Export fails with HDFS as src/dest
Pallavi Rao [Mon, 28 Mar 2016 06:43:32 +0000 (12:13 +0530)] 
FALCON-1874 Import and Export fails with HDFS as src/dest

1. job-xml was pointing to hive-site.xml even for HDFS, cause workflow to fail.
2. datain element was repeated in export coordinator, causing workflow to fail.

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: @vramachan, @PraveenAdlakha

Closes #78 from pallavi-rao/FALCON-1874

6 years agoFALCON-1864 Retry event does not get removed from delay queue
Pallavi Rao [Mon, 28 Mar 2016 06:41:35 +0000 (12:11 +0530)] 
FALCON-1864 Retry event does not get removed from delay queue

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: @sandeepSamudrala, @pavankumar526, @PraveenAdlakha

Closes #76 from pallavi-rao/1864

6 years agoFALCON-1840 Archive older definition in case of update
Praveen Adlakha [Wed, 23 Mar 2016 15:13:12 +0000 (20:43 +0530)] 
FALCON-1840 Archive older definition in case of update

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Peeyush Bishnoi<peeyushb@apache.org>, Balu<bvellanki@hortonworks.com>, Pallavi<pallavi.rao@inmobi.com>, Sandeep<sandysmdl@gmail.com>

Closes #75 from PraveenAdlakha/1840 and squashes the following commits:

c9c19d8 [Praveen Adlakha] documentation added for archival in delete and update
6c570e3 [Praveen Adlakha] comments addressed
b475fc4 [Praveen Adlakha] IT added
7ba81af [Praveen Adlakha] FALCON-1840 Archive older definition in case of update

6 years agoFALCON-1852 Make optional input to a process truly optional
Pallavi Rao [Tue, 22 Mar 2016 07:15:40 +0000 (12:45 +0530)] 
FALCON-1852 Make optional input to a process truly optional

The following changes have been made:
1. Creation of "empty dir" under the staging path of a cluster (during cluster creation).
2. Modify OozieELExtensions to look for availability flag and use only those instances that have the entire dataset. Use "empty dir" when no instances resolve.
3. Updated UT and IT for additional validation.

Tested manually with and without availabilityFlag supplied.

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: @sandeepSamudrala , @sriksun, @pavankumar526, @ajayyadava

Closes #73 from pallavi-rao/1852 and squashes the following commits:

7e087ed [Pallavi Rao] FALCON-1852 Documentation update for optional feeds
fc9c490 [Pallavi Rao] FALCON-1852 Added mockito to the pom
92ad20c [Pallavi Rao] FALCON-1852 Added UTs
533e221 [Pallavi Rao] FALCON-1852 Make optional input to a process truly optional

6 years agoFALCON-1838 Export instances are not added graph db for lineage tracking
Venkatesan Ramachandran [Thu, 17 Mar 2016 16:01:14 +0000 (09:01 -0700)] 
FALCON-1838 Export instances are not added graph db for lineage tracking

Author: Venkatesan Ramachandran <vramachandran@hortonworks.com>

Reviewers: "Balu Vellanki <bvellanki@hortonworks.com>, Ying Zheng <yzheng@hortonworks.com>"

Closes #74 from vramachan/master

6 years agoFALCON-1860 ADFProviderService should be optional as default setting
yzheng-hortonworks [Wed, 16 Mar 2016 21:41:32 +0000 (14:41 -0700)] 
FALCON-1860 ADFProviderService should be optional as default setting

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: "Balu <bvellanki@hortonworks.com>, Venkatesan Ramachandran <Me.venkatr@gmail.com>, Sowmya Ramesh <sowmya_kr@apache.org>"

Closes #70 from yzheng-hortonworks/FALCON-1860

6 years agoFALCON-1334 Improve search performance with Titan graph database indexing
yzheng-hortonworks [Wed, 16 Mar 2016 15:09:14 +0000 (20:39 +0530)] 
FALCON-1334 Improve search performance with Titan graph database indexing

Upgraded Titan to 0.5.4. Added indexing on status and nominal time. May add more later based on our user study on frequent filters.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: Balu <bvellanki@hortonworks.com>, Peeyush Bishnoi <peeyushb@apache.org>

Closes #69 from yzheng-hortonworks/FALCON-1334 and squashes the following commits:

51d5530 [yzheng-hortonworks] keep property falcon.graph.serialize.path (needed in two tests)
7939b81 [yzheng-hortonworks] FALCON-1333 Improve search performance with Titan graph database indexing

6 years agoFALCON-1836 Import from database to HCatalog
Venkatesan Ramachandran [Mon, 14 Mar 2016 21:34:28 +0000 (14:34 -0700)] 
FALCON-1836 Import from database to HCatalog

Author: Venkatesan Ramachandran <vramachandran@hortonworks.com>

Reviewers: "Balu Vellanki <bvellanki@hortonworks.com>, Ajay Yadava <ajayyadava@apache.org>, Peeyush Bishnoi <bpeeyush@yahoo.co.in>"

Closes #61 from vramachan/master

6 years agoFALCON-1847 Execution order not honored when instances are suspended/resumed
Pallavi Rao [Mon, 14 Mar 2016 12:06:08 +0000 (17:36 +0530)] 
FALCON-1847 Execution order not honored when instances are suspended/resumed

Have made the following changes:

1. Order of instances on which we perform action is sorted in desc only for status and params. Else, older instances should be suspended/resumed first.
2. Status is now read from DB and only additional info is obtained from Oozie.
3. Listening to suspend notification too, so further instances can be scheduled.
4. Resume does not immediately resume the instance in order to honor concurrency.

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: Sandeep Samudrala <sandeep.samudrala@inmobi.com>

Closes #64 from pallavi-rao/1847 and squashes the following commits:

069c223 [Pallavi Rao] FALCON-1847 Fixing the IT
9cd1249 [Pallavi Rao] FALCON-1847 Execution order not honored when instances are suspended/resumed

6 years agoFALCON-1743 Entity summary does not work via prism.
Ajay Yadava [Mon, 14 Mar 2016 10:03:47 +0000 (15:33 +0530)] 
FALCON-1743 Entity summary does not work via prism.

Author: Ajay Yadava <ajaynsit@gmail.com>

Reviewers: Sandeep Samudrala, Pallavi Rao, Deepak Barr

Closes #65 from ajayyadava/1743

6 years agoFALCON-1848 Late rerun has to be a forced reun
Pallavi Rao [Mon, 14 Mar 2016 09:36:22 +0000 (15:06 +0530)] 
FALCON-1848 Late rerun has to be a forced reun

Had accidentally changed it while fixing FALCON-1835

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: Sandeep Samudrala <sandeep.samudrala@inmobi.com>

Closes #66 from pallavi-rao/1848

6 years agoFALCON-1846 Fixing EntityDryRunTest
Pragya [Fri, 11 Mar 2016 10:03:35 +0000 (15:33 +0530)] 
FALCON-1846 Fixing EntityDryRunTest

FALCON-1846 Fixing ProcessInstanceStatusTest,ProcessInstanceResumeTest,EntityDryRunTest

Author: Pragya <mittal.pragya23@gmail.com>

Reviewers: Murali Ramasami <murali.msse@gmail.com>"

Closes #63 from pragya-mittal/testFix-1846

6 years agoFALCON-1839 Test case for APIs for entities scheduled on native scheduler
Pragya [Mon, 7 Mar 2016 06:13:05 +0000 (11:43 +0530)] 
FALCON-1839 Test case for APIs for entities scheduled on native scheduler

Author: Pragya <mittal.pragya23@gmail.com>

Reviewers: "Murali Ramasami <murali.msse@gmail.com>, Paul Isaychuk <pisaychuk@apache.org>, Deepak Kumar Barr <deepak.barr@gmail.com>"

Closes #49 from pragya-mittal/native-api and squashes the following commits:

a7baab5 [Pragya] Modified Changes.txt
c800405 [Pragya] Addressed review comments
4fd3cf8 [Pragya] Making waitTillInstanceReachState generic
9009d39 [Pragya] Adding assertion
8b95448 [Pragya] Adding doc for NativeInstanceUtil.waitTillInstancesAreCreated
d244279 [Pragya] Modifying NativeInstanceUtil.waitTillInstancesAreCreated based on review comments
9a882df [Pragya] Modifying totalMinutesToWait  in NativeInstancEUtil to wait for defined time
97a155f [Pragya] Review comments addressed
bbf551a [Pragya] Review commenst addressed
b674558 [Pragya] FALCON-1839 Test case for APIs for entities scheduled on native scheduler
26c462a [Pragya] Merge branch 'master' of https://github.com/apache/falcon
cd0664a [Pragya] Revert "FALCON-1832 Adding tags in SchedulableEntityInstance.compareTo()"
db094ea [Pragya] FALCON-1832 Adding tags in SchedulableEntityInstance.compareTo()
3c302e2 [Pragya] Merge branch 'master' of https://github.com/apache/falcon
f037385 [Pragya] Merge branch 'master' of https://github.com/apache/falcon
4c19ec0 [Pragya] Merge branch 'master' of https://github.com/apache/falcon
3b7fd63 [Pragya] FALCON-1829 Add regression for submit and schedule process on native scheduler (time based)

6 years agoyes
yzheng-hortonworks [Sat, 5 Mar 2016 01:18:38 +0000 (17:18 -0800)] 

Add running, succeeded, failed, killed and suspended instances to titan DB based on JMS notifications on workflow jobs. Instance-entity edge properties (e.g. nominal time, status) are also added for vertex-centric indexing.

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: Balu Vellanki <bvellanki@hortonworks.com>, Venkatesan Ramachandran <Me.venkatr@gmail.com>

Closes #47 from yzheng-hortonworks/FALCON-1111 and squashes the following commits:

f2c9faf [yzheng-hortonworks] review by Balu
ff9556b [yzheng-hortonworks] review from Venky
b76f642 [yzheng-hortonworks] FALCON-1111 Instance update on titan DB based on JMS notifications on workflow jobs

6 years agoFALCON-1825 Process end time not exclusive in case of Native Scheduler
pavankumar526 [Fri, 4 Mar 2016 12:12:22 +0000 (17:42 +0530)] 
FALCON-1825 Process end time not exclusive in case of Native Scheduler

Author: pavankumar526 <pavan.kolamuri@gmail.com>

Reviewers: "Pallavi Rao <pallavi.rao@inmobi.com>"

Closes #59 from pavankumar526/FALCON-1825

6 years agoFALCON-1826 Execution order not honoured when instances are KILLED
Pallavi Rao [Fri, 4 Mar 2016 06:48:26 +0000 (12:18 +0530)] 
FALCON-1826 Execution order not honoured when instances are KILLED

The problem was that the unregister for schedule did not unregister the instance immediately. It would add to instancesToIgnore and would get ignored while scheduling. This causes issues as instancesToIgnore will need to be scanned at more than one place.

The fix is to remove the instance from "awaitingSchedule" list synchronously for unregister.

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: "Pavan Kumar Kolamuri <pavan.kolamuri@gmail.com>"

Closes #58 from pallavi-rao/1826 and squashes the following commits:

9ace53d [Pallavi Rao] FALCON-1826 Addressed review comments
dec29c0 [Pallavi Rao] FALCON-1826 Execution order not honoured when instances are KILLED

6 years agoFALCON-1844 Replication to delete files removed in source by default
Pallavi Rao [Thu, 3 Mar 2016 10:30:07 +0000 (16:00 +0530)] 
FALCON-1844 Replication to delete files removed in source by default

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: Pavan Kolamuri <pavan.kolamuri@inmobi.com>, Sandeep Samudrala<sandeep.samudrala@inmobi.com>, Peeyush Bishnoi<peeyusb@hortonworks.com>

Closes #60 from pallavi-rao/1844

6 years agoFALCON-1842 Falcon build failed in Jenkins at OozieFeedWorkflowBuilderTest
bvellanki [Tue, 1 Mar 2016 08:09:00 +0000 (13:39 +0530)] 
FALCON-1842 Falcon build failed in Jenkins at OozieFeedWorkflowBuilderTest

…rTest (Balu Vellanki)

Author: bvellanki <bvellanki@hortonworks.com>

Reviewers: Ajay Yadava <ajayyadava@apache.org>, Pallavi Rao

Closes #56 from bvellanki/master

6 years agoFALCON-1841 Grouping test in falcon for running nightly regression(Pragya Mittal)
Pragya [Tue, 1 Mar 2016 06:43:55 +0000 (12:13 +0530)] 
FALCON-1841 Grouping test in falcon for running nightly regression(Pragya Mittal)

Author: Pragya <mittal.pragya23@gmail.com>

Reviewers: Murali Ramasami <murali.msse@gmail.com>

Closes #51 from pragya-mittal/grouping

6 years agoFALCON-1724 Falcon CLI.twiki in docs folder is not pointed by index page
Praveen Adlakha [Fri, 26 Feb 2016 08:46:28 +0000 (14:16 +0530)] 
FALCON-1724 Falcon CLI.twiki in docs folder is not pointed by index page

Currentlt Faclon CLI.twiki is not pointed by any page we need to remove it to avoid the confusion.

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Pallavi Rao <pallavi.rao@inmobi.com>

Closes #50 from PraveenAdlakha/1724 and squashes the following commits:

f4d183e [Praveen Adlakha] pallavi's comment addressed
d586d04 [Praveen Adlakha] FALCON-1724 Falcon CLI.twiki in docs folder is not pointed by index page

6 years agoFALCON-1835 [IMPROVEMENT] Falcon should do coord rerun rather than workflow rerun
Pallavi Rao [Wed, 24 Feb 2016 11:55:23 +0000 (17:25 +0530)] 
FALCON-1835 [IMPROVEMENT] Falcon should do coord rerun rather than workflow rerun

…to ensure concurrency

Patch contains the following changes:
1. Upgrade to Oozie client 4.2 (from 4.1).
2. Remove workflow rerun completely and switch to coord rerun.
3. Introduced parentId in WorkflowExecutionContext in order to store the coord action of a given workflow. This will be used during retry and late data rerun.
4. In cases where parentId cannot be populated (notification from Post processing action will not contain this), rerun method queries Oozie to retrieve the parent id.
5. The rerun method now returns the actual status after rerun, rather than hardcoded “RUNNING”.
6. In 4.2, Oozie does a Yarn job kill to kill any lingering executions of previous run of a workflow. It uses RMClientProxy to do the same. Since in local mode, RPC cannot happen, had to introduce LocalFalconRPCClientFactory to prevent RPC in local mode.

Author: Pallavi Rao <pallavi.rao@inmobi.com>

Reviewers: Sandeep Samudrala <sandeep.samudrala@inmobi.com>, Srikanth Sundarrajan<srikath.sundarrajan@inmobi.com>, Pavan Kolamuri <pavan.kolamuri@inmobi.com>

Closes #48 from pallavi-rao/1835 and squashes the following commits:

0e0b317 [Pallavi Rao] FALCON-1835 Clearing old settings from config retrieved
0c0d2c4 [Pallavi Rao] FALCON-1835 Updated doc.
418ac16 [Pallavi Rao] FALCON-1835 Fixed checkstyle issue
96cd509 [Pallavi Rao] FALCON-1835 Updated rerun method to throw exception when both skip.nodes and failnodes are specified
8b0cb1f [Pallavi Rao] FALCON-1835 Updated rerun method to merge user props and job conf.
43df66d [Pallavi Rao] FALCON-1835 Falcon should do coord rerun rather than workflow rerun to ensure concurrency

6 years agoFALCON-1811 Status API does not honour start option
Praveen Adlakha [Wed, 24 Feb 2016 11:00:46 +0000 (16:30 +0530)] 
FALCON-1811 Status API does not honour start option

$ bin/falcon instance -type process -name process1 -start 2015-10-09T14:00Z -status -numResults 2
Consolidated Status: SUCCEEDED

Instance Cluster SourceCluster Status Start End Details Log
2015-12-31T01:00Z local - SUSPENDED 2016-02-11T09:34Z - - http://blr-test-129.corp.inmobi.com:11000/oozie?job=0000028-151229192519233-oozie-oozi-W
 mr-node null null
 failed-post-processing null null
2016-01-01T01:00Z local - SUSPENDED 2016-02-11T10:18Z - - http://blr-test-129.corp.inmobi.com:11000/oozie?job=0000030-151229192519233-oozie-oozi-W
 mr-node null null
 failed-post-processing null null

Additional Information:
Response: default/STATUS
Request Id: default/124451500qtp-822600088-12 - 2ca2d36d-6e03-4e14-9e97-1448aed90084

Note we will not get the same result when status is waiting as at that time start time is null.

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Pallavi Rao <pallavi.rao@inmobi.com>

Closes #40 from PraveenAdlakha/1811 and squashes the following commits:

d9925d0 [Praveen Adlakha] pallavi's comments addressed
7784468 [Praveen Adlakha] checkstyle voilation issue solved
4530233 [Praveen Adlakha] changes done as per pallavi's comment
3c3da61 [Praveen Adlakha] Deepak's comments addressed
de08573 [Praveen Adlakha] FALCON-1811 Status API does not honour start option

6 years agoFALCON-1621 [BUG] Lifecycle of entity gets missed when prism and falcon server commun...
Praveen Adlakha [Fri, 19 Feb 2016 12:39:19 +0000 (18:09 +0530)] 
FALCON-1621 [BUG] Lifecycle of entity gets missed when prism and falcon server communicates

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: Sandeep Samudrala <sandeep.samudrala@inmobi.com>, Pallavi Rao <pallavi.rao@inmobi.com>

Closes #46 from PraveenAdlakha/1621

6 years agoUpdate CHANGES.txt with FALCON-1627 subject
peeyush b [Fri, 19 Feb 2016 05:02:56 +0000 (10:32 +0530)] 
Update CHANGES.txt with FALCON-1627 subject

6 years agoFALCON-1627 Provider integration with Azure Data Factory pipelines
yzheng-hortonworks [Fri, 19 Feb 2016 04:35:27 +0000 (10:05 +0530)] 
FALCON-1627 Provider integration with Azure Data Factory pipelines

This integration allows Microsoft Azure Data Factory pipelines to invoke Falcon activities (i.e. replication, hive and pig proessing work), so the user can build a hybrid Hadoop data pipelines leveraging on-premises Hadoop clusters and cloud based Cortana Analytics services like HDInsight Hadoop clusters and Azure Machine Learning.

Incorporated review comments from Balu on apache review board:

Author: yzheng-hortonworks <yzheng@hortonworks.com>

Reviewers: Sandeep Samudrala<sandysmdl@gmail.com>, Peeyush Bishnoi<peeyushb@apache.org>, Balu Vellanki<bvellanki@hortonworks.com>, Venkat Ranganathan<n.r.v@live.com>

Closes #27 from yzheng-hortonworks/FALCON-1627 and squashes the following commits:

9e13745 [yzheng-hortonworks] throw exception if no super user for adf in the startup properties
63eedfd [yzheng-hortonworks] polish log (review from peeyushb)
e5a1d97 [yzheng-hortonworks] review from PraveenAdlakha and peeyushb
2090aaf [yzheng-hortonworks] review from sandeepSamudrala
4c5f2b2 [yzheng-hortonworks] FALCON-1627 Provider integration with Azure Data Factory pipelines

6 years ago FALCON-1584 Falcon allows invalid hadoop queue name for schedulable feed entities...
bvellanki [Thu, 18 Feb 2016 22:16:55 +0000 (14:16 -0800)] 
 FALCON-1584 Falcon allows invalid hadoop queue name for schedulable feed entities. Contributed by Venkatesan Ramachandran via Balu Vellanki.

6 years agoFALCON-1566 Add test for SLA monitoring API
Pragya [Thu, 18 Feb 2016 09:50:02 +0000 (15:20 +0530)] 
FALCON-1566 Add test for SLA monitoring API

Author: Pragya <mittal.pragya23@gmail.com>

Reviewers: Paul Isaychuk <pisaychuk@apache.org>, Deepak Kumar Barr <deepak.barr@gmail.com>

Closes #44 from pragya-mittal/feed-sla and squashes the following commits:

c6c6f65 [Pragya] Review comments addressed
251d404 [Pragya] Resolved merge conflicts
f5b2888 [Pragya] FALCON-1566 Add test for SLA monitoring API
f037385 [Pragya] Merge branch 'master' of https://github.com/apache/falcon
4c19ec0 [Pragya] Merge branch 'master' of https://github.com/apache/falcon
3b7fd63 [Pragya] FALCON-1829 Add regression for submit and schedule process on native scheduler (time based)

6 years agoFALCON-887 Support for multiple lib paths in falcon process
Sowmya Ramesh [Thu, 18 Feb 2016 08:55:03 +0000 (14:25 +0530)] 
FALCON-887 Support for multiple lib paths in falcon process

Author: Sowmya Ramesh <sramesh@hortonworks.com>

Reviewers: Peeyush Bishnoi <peeyushb@apache.org>, Pavan Kumar<pavankumar@apache.org>

Closes #38 from sowmyaramesh/FALCON-887 and squashes the following commits:

95865df [Sowmya Ramesh] Moved getting lib path to private method
3209a7e [Sowmya Ramesh] Applied feedback
aa8f6a2 [Sowmya Ramesh] FALCON-887 Fixed checkstyle errors
ad4d455 [Sowmya Ramesh] FALCON-887 Support for multiple lib paths in falcon process