incubator-amaterasu.git
7 days agoMerge pull request #2 from kirupagaran/AMATERASU-13 master
Yaniv Rodenski [Sun, 14 Jan 2018 11:51:14 +0000 (22:51 +1100)] 
Merge pull request #2 from kirupagaran/AMATERASU-13

Amaterasu 13

7 days agoMerge branch 'master' into AMATERASU-13 2/head
Yaniv Rodenski [Sun, 14 Jan 2018 11:50:59 +0000 (22:50 +1100)] 
Merge branch 'master' into AMATERASU-13

10 days agoMerge branch 'master' into AMATERASU-13
Yaniv Rodenski [Thu, 11 Jan 2018 08:49:04 +0000 (19:49 +1100)] 
Merge branch 'master' into AMATERASU-13

10 days agoMerge pull request #3 from eyalbenivri/feature/yarn-support-pyfix
Yaniv Rodenski [Thu, 11 Jan 2018 08:45:35 +0000 (19:45 +1100)] 
Merge pull request #3 from eyalbenivri/feature/yarn-support-pyfix

Feature/yarn support pyfix

11 days agoYARN jobs run end-to-end 3/head
Yaniv Rodenski [Thu, 11 Jan 2018 01:41:38 +0000 (12:41 +1100)] 
YARN jobs run end-to-end

11 days agofixed directory creation by spark
Yaniv Rodenski [Wed, 10 Jan 2018 14:09:51 +0000 (01:09 +1100)] 
fixed directory creation by spark

11 days agofixed directory creation by spark
Yaniv Rodenski [Wed, 10 Jan 2018 14:07:50 +0000 (01:07 +1100)] 
fixed directory creation by spark

12 days agochanged mem and added chmod to yarn script that should be removed.
Eyal Ben Ivri [Tue, 9 Jan 2018 22:55:40 +0000 (23:55 +0100)] 
changed mem and added chmod to yarn script that should be removed.

12 days agochanged mem and added chmod to yarn script that should be removed.
Eyal Ben Ivri [Tue, 9 Jan 2018 22:54:54 +0000 (23:54 +0100)] 
changed mem and added chmod to yarn script that should be removed.

2 weeks agoadded driver opts to executor java command - solved issue with spark classpath
Eyal Ben Ivri [Sun, 7 Jan 2018 11:43:50 +0000 (12:43 +0100)] 
added driver opts to executor java command - solved issue with spark classpath

2 weeks agofixing executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 08:57:27 +0000 (19:57 +1100)] 
fixing executor classpath

2 weeks agofixing executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 08:07:26 +0000 (19:07 +1100)] 
fixing executor classpath

2 weeks agofixing executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 05:34:56 +0000 (16:34 +1100)] 
fixing executor classpath

2 weeks agologging executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 05:20:38 +0000 (16:20 +1100)] 
logging executor classpath

2 weeks agologging executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 04:28:42 +0000 (15:28 +1100)] 
logging executor classpath

2 weeks agologging executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 04:08:39 +0000 (15:08 +1100)] 
logging executor classpath

2 weeks agologging executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 03:18:39 +0000 (14:18 +1100)] 
logging executor classpath

2 weeks agologging executor classpath
Yaniv Rodenski [Sun, 7 Jan 2018 03:15:55 +0000 (14:15 +1100)] 
logging executor classpath

2 weeks agopersistance error logging not throwing exception
Yaniv Rodenski [Sun, 7 Jan 2018 03:02:06 +0000 (14:02 +1100)] 
persistance error logging not throwing exception

2 weeks agopersistance error logging not throwing exception
Yaniv Rodenski [Sun, 7 Jan 2018 03:00:20 +0000 (14:00 +1100)] 
persistance error logging not throwing exception

2 weeks agochanged executor cp order
Yaniv Rodenski [Sun, 7 Jan 2018 00:44:37 +0000 (11:44 +1100)] 
changed executor cp order

2 weeks agoadded scala compiler ref
Yaniv Rodenski [Sun, 7 Jan 2018 00:38:30 +0000 (11:38 +1100)] 
added scala compiler ref

2 weeks agofixed scala version
Yaniv Rodenski [Sat, 6 Jan 2018 10:47:11 +0000 (21:47 +1100)] 
fixed scala version

2 weeks agoadded container tokens
Yaniv Rodenski [Sat, 6 Jan 2018 02:30:26 +0000 (13:30 +1100)] 
added container tokens

2 weeks agoadded hadoop conf dir to classpath - moved us forward.
Eyal Ben Ivri [Sat, 6 Jan 2018 00:34:39 +0000 (01:34 +0100)] 
added hadoop conf dir to classpath - moved us forward.

2 weeks agoMerge branch 'feature/yarn-support-pyfix' of github.com:eyalbenivri/incubator-amatera...
Yaniv Rodenski [Fri, 5 Jan 2018 23:47:29 +0000 (10:47 +1100)] 
Merge branch 'feature/yarn-support-pyfix' of github.com:eyalbenivri/incubator-amaterasu into feature/yarn-support-pyfix

2 weeks agomem fix
Yaniv Rodenski [Fri, 5 Jan 2018 23:46:35 +0000 (10:46 +1100)] 
mem fix

2 weeks agoadded write logging and output
Eyal Ben Ivri [Fri, 5 Jan 2018 23:32:28 +0000 (00:32 +0100)] 
added write logging and output

2 weeks agoreadded anaconda
Yaniv Rodenski [Fri, 5 Jan 2018 12:46:33 +0000 (23:46 +1100)] 
readded anaconda

2 weeks agoreadded anaconda
Yaniv Rodenski [Fri, 5 Jan 2018 11:51:34 +0000 (22:51 +1100)] 
readded anaconda

2 weeks agoUpdated parser to parse advanced queries
kirupagaran [Thu, 4 Jan 2018 13:50:33 +0000 (00:50 +1100)] 
Updated parser to parse advanced queries

2 weeks agoIf no dataframe read format is specified by user, parquet is used as default file...
kirupagaran [Thu, 4 Jan 2018 09:54:27 +0000 (20:54 +1100)] 
If no dataframe read format is specified by user, parquet is used as default file format to load data from previous actions

2 weeks agofixed env issues
Eyal Ben Ivri [Thu, 4 Jan 2018 07:24:33 +0000 (08:24 +0100)] 
fixed env issues

2 weeks agoadded logs again
Yaniv Rodenski [Thu, 4 Jan 2018 07:06:35 +0000 (18:06 +1100)] 
added logs again

2 weeks agolog fix
Yaniv Rodenski [Thu, 4 Jan 2018 06:21:52 +0000 (17:21 +1100)] 
log fix

2 weeks agochanged JVM maximum mem
Yaniv Rodenski [Thu, 4 Jan 2018 06:16:49 +0000 (17:16 +1100)] 
changed JVM maximum mem

2 weeks agoremoved fs.hdfs.impl
Yaniv Rodenski [Thu, 4 Jan 2018 06:08:08 +0000 (17:08 +1100)] 
removed fs.hdfs.impl

2 weeks agoadded env YARN_CONF_DIR to container lounch
Yaniv Rodenski [Thu, 4 Jan 2018 04:14:44 +0000 (15:14 +1100)] 
added env YARN_CONF_DIR to container lounch

2 weeks agoadded yarn home dir
Yaniv Rodenski [Wed, 3 Jan 2018 23:09:11 +0000 (10:09 +1100)] 
added yarn home dir

2 weeks agoMerge branch 'feature/yarn-support-pyfix' of github.com:eyalbenivri/incubator-amatera...
Yaniv Rodenski [Wed, 3 Jan 2018 22:30:35 +0000 (09:30 +1100)] 
Merge branch 'feature/yarn-support-pyfix' of github.com:eyalbenivri/incubator-amaterasu into feature/yarn-support-pyfix

2 weeks agotest serielizing jeckson
Yaniv Rodenski [Wed, 3 Jan 2018 22:26:01 +0000 (09:26 +1100)] 
test serielizing jeckson

2 weeks agoadded logs
Eyal Ben Ivri [Wed, 3 Jan 2018 22:17:59 +0000 (23:17 +0100)] 
added logs

2 weeks agoadded persistance log
Yaniv Rodenski [Wed, 3 Jan 2018 21:59:08 +0000 (08:59 +1100)] 
added persistance log

2 weeks agoadded persistance log
Yaniv Rodenski [Wed, 3 Jan 2018 21:46:37 +0000 (08:46 +1100)] 
added persistance log

2 weeks agoadded persistance log
Yaniv Rodenski [Wed, 3 Jan 2018 21:35:31 +0000 (08:35 +1100)] 
added persistance log

2 weeks agopyfiles moved to comma delimited
Yaniv Rodenski [Wed, 3 Jan 2018 21:06:56 +0000 (08:06 +1100)] 
pyfiles moved to comma delimited

2 weeks agofiltered only pyfiles
Yaniv Rodenski [Wed, 3 Jan 2018 11:39:24 +0000 (22:39 +1100)] 
filtered only pyfiles

2 weeks ago1.SparkSql runner can read a dataframe from previous action through AmaContext; 2...
kirupagaran [Wed, 3 Jan 2018 04:22:49 +0000 (15:22 +1100)] 
1.SparkSql runner can read a dataframe from previous action through AmaContext; 2.SparkSql runner can parse amacontext in user supplied query; 3.SparkSql can read/write data directly from a sql,json,parquet, orm and text file. JDBC support will be available in future releases

2 weeks agoparameteriazed spark home
Yaniv Rodenski [Wed, 3 Jan 2018 03:46:12 +0000 (14:46 +1100)] 
parameteriazed spark home

3 weeks agoyarn work
Eyal Ben Ivri [Fri, 29 Dec 2017 10:31:52 +0000 (11:31 +0100)] 
yarn work

3 weeks agobarely merged with origin/master, may the schwartz be with us
Yaniv Rodenski [Thu, 28 Dec 2017 12:12:35 +0000 (23:12 +1100)] 
barely merged with origin/master, may the schwartz be with us

3 weeks agomissing code
Eyal Ben Ivri [Thu, 28 Dec 2017 07:49:29 +0000 (08:49 +0100)] 
missing code

3 weeks agoyarn-support (copied from old fork) - preparing for merge.
Eyal Ben Ivri [Wed, 27 Dec 2017 00:55:52 +0000 (01:55 +0100)] 
yarn-support (copied from old fork) - preparing for merge.

5 weeks agoMerge pull request #1 from kirupagaran/jetty-sql-gradle-apache-upgrade
Yaniv Rodenski [Tue, 12 Dec 2017 00:31:46 +0000 (11:31 +1100)] 
Merge pull request #1 from kirupagaran/jetty-sql-gradle-apache-upgrade

Jetty Web Server and SparkSqlRunner upgraded to gradle,Spark 2.0 and code refactored to use apache in package names

2 months agoJetty Web Server and SparkSqlRunner upgraded to gradle,Spark 2.0 and refactored to... 1/head
kirupagaran [Mon, 30 Oct 2017 06:06:02 +0000 (17:06 +1100)] 
Jetty Web Server and SparkSqlRunner upgraded to gradle,Spark 2.0 and refactored to use apache in package names

3 months agoMerge pull request #48 from nadav-har-tzvi/feature/conda-dependencies-fixed
Yaniv Rodenski [Sat, 30 Sep 2017 11:57:32 +0000 (21:57 +1000)] 
Merge pull request #48 from nadav-har-tzvi/feature/conda-dependencies-fixed

Feature - pyspark support with dependencies

4 months agoMerge branch 'ama-master' into feature/conda-dependencies-fixed
Nadav Har Tzvi [Tue, 12 Sep 2017 07:59:24 +0000 (10:59 +0300)] 
Merge branch 'ama-master' into feature/conda-dependencies-fixed

# Conflicts:
# common/src/main/scala/org/apache/ameterasu/common/dataobjects/ExecData.scala
# common/src/main/scala/org/apache/ameterasu/common/execution/dependencies/Dependencies.scala
# executor/build.gradle
# executor/src/main/resources/spark_intp.py
# executor/src/main/scala/org/apache/amaterasu/executor/execution/actions/runners/spark/PySpark/PySparkRunner.scala
# executor/src/main/scala/org/apache/amaterasu/executor/execution/actions/runners/spark/SparkRunnersProvider.scala
# executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/ActionsExecutor.scala
# executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkRunnerHelper.scala
# executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala
# executor/src/test/resources/pyspark.zip
# executor/src/test/resources/spark_intp.py
# executor/src/test/scala/io/shinto/amaterasu/RunnersTests/RunnersLoadingTests.scala
# executor/src/test/scala/org/apache/amaterasu/spark/PySparkRunnerTests.scala
# executor/src/test/scala/org/apache/amaterasu/spark/SparkTestsSuite.scala
# leader/src/main/scala/org/apache/amaterasu/leader/utilities/DataLoader.scala
# leader/src/main/scripts/ama-start.sh

4 months agoMerge branch 'ama-master' into feature/conda-dependencies-fixed
Nadav Har Tzvi [Tue, 12 Sep 2017 07:59:16 +0000 (10:59 +0300)] 
Merge branch 'ama-master' into feature/conda-dependencies-fixed

# Conflicts:
# common/src/main/scala/org/apache/ameterasu/common/dataobjects/ExecData.scala
# common/src/main/scala/org/apache/ameterasu/common/execution/dependencies/Dependencies.scala
# executor/build.gradle
# executor/src/main/resources/spark_intp.py
# executor/src/main/scala/org/apache/amaterasu/executor/execution/actions/runners/spark/PySpark/PySparkRunner.scala
# executor/src/main/scala/org/apache/amaterasu/executor/execution/actions/runners/spark/SparkRunnersProvider.scala
# executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/ActionsExecutor.scala
# executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkRunnerHelper.scala
# executor/src/main/scala/org/apache/spark/repl/amaterasu/runners/spark/SparkScalaRunner.scala
# executor/src/test/resources/pyspark.zip
# executor/src/test/resources/spark_intp.py
# executor/src/test/scala/io/shinto/amaterasu/RunnersTests/RunnersLoadingTests.scala
# executor/src/test/scala/org/apache/amaterasu/spark/PySparkRunnerTests.scala
# executor/src/test/scala/org/apache/amaterasu/spark/SparkTestsSuite.scala
# leader/src/main/scala/org/apache/amaterasu/leader/utilities/DataLoader.scala
# leader/src/main/scripts/ama-start.sh

4 months agoMerge branch 'nadav-har-tzvi-feature/conda-dependencies-fixed'
Nadav Har Tzvi [Mon, 11 Sep 2017 09:14:28 +0000 (12:14 +0300)] 
Merge branch 'nadav-har-tzvi-feature/conda-dependencies-fixed'

4 months agocleanup and tests passing
Nadav Har Tzvi [Mon, 11 Sep 2017 09:06:24 +0000 (12:06 +0300)] 
cleanup and tests passing

4 months agofixed double instanciation of tests
Yaniv Rodenski [Mon, 11 Sep 2017 08:55:23 +0000 (18:55 +1000)] 
fixed double instanciation of tests

4 months agotrying to inject factory
Nadav Har Tzvi [Mon, 11 Sep 2017 08:46:00 +0000 (11:46 +0300)] 
trying to inject factory

4 months agoforgot to add pyspark.zip
Nadav Har Tzvi [Sun, 10 Sep 2017 09:07:55 +0000 (12:07 +0300)] 
forgot to add pyspark.zip

4 months agowrite test exports to /tmp to get rid of them
Nadav Har Tzvi [Sun, 10 Sep 2017 07:24:01 +0000 (10:24 +0300)] 
write test exports to /tmp to get rid of them

4 months agotest tried writing into the fs root instead of the CWD
Nadav Har Tzvi [Sun, 10 Sep 2017 06:31:38 +0000 (09:31 +0300)] 
test tried writing into the fs root instead of the CWD

4 months agowrong one...
Nadav Har Tzvi [Sat, 9 Sep 2017 02:43:29 +0000 (05:43 +0300)] 
wrong one...

4 months agostill couldn't catch this, no idea why
Nadav Har Tzvi [Sat, 9 Sep 2017 02:34:44 +0000 (05:34 +0300)] 
still couldn't catch this, no idea why

4 months agohardcode it for now, unless someone wants to control this
Nadav Har Tzvi [Sat, 9 Sep 2017 02:27:12 +0000 (05:27 +0300)] 
hardcode it for now, unless someone wants to control this

4 months agoduh
Nadav Har Tzvi [Sat, 9 Sep 2017 02:20:14 +0000 (05:20 +0300)] 
duh

4 months agopyspark tests almost work, exports fail
Nadav Har Tzvi [Sat, 9 Sep 2017 00:18:52 +0000 (03:18 +0300)] 
pyspark tests almost work, exports fail

4 months agoMerge branch 'feature/conda-dependencies-fixed' of https://github.com/nadav-har-tzvi...
Nadav Har Tzvi [Thu, 7 Sep 2017 07:20:59 +0000 (10:20 +0300)] 
Merge branch 'feature/conda-dependencies-fixed' of https://github.com/nadav-har-tzvi/amaterasu into feature/conda-dependencies-fixed

4 months agotrying to make this work by configuration, so tests can actually work
Nadav Har Tzvi [Thu, 7 Sep 2017 07:17:54 +0000 (10:17 +0300)] 
trying to make this work by configuration, so tests can actually work

4 months agocleaned up
Yaniv Rodenski [Wed, 6 Sep 2017 07:26:15 +0000 (17:26 +1000)] 
cleaned up

4 months agomissed passing pythonpath to actual executed script
Nadav Har Tzvi [Tue, 5 Sep 2017 17:54:21 +0000 (20:54 +0300)] 
missed passing pythonpath to actual executed script

4 months agostarted fitting tests and trying to solve an issue with Pyspark not recognized
Nadav Har Tzvi [Tue, 5 Sep 2017 17:28:01 +0000 (20:28 +0300)] 
started fitting tests and trying to solve an issue with Pyspark not recognized

4 months agodocumentation
Nadav Har Tzvi [Tue, 5 Sep 2017 03:36:48 +0000 (06:36 +0300)] 
documentation

4 months agonon-conda code fails
Yaniv Rodenski [Mon, 4 Sep 2017 12:19:31 +0000 (22:19 +1000)] 
non-conda code fails

4 months agoremoved notifier calls
Nadav Har Tzvi [Sun, 3 Sep 2017 11:28:22 +0000 (14:28 +0300)] 
removed notifier calls

4 months agotrying to figure out why scala runner is throwing null
Nadav Har Tzvi [Sun, 3 Sep 2017 06:24:56 +0000 (09:24 +0300)] 
trying to figure out why scala runner is throwing null

4 months agotrying to figure out why scala runner is throwing null
Nadav Har Tzvi [Sun, 3 Sep 2017 05:55:17 +0000 (08:55 +0300)] 
trying to figure out why scala runner is throwing null

4 months agoseems wrong?
Nadav Har Tzvi [Sat, 2 Sep 2017 08:56:47 +0000 (11:56 +0300)] 
seems wrong?

4 months agoMerge branch 'feature/conda-dependencies-fixed' of https://github.com/nadav-har-tzvi...
Nadav Har Tzvi [Sat, 2 Sep 2017 08:34:47 +0000 (11:34 +0300)] 
Merge branch 'feature/conda-dependencies-fixed' of https://github.com/nadav-har-tzvi/amaterasu into feature/conda-dependencies-fixed

# Conflicts:
# executor/src/main/scala/org/apache/amaterasu/executor/execution/actions/runners/spark/SparkRunnersProvider.scala
# leader/src/main/scripts/ama-start.sh

4 months agotried to open a file, it is wrong, should pass the source as it is.
Nadav Har Tzvi [Sat, 2 Sep 2017 08:34:10 +0000 (11:34 +0300)] 
tried to open a file, it is wrong, should pass the source as it is.

4 months agocorrected spark_intp and other good things
Yaniv Rodenski [Sat, 2 Sep 2017 08:06:29 +0000 (18:06 +1000)] 
corrected spark_intp and other good things

4 months agocorrected spark_intp and other good things
Yaniv Rodenski [Sat, 2 Sep 2017 08:03:56 +0000 (18:03 +1000)] 
corrected spark_intp and other good things

4 months agospark 2.0 uses a different executable
Nadav Har Tzvi [Sat, 2 Sep 2017 06:44:21 +0000 (09:44 +0300)] 
spark 2.0 uses a different executable

"Running python applications through 'pyspark' is not supported as of Spark 2.0.
Use ./bin/spark-submit <python file>"

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:53:38 +0000 (14:53 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:45:26 +0000 (14:45 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:29:28 +0000 (14:29 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:22:53 +0000 (14:22 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:20:18 +0000 (14:20 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:09:53 +0000 (14:09 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:06:56 +0000 (14:06 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:05:21 +0000 (14:05 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:03:35 +0000 (14:03 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 11:01:32 +0000 (14:01 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 10:58:52 +0000 (13:58 +0300)] 
logging

4 months agologging
Nadav Har Tzvi [Fri, 1 Sep 2017 10:51:30 +0000 (13:51 +0300)] 
logging

4 months agochanged dependencies format
Nadav Har Tzvi [Fri, 1 Sep 2017 10:31:03 +0000 (13:31 +0300)] 
changed dependencies format

4 months agocopy python resources
Nadav Har Tzvi [Fri, 1 Sep 2017 09:18:44 +0000 (12:18 +0300)] 
copy python resources