incubator-sdap-nexus.git
3 weeks agoSDAP-151 Determine parallelism automatically for Spark analytics (#50) master 60/head
Joseph Jacob [Tue, 20 Nov 2018 22:00:25 +0000 (14:00 -0800)] 
SDAP-151 Determine parallelism automatically for Spark analytics (#50)

* Removed spark configuration, added nparts configuration, and autocompute parallelism for spark-based time series.

* SDAP-151 Determine parallelism automatically for Spark analytics

5 weeks agoSDAP-137 Webapp should have a mesos and a standalone image (#51)
fgreg [Wed, 7 Nov 2018 16:45:04 +0000 (08:45 -0800)] 
SDAP-137 Webapp should have a mesos and a standalone image (#51)

* Updating images.

* Updating images.

* ignore .nc files

* removed unused files.

* removed unused files.

* Updating images.

* ignore .nc files

* removed unused files.

* removed unused files.

* Updated bulid scripts and quickstart.

* workaround for basemap. change workdir of agent

* moved files to make more sense

* fixed error with conda home env var.

* don't remove spark package

* Need PROJ_LIB defined as agent env var.

8 weeks agoSDAP-157 Bump nexusproto to 1.0.0 (#46)
fgreg [Thu, 18 Oct 2018 00:21:40 +0000 (17:21 -0700)] 
SDAP-157 Bump nexusproto to 1.0.0 (#46)

* updated version dependencies.

* update dockerfiles

2 months agoSDAP-152 Error in nexus client (#43)
fgreg [Sat, 22 Sep 2018 01:38:00 +0000 (18:38 -0700)] 
SDAP-152 Error in nexus client (#43)

* Updated client to parse the new 'iso_time' returned from the time series function.

* handle both cases for backwards compatibility.

* also need to cast now.

* using dtype appropriately

2 months agoRevert "SDAP-152 Error in nexus client (#41)" (#42)
fgreg [Sat, 22 Sep 2018 01:34:57 +0000 (18:34 -0700)] 
Revert "SDAP-152 Error in nexus client (#41)" (#42)

This reverts commit 29a15f5815064a1c8889d51ee4ebdab1963dadf0.

2 months agoSDAP-152 Error in nexus client (#41)
fgreg [Sat, 22 Sep 2018 01:32:35 +0000 (18:32 -0700)] 
SDAP-152 Error in nexus client (#41)

* Updated client to parse the new 'iso_time' returned from the time series function.

* handle both cases for backwards compatibility.

* also need to cast now.

2 months agoRevert "SDAP-152 Error in nexus client (#39)" (#40)
fgreg [Sat, 22 Sep 2018 01:32:00 +0000 (18:32 -0700)] 
Revert "SDAP-152 Error in nexus client (#39)" (#40)

This reverts commit a888889f825c363aba8df43f0a94b3eec9a7dd22.

2 months agoSDAP-152 Error in nexus client (#39)
fgreg [Sat, 22 Sep 2018 01:21:54 +0000 (18:21 -0700)] 
SDAP-152 Error in nexus client (#39)

* Updated client to parse the new 'iso_time' returned from the time series function.

* handle both cases for backwards compatibility.

2 months agoSDAP-150 Error processing dailydifferenceaverage (#38)
fgreg [Thu, 20 Sep 2018 21:51:09 +0000 (14:51 -0700)] 
SDAP-150 Error processing dailydifferenceaverage (#38)

* DDA was not passing in the metadata to the results object resulting in an Exception. Now it is passing in the metadata correctly. I also improved the logic so it will not error if no parameters are provided.

* moved build args to right before they are used to preserve cache.

* adding more broken

2 months agoSDAP-148 Disabled deseason calculation (#37)
fgreg [Wed, 19 Sep 2018 19:12:16 +0000 (12:12 -0700)] 
SDAP-148 Disabled deseason calculation (#37)

2 months agoSDAP-93 Clean up and document HofMoellerSpark (#35)
fgreg [Tue, 18 Sep 2018 17:40:51 +0000 (10:40 -0700)] 
SDAP-93 Clean up and document HofMoellerSpark (#35)

* updated params and return values

* SDAP-50 Fix incorrect Hovmoller Implementation

2 months agoupdated to return iso times and mean (#33)
fgreg [Tue, 18 Sep 2018 17:40:17 +0000 (10:40 -0700)] 
updated to return iso times and mean (#33)

2 months agoupdated time series spark to use mean and iso times (#32)
fgreg [Tue, 18 Sep 2018 17:40:02 +0000 (10:40 -0700)] 
updated time series spark to use mean and iso times (#32)

2 months agoupdated data series list to include iso time (#34)
fgreg [Tue, 18 Sep 2018 17:39:44 +0000 (10:39 -0700)] 
updated data series list to include iso time (#34)

2 months agoSDAP-145 Correct URL for Oracle JDK download in NEXUS webapp Docker image (#36)
Joseph Jacob [Mon, 17 Sep 2018 23:00:47 +0000 (16:00 -0700)] 
SDAP-145 Correct URL for Oracle JDK download in NEXUS webapp Docker image (#36)

SDAP-145 Updated Oracle JDK to version 1.8u182-b13 in Nexus webapp Docker image.

2 months agoSDAP-50 Fix incorrect Hovmoller Implementation (#28)
Joseph Jacob [Mon, 17 Sep 2018 22:52:52 +0000 (15:52 -0700)] 
SDAP-50 Fix incorrect Hovmoller Implementation (#28)

This update combines Hovmoller calculation across NEXUS tiles, combines duplicate time stamps, and adds cos(lat) weighting for latitude-averaging.

3 months agoSDAP-139 Update Cassandra Image Version (#31)
fgreg [Mon, 10 Sep 2018 23:37:39 +0000 (16:37 -0700)] 
SDAP-139 Update Cassandra Image Version (#31)

* Update cassandra image version. Made init script run on every startup.

* update docs

3 months agoUpdate cassandra image version. Made init script run on every startup. (#30)
fgreg [Mon, 10 Sep 2018 23:33:50 +0000 (16:33 -0700)] 
Update cassandra image version. Made init script run on every startup. (#30)

3 months agoSDAP-138 Permission Denied Solr Image (#29)
fgreg [Mon, 10 Sep 2018 23:21:50 +0000 (16:21 -0700)] 
SDAP-138 Permission Denied Solr Image (#29)

* Fixing the permissions issues for solr singlenode.

The image still runs as the `solr` user but now `solr` has been given permission to use sudo to `cp` and `chown` the `${SOLR_HOME}/nexustiles` directory. Tested locally on Mac, a linux server, and a linux server running kubernetes.

3 months agoSDAP-109 Upgrade Solr server to version 7 (#19)
fgreg [Thu, 6 Sep 2018 15:46:09 +0000 (08:46 -0700)] 
SDAP-109 Upgrade Solr server to version 7 (#19)

* upgrade solr image

* Using init db script instead of other stuff.

3 months agoAdded note about needing to re-export VERSION
fgreg [Wed, 22 Aug 2018 19:27:30 +0000 (12:27 -0700)] 
Added note about needing to re-export VERSION

5 months agoUpdated docs. (#25)
fgreg [Wed, 18 Jul 2018 05:40:57 +0000 (22:40 -0700)] 
Updated docs. (#25)

5 months agoSDAP-125 Conflict with global variable in client (#24)
fgreg [Tue, 17 Jul 2018 23:47:10 +0000 (16:47 -0700)] 
SDAP-125 Conflict with global variable in client (#24)

* Remove duplicate global variable and function.

* Add log statement

5 months agoSDAP-124 Add ability to submit custom code for execution. (#23)
Joseph Jacob [Tue, 17 Jul 2018 18:07:15 +0000 (11:07 -0700)] 
SDAP-124 Add ability to submit custom code for execution. (#23)

* SDAP-123 Allow Jupyter image to configure nexus repo at build time.

* SDAP-124 Add ability to submit custom code for execution.

* SDAP-124 Add ability to submit custom code for execution.

* Get livy server host from configuration.

* Removed jupyter-livy imaage.

* Set livy_host to localhost instead of 127.0.0.1

5 months agoSDAP-123 Allow Jupyter image to configure nexus repo at build time. (#22)
Joseph Jacob [Mon, 16 Jul 2018 16:41:04 +0000 (09:41 -0700)] 
SDAP-123 Allow Jupyter image to configure nexus repo at build time. (#22)

5 months agoSDAP-119 Fix bug that makes TimeAvgMapSpark appear to hang if there is no data in...
Joseph Jacob [Thu, 12 Jul 2018 23:19:37 +0000 (16:19 -0700)] 
SDAP-119 Fix bug that makes TimeAvgMapSpark appear to hang if there is no data in bounding box (#21)

5 months agoMade use of session object optional. (#20)
fgreg [Wed, 11 Jul 2018 17:08:06 +0000 (10:08 -0700)] 
Made use of session object optional. (#20)

5 months agoSDAP-116 Fix bug in TimeAvgMapSpark that results in row of zeros for certain job...
Joseph Jacob [Mon, 9 Jul 2018 22:01:07 +0000 (15:01 -0700)] 
SDAP-116 Fix bug in TimeAvgMapSpark that results in row of zeros for certain job parameters (#18)

Fixed incorrect rounding of the calculations of latitude/longitude index and number of coordinates.

5 months agoUpdate README.md
fgreg [Sat, 7 Jul 2018 23:55:59 +0000 (16:55 -0700)] 
Update README.md

5 months agoUpdate README.md
fgreg [Sat, 7 Jul 2018 23:55:12 +0000 (16:55 -0700)] 
Update README.md

5 months agoSDAP-103 Update developer documentation (#17)
fgreg [Fri, 29 Jun 2018 14:48:30 +0000 (07:48 -0700)] 
SDAP-103 Update developer documentation (#17)

* Added Introduction, Quickstart, and Ningester sections to docs

5 months agoSDAP-57 Update nexus docker images (#16)
fgreg [Fri, 29 Jun 2018 14:44:48 +0000 (07:44 -0700)] 
SDAP-57 Update nexus docker images (#16)

* Removed unused ingest images

* Added copyright to all files

* All images use SDAP code and repository

5 months agoSDAP-115 Remove old code (#15)
fgreg [Mon, 18 Jun 2018 21:18:39 +0000 (14:18 -0700)] 
SDAP-115 Remove old code (#15)

* update minor versions of requirements

* removed obsolete code

8 months agoSDAP-19 Upgrade Solr client in NEXUS data-access (#14)
Nga Quach [Wed, 4 Apr 2018 15:10:24 +0000 (08:10 -0700)] 
SDAP-19 Upgrade Solr client in NEXUS data-access (#14)

8 months agoSDAP-47 Update NEXUS CLI to support datainbounds algorithm (#12)
Nga Quach [Sat, 24 Mar 2018 03:36:54 +0000 (20:36 -0700)] 
SDAP-47 Update NEXUS CLI to support datainbounds algorithm (#12)

* SDAP-47 Update NEXUS CLI to support datainbounds algorithm

* SDAP-47 Changed method name to subset, allow only bounding_box or metadata_filter, and return list of Point.

* SDAP-47 Changed method name to subset.

* SDAP-47 Updated description of parameter.

9 months agoSDAP-41 Update NEXUS tile service to apply a mask for start/end time (#11)
Nga Quach [Thu, 15 Mar 2018 03:42:20 +0000 (20:42 -0700)] 
SDAP-41 Update NEXUS tile service to apply a mask for start/end time (#11)

9 months agoSDAP-47 Update NEXUS CLI to support datainbounds algorithm (#13)
Nga Quach [Wed, 14 Mar 2018 22:43:07 +0000 (15:43 -0700)] 
SDAP-47 Update NEXUS CLI to support datainbounds algorithm (#13)

9 months agoMerge pull request #10 from ntquach/SDAP-43
Nga Quach [Tue, 13 Mar 2018 21:36:53 +0000 (14:36 -0700)] 
Merge pull request #10 from ntquach/SDAP-43

SDAP-43 Update datainbounds algorithm to handle multiple time steps p…

9 months agoSDAP-42 Fix NEXUS tile service find_tiles_by_metadata (#9)
Nga Quach [Tue, 13 Mar 2018 21:34:38 +0000 (14:34 -0700)] 
SDAP-42 Fix NEXUS tile service find_tiles_by_metadata (#9)

* SDAP-42 Fix NEXUS tile service find_tiles_by_metadata

* SDAP-42 Fix NEXUS tile service find_tiles_by_metadata

* SDAP-42 Fix NEXUS tile service find_tiles_by_metadata

9 months agoSDAP-43 Update datainbounds algorithm to handle multiple time steps per tile 10/head
nchung [Tue, 13 Mar 2018 18:39:47 +0000 (11:39 -0700)] 
SDAP-43 Update datainbounds algorithm to handle multiple time steps per tile

9 months agoMerge pull request #6 from fgreg/SDAP-32
fgreg [Mon, 12 Mar 2018 18:22:52 +0000 (11:22 -0700)] 
Merge pull request #6 from fgreg/SDAP-32

SDAP-32 CassandraProxy time series tiles to support time arrays
SDAP-27 Refactor data-access to use nexusproto

9 months agoMerge pull request #7 from ntquach/SDAP-37
Nga Quach [Sat, 10 Mar 2018 03:04:50 +0000 (19:04 -0800)] 
Merge pull request #7 from ntquach/SDAP-37

SDAP-37 Update SolrProxy to allow tile search by arbitrary tile metadata

9 months agoSDAP-37 Update SolrProxy to allow tile search by arbitrary tile metadata 7/head
nchung [Fri, 9 Mar 2018 18:47:03 +0000 (10:47 -0800)] 
SDAP-37 Update SolrProxy to allow tile search by arbitrary tile metadata

9 months agoSDAP-37 Update SolrProxy to allow tile search by arbitrary tile metadata
nchung [Thu, 8 Mar 2018 19:46:29 +0000 (11:46 -0800)] 
SDAP-37 Update SolrProxy to allow tile search by arbitrary tile metadata

9 months agotime_series_tile now works with an array of times 6/head
Frank Greguska [Tue, 6 Mar 2018 19:04:07 +0000 (11:04 -0800)] 
time_series_tile now works with an array of times

9 months agoMerge branch 'sdap-27' of https://github.com/fgreg/incubator-sdap-nexus into SDAP-32
Frank Greguska [Tue, 6 Mar 2018 18:02:38 +0000 (10:02 -0800)] 
Merge branch 'sdap-27' of https://github.com/fgreg/incubator-sdap-nexus into SDAP-32

9 months agofirst attempt
Frank Greguska [Tue, 6 Mar 2018 18:02:32 +0000 (10:02 -0800)] 
first attempt

9 months agochanged to new library
Frank Greguska [Thu, 1 Mar 2018 01:46:54 +0000 (17:46 -0800)] 
changed to new library

9 months agomade solr delete faster by using query. changed unique key field
Frank Greguska [Fri, 23 Feb 2018 02:40:50 +0000 (18:40 -0800)] 
made solr delete faster by using query. changed unique key field

10 months agoMerge branch 'master' into SDAP-12 4/head
fgreg [Wed, 17 Jan 2018 19:15:26 +0000 (11:15 -0800)] 
Merge branch 'master' into SDAP-12

10 months agoMerge branch 'master' into SDAP-8 3/head
fgreg [Wed, 17 Jan 2018 19:05:48 +0000 (11:05 -0800)] 
Merge branch 'master' into SDAP-8

10 months agoSDAP-14 updated gitignore files 5/head
Frank Greguska [Wed, 17 Jan 2018 18:51:51 +0000 (10:51 -0800)] 
SDAP-14 updated gitignore files

11 months agoadded apache license to .java and .groovy and .gradle files
Frank Greguska [Thu, 11 Jan 2018 20:30:05 +0000 (12:30 -0800)] 
added apache license to .java and .groovy and .gradle files

11 months agoadded apache license to .java and .groovy files
Frank Greguska [Thu, 11 Jan 2018 20:26:42 +0000 (12:26 -0800)] 
added apache license to .java and .groovy files

11 months agoadded apache header to all .py and .pyx files
Frank Greguska [Thu, 11 Jan 2018 20:14:46 +0000 (12:14 -0800)] 
added apache header to all .py and .pyx files

11 months agoremoved JPL copyright header
Frank Greguska [Thu, 11 Jan 2018 20:12:03 +0000 (12:12 -0800)] 
removed JPL copyright header

11 months agoadded apache license header to all py files
Frank Greguska [Thu, 11 Jan 2018 18:43:40 +0000 (10:43 -0800)] 
added apache license header to all py files

11 months agoreformated code
Frank Greguska [Thu, 11 Jan 2018 18:27:01 +0000 (10:27 -0800)] 
reformated code

11 months agoremoved need for Roboto font file
Frank Greguska [Thu, 11 Jan 2018 18:25:03 +0000 (10:25 -0800)] 
removed need for Roboto font file

11 months agoignored these files
Frank Greguska [Thu, 11 Jan 2018 17:40:30 +0000 (09:40 -0800)] 
ignored these files

11 months agoMerge pull request #3 from fgreg/SDAP-11
fgreg [Thu, 11 Jan 2018 01:18:56 +0000 (17:18 -0800)] 
Merge pull request #3 from fgreg/SDAP-11

SDAP-11 Pull in changes from dataplumber/nexus time-series-tiles

11 months agoPull in changes from dataplumber/nexus time-series-tiles
fgreg [Thu, 11 Jan 2018 01:14:11 +0000 (17:14 -0800)] 
Pull in changes from dataplumber/nexus time-series-tiles

11 months agoMerge pull request #2 from fgreg/SDAP-10
fgreg [Thu, 11 Jan 2018 01:06:16 +0000 (17:06 -0800)] 
Merge pull request #2 from fgreg/SDAP-10

SDAP-10 Pull in changes from dataplumber/nexus image-gen

11 months agoMerge branch 'SDAP-8' into SDAP-10
fgreg [Thu, 11 Jan 2018 01:06:02 +0000 (17:06 -0800)] 
Merge branch 'SDAP-8' into SDAP-10

11 months agoremoving mac file
Frank Greguska [Thu, 11 Jan 2018 01:01:47 +0000 (17:01 -0800)] 
removing mac file

11 months agoPulling in changes from dataplumber/nexus image-gen
fgreg [Thu, 11 Jan 2018 00:31:49 +0000 (16:31 -0800)] 
Pulling in changes from dataplumber/nexus image-gen

11 months agoMerge pull request #1 from fgreg/SDAP-9
fgreg [Thu, 11 Jan 2018 00:55:09 +0000 (16:55 -0800)] 
Merge pull request #1 from fgreg/SDAP-9

SDAP-9 Pulling in changes from dataplumber/s3 branch

11 months agoMerge branch 'master' into s3 2/head
fgreg [Wed, 10 Jan 2018 22:08:59 +0000 (14:08 -0800)] 
Merge branch 'master' into s3

13 months agoSDAP-6 Create CONTRIBUTE.md for all SDAP repositories 1/head
Lewis John McGibbney [Thu, 16 Nov 2017 07:37:14 +0000 (23:37 -0800)] 
SDAP-6 Create CONTRIBUTE.md for all SDAP repositories

13 months agoSDAP-1 Import all code under the SDAP SGA
Lewis John McGibbney [Fri, 27 Oct 2017 22:39:29 +0000 (15:39 -0700)] 
SDAP-1 Import all code under the SDAP SGA