hawq.git
2 years agoHAWQ-1760. datalocality information should be showed by default huor 1491/head
Ruilong Huo [Wed, 11 Mar 2020 09:46:06 +0000 (17:46 +0800)] 
HAWQ-1760. datalocality information should be showed by default

2 years agoHAWQ-1758. Add debug info for udf 1488/head
YangHao1992 [Thu, 21 Nov 2019 08:22:05 +0000 (16:22 +0800)] 
HAWQ-1758. Add debug info for udf

2 years agoHAWQ-1756. add comment for struct TargetSegmentIDMap. 1486/head
ZongtianHou [Tue, 19 Nov 2019 14:08:18 +0000 (22:08 +0800)] 
HAWQ-1756. add comment for struct TargetSegmentIDMap.

2 years agoHAWQ-2742. add comment for enforce virtual segment number. 1485/head
ZongtianHou [Sun, 17 Nov 2019 09:34:13 +0000 (17:34 +0800)] 
HAWQ-2742. add comment for enforce virtual segment number.

2 years agoHAWQ-1754. add comment for max segment number count for random table. 1484/head
ZongtianHou [Wed, 13 Nov 2019 13:04:26 +0000 (21:04 +0800)] 
HAWQ-1754. add comment for max segment number count for random table.

2 years agoHAWQ-1753. location number of external table should less than bucketnum of target... 1483/head
ZongtianHou [Thu, 7 Nov 2019 01:25:31 +0000 (09:25 +0800)] 
HAWQ-1753. location number of external table should less than bucketnum of target table

2 years agoHAWQ-1752. Change .travis
wang.ziming [Tue, 27 Aug 2019 08:50:23 +0000 (16:50 +0800)] 
HAWQ-1752. Change .travis

2 years agoHAWQ-1752. Remove tracked Makefile.global
wang.ziming [Tue, 27 Aug 2019 06:43:34 +0000 (14:43 +0800)] 
HAWQ-1752. Remove tracked Makefile.global

2 years agoHAWQ-1751. Fix Travis CI build
wang.ziming [Wed, 21 Aug 2019 01:22:11 +0000 (09:22 +0800)] 
HAWQ-1751. Fix Travis CI build

2 years agoHAWQ-1750. Fix link of apr in travis
wang.ziming [Mon, 19 Aug 2019 10:12:54 +0000 (18:12 +0800)] 
HAWQ-1750. Fix link of apr in travis

2 years agoHAWQ-1749. Add some license and change pom.xml
wang.ziming [Mon, 19 Aug 2019 02:41:31 +0000 (10:41 +0800)] 
HAWQ-1749. Add some license and change pom.xml

2 years agoHAWQ-1748. Idle executors keep exist instead of quit even when gp_vmem_idle_resource_... 1476/head
Ruilong Huo [Thu, 15 Aug 2019 10:21:11 +0000 (18:21 +0800)] 
HAWQ-1748. Idle executors keep exist instead of quit even when gp_vmem_idle_resource_timeout specified timer expires

2 years agoHAWQ-1747. Change pip downloading
wang.ziming [Thu, 15 Aug 2019 09:14:57 +0000 (17:14 +0800)] 
HAWQ-1747. Change pip downloading

2 years agoHAWQ-1746. fix make install bug when building hawq rpm
oushu1tuyu1 [Thu, 15 Aug 2019 02:55:15 +0000 (10:55 +0800)] 
HAWQ-1746. fix make install bug when building hawq rpm

2 years agoHAWQ-1745. include *.h in storage in contrib's makefile
oushu1tuyu1 [Wed, 14 Aug 2019 15:43:32 +0000 (23:43 +0800)] 
HAWQ-1745. include *.h in storage in contrib's makefile

2 years agoHAWQ-1744. fix lib jvm not found in docker bug
oushu1tuyu1 [Wed, 14 Aug 2019 11:22:21 +0000 (19:22 +0800)] 
HAWQ-1744. fix lib jvm not found in docker bug

2 years agoHAWQ-1743. Fix download link for tomcat 7
oushu1tuyu1 [Tue, 13 Aug 2019 09:56:16 +0000 (17:56 +0800)] 
HAWQ-1743. Fix download link for tomcat 7

2 years agoHAWQ-1742. include *.h in dbcommon in src's makefile
oushu1tuyu1 [Sun, 11 Aug 2019 09:34:24 +0000 (17:34 +0800)] 
HAWQ-1742. include *.h in dbcommon in src's makefile

2 years agoHAWQ-1740. fix build bug when build in ci
oushu1tuyu1 [Sun, 11 Aug 2019 04:10:23 +0000 (12:10 +0800)] 
HAWQ-1740. fix build bug when build in ci

2 years agoHAWQ-1736.Fix bug timestamp orc select
wang.ziming [Mon, 5 Aug 2019 09:57:42 +0000 (17:57 +0800)] 
HAWQ-1736.Fix bug timestamp orc select

2 years agoHAWQ-1739. remove old version protobuf in Dockerfile
oushu1tuyu1 [Sat, 3 Aug 2019 14:19:34 +0000 (22:19 +0800)] 
HAWQ-1739. remove old version protobuf in Dockerfile

2 years agoHAWQ-1737. set with_orc's default value as yes to build orc automatically
oushu1tuyu1 [Fri, 2 Aug 2019 09:57:38 +0000 (17:57 +0800)] 
HAWQ-1737. set with_orc's default value as yes to build orc automatically

2 years agoHAWQ-1735. enable orc feature test
tuyu [Fri, 2 Aug 2019 03:29:46 +0000 (11:29 +0800)] 
HAWQ-1735. enable orc feature test

2 years agoHAWQ-1734. Resolve insert issue in external table of orc
tuyu [Thu, 1 Aug 2019 07:29:47 +0000 (15:29 +0800)] 
HAWQ-1734. Resolve insert issue in external table of orc

2 years agoHAWQ-1733.Resolve select issue in external table of orc
wang.ziming [Mon, 29 Jul 2019 09:12:45 +0000 (17:12 +0800)] 
HAWQ-1733.Resolve select issue in external table of orc

2 years agoHAWQ-1732. Make writable table can read
wang.ziming [Mon, 29 Jul 2019 09:15:16 +0000 (17:15 +0800)] 
HAWQ-1732. Make writable table can read

2 years agoHAWQ-1731. fix orc table create information bug
tuyu [Mon, 29 Jul 2019 08:22:10 +0000 (16:22 +0800)] 
HAWQ-1731. fix orc table create information bug

2 years agoHAWQ-1728. fix dependency path hard code
tuyu [Tue, 23 Jul 2019 08:39:06 +0000 (16:39 +0800)] 
HAWQ-1728. fix dependency path hard code

2 years agoHAWQ-1729.fix memory wrong usage
ZongtianHou [Tue, 23 Jul 2019 02:41:42 +0000 (10:41 +0800)] 
HAWQ-1729.fix memory wrong usage

2 years agoHAWQ-1726. install some necessary software in Dockerfile
tuyu [Wed, 17 Jul 2019 11:59:08 +0000 (19:59 +0800)] 
HAWQ-1726. install some necessary software in Dockerfile

2 years agoHAWQ-1725 install cogapp in Dockerfile
tuyu [Tue, 16 Jul 2019 11:19:29 +0000 (19:19 +0800)] 
HAWQ-1725 install cogapp in Dockerfile

3 years agoHAWQ-1724. Fix download link for tomcat 7 1455/head
Ruilong Huo [Sat, 15 Jun 2019 10:08:50 +0000 (18:08 +0800)] 
HAWQ-1724. Fix download link for tomcat 7

3 years agoHAWQ-1722. Fix core dump due to lock is not released before reporting errors when... 1454/head
Ruilong Huo [Sun, 2 Jun 2019 04:03:39 +0000 (12:03 +0800)] 
HAWQ-1722. Fix core dump due to lock is not released before reporting errors when exceeding MaxAORelSegFileStatus

3 years agoHAWQ-1720. support create table in orc format
tuyu [Wed, 22 May 2019 08:18:41 +0000 (16:18 +0800)] 
HAWQ-1720. support create table in orc format

3 years agoHAWQ-1711. Add feature test for ORC format 1451/head
tuyu [Sun, 12 May 2019 12:17:43 +0000 (20:17 +0800)] 
HAWQ-1711. Add feature test for ORC format

3 years agoHAWQ-1713. Disable TestMbConverter.Canonical in hawq
tuyu [Mon, 13 May 2019 09:13:44 +0000 (17:13 +0800)] 
HAWQ-1713. Disable TestMbConverter.Canonical in hawq

3 years agoHAWQ-1718. Disable agg function unittest in dbcommon
tuyu [Mon, 13 May 2019 09:23:04 +0000 (17:23 +0800)] 
HAWQ-1718. Disable agg function unittest in dbcommon

3 years agoHAWQ-1719. Reopen unittest in hawq 1448/head
tuyu [Mon, 13 May 2019 09:25:10 +0000 (17:25 +0800)] 
HAWQ-1719. Reopen unittest in hawq

3 years agoHAWQ-1717. Fix incorrect seeding for random function 1447/head
Ruilong Huo [Mon, 13 May 2019 02:15:26 +0000 (10:15 +0800)] 
HAWQ-1717. Fix incorrect seeding for random function

3 years agoHAWQ-1716. enable feature-test in docker 1446/head
tuyu [Sun, 12 May 2019 05:01:24 +0000 (13:01 +0800)] 
HAWQ-1716. enable feature-test in docker

3 years agoHAWQ-1712. enable apache hawq build in docker
tuyu [Wed, 8 May 2019 05:48:17 +0000 (13:48 +0800)] 
HAWQ-1712. enable apache hawq build in docker

3 years agoHAWQ-1714. bump hawq version to 2.5 in pom.xml for Apache HAWQ 2.5.0
tuyu [Wed, 8 May 2019 06:53:41 +0000 (14:53 +0800)] 
HAWQ-1714. bump hawq version to 2.5 in pom.xml for Apache HAWQ 2.5.0

3 years agoHAWQ-1715. bump hawq version to 2.5 in contrib/hawq-ambari-plugin/build.properties... 1445/head
tuyu [Wed, 8 May 2019 07:01:13 +0000 (15:01 +0800)] 
HAWQ-1715. bump hawq version to 2.5 in contrib/hawq-ambari-plugin/build.properties for Apache HAWQ 2.5.0

3 years agoHAWQ-1710. Add ORC reader implement in hawq
oushu1tuyu1 [Mon, 29 Apr 2019 03:35:24 +0000 (11:35 +0800)] 
HAWQ-1710. Add ORC reader implement in hawq

3 years agoHAWQ-1695. Optimize hawq compilation structure 1442/head
oushu1tuyu1 [Mon, 29 Apr 2019 10:08:35 +0000 (18:08 +0800)] 
HAWQ-1695. Optimize hawq compilation structure

3 years agoHAWQ-1709. Add ORC reader interface in hawq 1440/head
oushu1tuyu1 [Sun, 28 Apr 2019 10:31:31 +0000 (18:31 +0800)] 
HAWQ-1709. Add ORC reader interface in hawq

3 years agoHAWQ-1707. install dbcommon,univplan,storage once they are made
oushu1tuyu1 [Sun, 28 Apr 2019 02:50:32 +0000 (10:50 +0800)] 
HAWQ-1707. install dbcommon,univplan,storage once they are made

3 years agoHAWQ-1708. Add ORC writer in hawq 1439/head
oushu1tuyu1 [Sun, 28 Apr 2019 06:57:40 +0000 (14:57 +0800)] 
HAWQ-1708. Add ORC writer in hawq

3 years agoHAWQ-1706. enable orc build in makefile
oushu1tuyu1 [Sat, 27 Apr 2019 03:53:28 +0000 (11:53 +0800)] 
HAWQ-1706. enable orc build in makefile

3 years agoHAWQ-1705. fix orc build bug 1436/head
oushu1tuyu1 [Sat, 27 Apr 2019 03:50:48 +0000 (11:50 +0800)] 
HAWQ-1705. fix orc build bug

3 years agoHAWQ-1704. Add ORC protocol validators in hawq
oushu1tuyu1 [Fri, 19 Apr 2019 05:14:41 +0000 (13:14 +0800)] 
HAWQ-1704. Add ORC protocol validators in hawq

3 years agoHAWQ-1703. delete legacy orc in hawq 1434/head
oushu1tuyu1 [Thu, 18 Apr 2019 03:15:47 +0000 (11:15 +0800)] 
HAWQ-1703. delete legacy orc in hawq

3 years agoHAWQ-1702. modify .ans file in hawq feature-test
oushu1tuyu1 [Wed, 17 Apr 2019 11:09:12 +0000 (19:09 +0800)] 
HAWQ-1702. modify .ans file in hawq feature-test

3 years agoHAWQ-1701. enable feature test in apache hawq
oushu1tuyu1 [Tue, 16 Apr 2019 09:45:45 +0000 (17:45 +0800)] 
HAWQ-1701. enable feature test in apache hawq

3 years agoHAWQ-1700. fix hawq cannot be build in linux bug
oushu1tuyu1 [Mon, 15 Apr 2019 09:41:01 +0000 (17:41 +0800)] 
HAWQ-1700. fix hawq cannot be build in linux bug

3 years agoHAWQ-1698. fix univplan cannot be build in linux bug
oushu1tuyu1 [Sun, 14 Apr 2019 12:02:52 +0000 (20:02 +0800)] 
HAWQ-1698. fix univplan cannot be build in linux bug

3 years agoHAWQ-1699. modify json's path in hawq build
oushu1tuyu1 [Sun, 14 Apr 2019 12:21:37 +0000 (20:21 +0800)] 
HAWQ-1699. modify json's path in hawq build

3 years agoHAWQ-1697. fix bug of window func unsafe push down
ZongtianHou [Thu, 11 Apr 2019 02:43:16 +0000 (10:43 +0800)] 
HAWQ-1697. fix bug of window func unsafe push down

3 years agoHAWQ-1694. fix dbcommon,univplan,storage ut couldn't run issue 1426/head
oushu1tuyu1 [Wed, 10 Apr 2019 03:21:46 +0000 (11:21 +0800)] 
HAWQ-1694. fix dbcommon,univplan,storage ut couldn't run issue

3 years agoHAWQ-1694. enable dbcommon,univplan,storage ut 1425/head
oushu1tuyu1 [Tue, 9 Apr 2019 01:59:03 +0000 (09:59 +0800)] 
HAWQ-1694. enable dbcommon,univplan,storage ut

3 years agoapache hawq make dbcommon univplan storage 1423/head
oushu1tuyu1 [Fri, 5 Apr 2019 10:33:59 +0000 (18:33 +0800)] 
apache hawq make dbcommon univplan storage

3 years agoHAWQ-1692. Add the ORC format implementation 1422/head
Ruilong Huo [Sun, 24 Mar 2019 09:15:47 +0000 (17:15 +0800)] 
HAWQ-1692. Add the ORC format implementation

3 years agoHAWQ-1691. Add universal plan used by ORC format 1421/head
Ruilong Huo [Sun, 24 Mar 2019 07:52:19 +0000 (15:52 +0800)] 
HAWQ-1691. Add universal plan used by ORC format

3 years agoHAWQ-1690. Add common utility used by ORC foramt 1420/head
Ruilong Huo [Fri, 22 Mar 2019 13:34:52 +0000 (21:34 +0800)] 
HAWQ-1690. Add common utility used by ORC foramt

3 years agoHAWQ-1689. Add test for cloud environment support 1419/head
Ruilong Huo [Mon, 18 Mar 2019 07:43:11 +0000 (15:43 +0800)] 
HAWQ-1689. Add test for cloud environment support

3 years agoupdate hawq help
daixiang0 [Tue, 11 Dec 2018 09:44:15 +0000 (17:44 +0800)] 
update hawq help

Signed-off-by: daixiang0 <764524258@qq.com>
3 years agoHAWQ-1687. Fix travis broken due to missing apr-1-config
Ruilong Huo [Fri, 15 Mar 2019 13:51:53 +0000 (21:51 +0800)] 
HAWQ-1687. Fix travis broken due to missing apr-1-config

3 years agoHAWQ-1688. Fix apache license header for cloud support
Ruilong Huo [Sat, 16 Mar 2019 01:41:17 +0000 (09:41 +0800)] 
HAWQ-1688. Fix apache license header for cloud support

3 years agoHAWQ-1685. Fix bug in CheckUserExistOnCloud
wcl14 [Mon, 2 Jul 2018 03:45:48 +0000 (11:45 +0800)] 
HAWQ-1685. Fix bug in CheckUserExistOnCloud

3 years agoHAWQ-1684. Add connection close in header for cloud rest
wcl14 [Fri, 29 Jun 2018 03:18:36 +0000 (11:18 +0800)] 
HAWQ-1684. Add connection close in header for cloud rest

3 years agoHAWQ-1683. Using DEBUG3 instead INFO log in cloudrest.c
wcl14 [Thu, 28 Jun 2018 03:24:48 +0000 (11:24 +0800)] 
HAWQ-1683. Using DEBUG3 instead INFO log in cloudrest.c

3 years agoHAWQ-1682. Remove useless log for cloud authentication
wcl14 [Tue, 29 May 2018 08:46:53 +0000 (16:46 +0800)] 
HAWQ-1682. Remove useless log for cloud authentication

3 years agoHAWQ-1681. Support manage user in cloud
wcl14 [Tue, 29 May 2018 08:26:22 +0000 (16:26 +0800)] 
HAWQ-1681. Support manage user in cloud

3 years agoHAWQ-1679. Fix travis failure due to missing apache license information 1417/head
Ruilong Huo [Thu, 14 Mar 2019 09:37:02 +0000 (17:37 +0800)] 
HAWQ-1679. Fix travis failure due to missing apache license information

3 years agoHAWQ-1679. Remove unnecessary error in hdfsprotocol_validate function
Ruilong Huo [Thu, 14 Mar 2019 03:51:13 +0000 (11:51 +0800)] 
HAWQ-1679. Remove unnecessary error in hdfsprotocol_validate function

3 years agoHAWQ-1679. Fix typo of orc in contrib
Ruilong Huo [Thu, 14 Mar 2019 03:48:11 +0000 (11:48 +0800)] 
HAWQ-1679. Fix typo of orc in contrib

3 years agoHAWQ-1678. Add missing header file for libyarn client
Ruilong Huo [Thu, 14 Mar 2019 03:23:19 +0000 (11:23 +0800)] 
HAWQ-1678. Add missing header file for libyarn client

3 years agoHAWQ-1677. Disable orc compilation until it is available
Ruilong Huo [Sat, 9 Mar 2019 10:01:43 +0000 (18:01 +0800)] 
HAWQ-1677. Disable orc compilation until it is available

3 years agoUpdate PXF README to include profiling info
Shivram Mani [Fri, 25 Jan 2019 19:25:15 +0000 (11:25 -0800)] 
Update PXF README to include profiling info

3 years agoHAWQ-1674. Add support for compiling CPP projects in orc_format. 1400/head
ion2014 [Fri, 9 Nov 2018 07:19:11 +0000 (15:19 +0800)] 
HAWQ-1674. Add support for compiling CPP projects in orc_format.
add cwrapper.cpp to generate interface code for C/C++ link.
changed Makefile to support cpp compiling.

3 years agoHAWQ-1671. Add compile support for orc formatter and wrapper
ion2014 [Thu, 8 Nov 2018 07:27:36 +0000 (15:27 +0800)] 
HAWQ-1671. Add compile support for orc formatter and wrapper
Changed Makefile in contrib/orc
Add folder format and cwrapper for split compilation
Add dummy code for testing the Makefile

3 years agoHAWQ-1658. Easy Hawq and PXF initialization with docker.
Mariano Gonzalez [Fri, 31 Aug 2018 21:27:29 +0000 (16:27 -0500)] 
HAWQ-1658. Easy Hawq and PXF initialization with docker.

Added new options to build and run HAWQ and PXF in docker.
Check options in contrib/hawq-docker/README.md for details.

3 years agoHAWQ-1617. Fix function calls from unit tests
Ivan Leskin [Mon, 3 Sep 2018 09:47:48 +0000 (12:47 +0300)] 
HAWQ-1617. Fix function calls from unit tests

Fix calls to 'pxf_free_expression_items_list' from unit tests (the 2nd parameter is always true)
'enrich_trivial_expression' was renamed to 'add_extra_expression_items'. The necessary changes in tests are made by this commit

3 years agoHAWQ-1605. Support INSERT in PXF JDBC plugin
Ivan Leskin [Mon, 5 Mar 2018 14:19:26 +0000 (17:19 +0300)] 
HAWQ-1605. Support INSERT in PXF JDBC plugin

(closes #1353)

Fix incorrect TIMESTAMP handling

PXF JDBC plugin update

* Add support for INSERT queries:
* The INSERT queries are processed by the same classes as the SELECT queries;
* INSERTs are processed by the JDBC PreparedStatement;
* INSERTs support batching (by means of JDBC);

* Minor changes in WhereSQLBuilder and JdbcPartitionFragmenter:
* Removed 'WHERE 1=1';
* The same pattern of spaces around operators everywhere ('a = b', not 'a=b');
* JdbcPartitionFragmenter.buildFragmenterSql() made static to avoid extra checks of InputData (proposed by @sansanichfb);

* Refactoring and some microoptimizations;

PXF JDBC refactoring

* The README.md is completely rewritten;

* Lots of changes in comments and javadoc comments;

* Code refactoring and minor changes in codestyle

Fixes proposed by @sansanichfb

Add DbProduct for Microsoft SQL Server

Notes on consistency in README and errors

* Add an explicit note on consistency of INSERT queries (it is not guaranteed).

* Change error message on INSERT failure

* Minor corrections of README

The fixes were proposed by @sansanichfb

Improve WhereSQLBuilder

* Add support of TIMESTAMP values;

* Add support of operations <>, LIKE, IS NULL, IS NOT NULL.

Fix proposed by @sansanichfb

Throw an exception when trying to open an already open connection when writing to an external database using `openForWrite()`.

Although the behaviour is different in case of `openForRead()`, it does not apply here. The second call to `openForWrite()` could be made from another thread, and that would result in a race: the `PreparedStatement` we use to write to an external database is the same object for all threads, and the procedure `writeNextObject()` is not `synchronized` (or "protected" some other way).

Simplify logging; BatchUpdateException

Simplify logging so that the logs produced by pxf-jdbc do not grow too big in case DEBUG is enabled (the removed logging calls provide the field types and names, and in most cases they are the same as in the data provided. The exceptions are still being logged).

Add processing of BatchUpdateException, so that the real cause of an exception is returned to the user.

PXF JDBC thread pool support

Implement support of multi-threaded processing of INSERT queries, using a thread pool. To use the feature, set the parameter POOL_SIZE in the LOCATION clause of an external table (<1: Pool size is equal to a number of CPUs available to JVM; =1: Disable thread pool; >1: Use the given size of a pool.

Not all operations are processed by pool threads: pool threads only execute() the queries, but they do not fill the PreparedStatement from OneRow.

Redesign connection pooling

* Redesign connection pooling: move OneRow objects processing to threads from the pool. This decreases the load of a single-thread part of PXF;

* Introduce WriterCallable & related. This significantly simplifies the code of JdbcAccessor and allows to introduce new methods of processing INSERT queries with ease and enables fast hardcode tweaks for the same purpose.

* Add docs on thread pool feature

Support long values in PARTITION clause

Support values of Java primitive type 'long' in PARTITION clause (both for RANGE and INTERVAL variables).

* Modify JdbcPartitionFragmenter (convert all int variables to long)
* Move parsing of INTERVAL values for PARTITION_TYPE "INT" to class constructor (and add a parse exception handler)
* Simplify ByteUtil (remove methods to deal with values of type 'int')
* Update JdbcPartitionFragmenterTest
* Minor changes in comments

Fix pxf-profiles-default.xml

Remove ampersand from a description of JDBC profile from pxf-profiles-default.xml

Remove explicit throws of IllegalArgumentException

Remove explicit references to 'IllegalArgumentException', as the caller is probably unable to recover from them.
'IllegalStateException' is left unchanged, as it is thrown when the caller must perform an action that will resolve the problem ('WriterCallable' is full).

Other runtime exceptions are explicitly listed in function definitions as before; their causes are usually known to the caller, so it could do something about them or at least send a more meaningful message about the error cause to the user.

Proposed by Alex Denissov <adenissov@pivotal.io>

Simplify isCallRequired()

Make function 'isCallRequired()' body a one-line expression in all implementations of 'WriterCallable'.

Proposed by Alex Denissov <adenissov@pivotal.io>

Remove rollback and change BATCH_SIZE logic

Remove calls to 'tryRollback()' and all processing of rollbacks in INSERT queries.
The reason for the change is that rollback is effective for only one case: INSERT is performed from one PXF segment that uses one thread to perform that INSERT, and the external database supports transactions. In most cases, there are more than one PXF segment that performs INSERT, and rollback is of no use then.
On the other hand, rollback logic is cumbersome and notably increases code complexity.

Due to the removal of rollback, there is no longer a need to keep BATCH_SIZE infinite as often as possible (when BATCH_SIZE is infinite, the number of scenarious of rollback() failing is lower (but this number is not zero)).
Thus, setting a recommended (https://docs.oracle.com/cd/E11882_01/java.112/e16548/oraperf.htm#JJDBC28754) value makes sense.
The old logic of infinite batch size also remains active.

Modify README.md: minor corrections, new BATCH_SIZE logic

Proposed by Alex Denissov <adenissov@pivotal.io>

Change BATCH_SIZE logic

* Modify BATCH_SIZE parameter processing according to new proposals https://github.com/apache/hawq/pull/1353#discussion_r214413534
* Update README.md
* Restore fallback to non-batched INSERTs in case the external database (or JDBC connector) does not support batch updates

Proposed by Alex Denissov <adenissov@pivotal.io>
Proposed by Dmitriy Pavlov <pavlovdmst@gmail.com>

Modify processing of BATCH_SIZE parameter

Modify BATCH_SIZE parameter processing according to the proposal https://github.com/apache/hawq/pull/1353#discussion_r215023775:
* Update allowed values of BATCH_SIZE and their meanings
* Introduce explicit flag of presentness of a BATCH_SIZE parameter
* Introduce DEFAULT_BATCH_SIZE constant in JdbcPlugin
* Move processing of BATCH_SIZE values to JdbcAccessor
* Update README.md

Proposed by @divyabhargov, @denalex

Fix column type for columns converted to TEXT

Modify column type processing so that the column type is set correctly for fields that:
* Are represented as columns of type TEXT by GPDBWritable, but whose actual type is different
* Contain NULL value

Before, the column type code was not set correctly for such columns due to a check of NULL field value.

Proposed and authored by @divyabhargov

removed parseUnsignedInt

3 years agoHAWQ-1647. Update HAWQ version to 2.4.0.0
rlei [Tue, 7 Aug 2018 09:43:23 +0000 (17:43 +0800)] 
HAWQ-1647. Update HAWQ version to 2.4.0.0

- Update HAWQ version from 2.3.0.0 to 2.4.0.0.
- Removed incubating/incubator strings from HAWQ versions.

3 years agoFix handling of implicit AND expressions
Ivan Leskin [Fri, 31 Aug 2018 13:37:13 +0000 (16:37 +0300)] 
Fix handling of implicit AND expressions

Co-authored-by: Alex Denissov <adenissov@pivotal.io>
Co-authored-by: Shivram Mani <smani@pivotal.io>
Modify 'pxf_make_expression_items_list()' and 'add_extra_and_expression_items()' so that the latter procedure processes the expression list completely, without any interference with 'pxf_make_expression_items_list()'.

This is identical to https://github.com/greenplum-db/gpdb/commit/3ac93fb4259436c87b6c1705bdb05c003ca93423; see also https://github.com/greenplum-db/gpdb/pull/5470.

3 years agoFix implicit AND expression addition
Ivan Leskin [Fri, 25 May 2018 13:19:36 +0000 (16:19 +0300)] 
Fix implicit AND expression addition

Fix implicit addition of extra 'BoolExpr' to a list of expression items.

Before, there was a check that the expression items list did not contain logical operators
(and if it did, no extra implicit AND operators were added).

This behaviour is incorrect. Consider the following query:

SELECT * FROM table_ex WHERE bool1=false AND id1=60003;

Such query will be translated as a list of three items: 'BoolExpr', 'Var' and 'OpExpr'.
Due to the presence of a 'BoolExpr', extra implicit 'BoolExpr' will not be added, and
we get an error "stack is not empty ...".

This commit changes the signatures of some internal functions in pxffilters.c to fix this error.
We pass a number of required extra 'BoolExpr's to 'add_extra_and_expression_items'.

As 'BoolExpr's of different origin may be present in the list of expression items,
the mechanism of freeing the BoolExpr node changes.

The current mechanism of implicit AND expressions addition is suitable only before
OR operators are introduced (in that case, we will have to add ANDs to different parts
of a list, not just the end, as done now).

3 years agoHAWQ-1629. Add ORC format using pluggable storage framework.
oushu1wangziming1 [Fri, 24 Aug 2018 02:40:02 +0000 (10:40 +0800)] 
HAWQ-1629. Add ORC format using pluggable storage framework.

3 years agoHAWQ-1656. Fix build issues of Bison dependency.
Fenggang [Tue, 28 Aug 2018 00:56:52 +0000 (20:56 -0400)] 
HAWQ-1656. Fix build issues of Bison dependency.

3 years agoHAWQ-1651. Fix some bugs in bulid and unittest.
oushu1wangziming1 [Tue, 21 Aug 2018 08:14:31 +0000 (16:14 +0800)] 
HAWQ-1651. Fix some bugs in bulid and unittest.

3 years agoHAWQ-1655. Fix doc and makefile issue for hawq-docker
Fenggang [Mon, 13 Aug 2018 07:00:50 +0000 (03:00 -0400)] 
HAWQ-1655. Fix doc and makefile issue for hawq-docker

3 years agoHAWQ-1652. Update PXF keytab path to use PXF_HOME 1392/head
Francisco Guerrero [Sat, 18 Aug 2018 00:26:44 +0000 (17:26 -0700)] 
HAWQ-1652. Update PXF keytab path to use PXF_HOME

- Updates `PXF_KEYTAB` path to live inside `${PXF_HOME}/conf` and
  `PXF_PRINCIPAL` to default to gpadmin.
- Fixes nanoseconds to milliseconds conversion in UGICache.

Co-authored-by: Divya Bhargov <dbhargov@pivotal.io>
Co-authored-by: Francisco Guerrero <aguerrero@pivotal.io>
3 years agoHAWQ-1628. Fix a bug of cassert mode
oushu1wangziming1 [Thu, 16 Aug 2018 05:25:21 +0000 (13:25 +0800)] 
HAWQ-1628. Fix a bug of cassert mode

3 years agoHAWQ-1628. Add hdfs protocol for pluggable storage framework
oushu1wangziming1 [Tue, 17 Jul 2018 07:43:36 +0000 (15:43 +0800)] 
HAWQ-1628. Add hdfs protocol for pluggable storage framework

3 years agoHAWQ-1650. Fix compilation issue in Java 7 1390/head
Francisco Guerrero [Mon, 13 Aug 2018 21:43:06 +0000 (14:43 -0700)] 
HAWQ-1650. Fix compilation issue in Java 7

3 years agoHAWQ-1644. Make delegation token optional in PXF
Francisco Guerrero [Mon, 13 Aug 2018 21:30:28 +0000 (14:30 -0700)] 
HAWQ-1644. Make delegation token optional in PXF

Make delegation token property optional for PXF when used against secure
hadoop. Delegation token is a complementary means for authentication
with hadoop along with Kerberos. There is not much value in using the
delegation token with masterless and segment only PXF architecture, as
each pxf jvm will need to establish the token anyway. Simply
authenticating with Keberos should be good enough in such a deployment
mode.

Co-authored-by: Alexander Denissov <adenissov@pivotal.io>
Co-authored-by: Francisco Guerrero <aguerrero@pivotal.io>
Co-authored-by: Shivram Mani <smani@pivotal.io>
3 years agoHAWQ-1649. Add parallel true and daemon to gradle properties 1389/head
Francisco Guerrero [Mon, 13 Aug 2018 18:45:14 +0000 (11:45 -0700)] 
HAWQ-1649. Add parallel true and daemon to gradle properties

3 years agoHAWQ-1645. Use the latest available version of gradle (4.9) 1385/head
Francisco Guerrero [Tue, 7 Aug 2018 22:24:07 +0000 (15:24 -0700)] 
HAWQ-1645. Use the latest available version of gradle (4.9)

3 years agoHAWQ-1646. Fixes travis CI issues
Francisco Guerrero [Tue, 7 Aug 2018 06:14:27 +0000 (23:14 -0700)] 
HAWQ-1646. Fixes travis CI issues

(closes #1386)

3 years agoRevert "HAWQ-1645. Remove autogeneration of version number from pxf/build.gradle"
Lav Jain [Fri, 3 Aug 2018 19:08:42 +0000 (12:08 -0700)] 
Revert "HAWQ-1645. Remove autogeneration of version number from pxf/build.gradle"

This reverts commit 67b64bc93031a6ef86b3ac33a732ce02b162d71b.