incubator-doris.git
2 hours ago[website] fix doris website with no link to the Privacy Policy. (#9665) master
Zhengguo Yang [Wed, 18 May 2022 14:49:49 +0000 (22:49 +0800)] 
[website] fix doris website with no link to the Privacy Policy. (#9665)

All websites must link to the Privacy Policy

2 hours ago[fix](planner) unnecessary cast will be added on children in CaseExpr sometimes ...
morrySnow [Wed, 18 May 2022 14:44:51 +0000 (22:44 +0800)] 
[fix](planner) unnecessary cast will be added on children in CaseExpr sometimes (#9600)

unnecessary cast will be added on children in CaseExpr because use symbolized equal to compare to `Expr`'s type.
it will lead to expression compare mistake and then lead to expression substitute failed when use `ExprSubstitutionMap`

7 hours ago[regression test] Add compaction regression test case for different data models ...
Gabriel [Wed, 18 May 2022 09:12:20 +0000 (17:12 +0800)] 
[regression test] Add compaction regression test case for different data models (#9660)

10 hours ago[fix](storage) low_cardinality_optimize core dump when is null predicate (#9586)
ZenoYang [Wed, 18 May 2022 06:57:13 +0000 (14:57 +0800)] 
[fix](storage) low_cardinality_optimize core dump when is null predicate (#9586)

Issue Number: close #9555
Make the last value of the dictionary null, when ColumnDict inserts a null value,
add the encoding corresponding to the last value of the dictionary·

13 hours ago[fixbug](vec-load) fix core of segment_writer while it is not thread-safe (#9569)
yixiutt [Wed, 18 May 2022 03:29:15 +0000 (11:29 +0800)] 
[fixbug](vec-load) fix core of segment_writer while it is not thread-safe (#9569)

introduce in stream-load-vec #9280, it will cause multi-thread
operate to same segment_write cause BetaRowset enable multi-thread
of memtable flush, memtable flush call rowset_writer.add_block, it
use member variable _segment_writer to write, so it will cause
multi-thread in segment write.

Co-authored-by: yixiutt <yixiu@selectdb.com>
14 hours ago[improment](planner) push down predicate past two phase aggregate (#9498)
morrySnow [Wed, 18 May 2022 02:09:39 +0000 (10:09 +0800)] 
[improment](planner) push down predicate past two phase aggregate (#9498)

Push down predicate past aggregate cannot push down predicate past 2 phase aggregate.

origin plan is like this:
```
second phase agg (conjuncts on olap scan node tuples)
|
first phase agg
|
olap scan node
```
should be optimized to
```
second phase agg
|
first phase agg
|
olap scan node (conjuncts on olap scan node tuples)
```

17 hours ago[bug] (init) Java version check fail (#9607)
Hui Tian [Tue, 17 May 2022 23:47:03 +0000 (07:47 +0800)] 
[bug] (init) Java version check fail (#9607)

17 hours ago[BUG] fix information_schema.columns results not correctly on vec engine (#9612)
camby [Tue, 17 May 2022 23:44:32 +0000 (07:44 +0800)] 
[BUG] fix information_schema.columns results not correctly on vec engine (#9612)

* VSchemaScanNode get_next bugfix

* add regression-test case for VSchemaScanNode

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
17 hours ago[regression test] add some case for json load regression test (#9614)
carlvinhust2012 [Tue, 17 May 2022 23:43:51 +0000 (07:43 +0800)] 
[regression test] add some case for json load regression test (#9614)

Co-authored-by: hucheng01 <hucheng01@baidu.com>
17 hours ago[clang build]fix clang compile error (#9615)
Pxl [Tue, 17 May 2022 23:42:31 +0000 (07:42 +0800)] 
[clang build]fix clang compile error (#9615)

17 hours ago[Improvement][ASAN] make BE can exit normally and ASAN memory leak checking work...
jacktengg [Tue, 17 May 2022 23:40:57 +0000 (07:40 +0800)] 
[Improvement][ASAN] make BE can exit normally and ASAN memory leak checking work (#9620)

26 hours ago[Improvement] reduce string size in serialization (#9550)
Gabriel [Tue, 17 May 2022 14:38:34 +0000 (22:38 +0800)] 
[Improvement] reduce string size in serialization (#9550)

26 hours ago[config] Remove some old config and session variable (#9495)
Mingyu Chen [Tue, 17 May 2022 14:37:11 +0000 (22:37 +0800)] 
[config] Remove some old config and session variable (#9495)

1. Remove session variable "enable_lateral_view"
2. Remove Fe config: enable_materialized_view
3. Remove Fe config: enable_create_sync_job
4. Fe config dynamic_partition_enable is only used for disable dynamic partition scheduler.

26 hours ago[fix] fix bug that replica can not be repaired duo to DECOMMISSION state (#9424)
Mingyu Chen [Tue, 17 May 2022 14:36:30 +0000 (22:36 +0800)] 
[fix] fix bug that replica can not be repaired duo to DECOMMISSION state (#9424)

Reset state of replica which state are in DECOMMISSION after finished scheduling.

28 hours ago[feature] Add StoragePolicyResource for Remote Storage (#9554)
pengxiangyu [Tue, 17 May 2022 12:17:33 +0000 (20:17 +0800)] 
[feature] Add StoragePolicyResource for Remote Storage (#9554)

Add StoragePolicyResource for Remote Storage

29 hours ago[feature] group_concat support distinct (#9576)
Stalary [Tue, 17 May 2022 11:29:47 +0000 (19:29 +0800)] 
[feature] group_concat support distinct (#9576)

29 hours ago[code format]Upgrade clang-format in BE Code Formatter from 8 to 13 (#9602)
Adonis Ling [Tue, 17 May 2022 11:28:15 +0000 (19:28 +0800)] 
[code format]Upgrade clang-format in BE Code Formatter from 8 to 13 (#9602)

29 hours ago[doc]modified the spark-load doc (#9605)
zy-kkk [Tue, 17 May 2022 11:27:02 +0000 (19:27 +0800)] 
[doc]modified the spark-load doc (#9605)

29 hours ago[doc]add largeint doc (#9609)
jiafeng.zhang [Tue, 17 May 2022 11:26:45 +0000 (19:26 +0800)] 
[doc]add largeint doc (#9609)

add largeint doc

34 hours ago[Bug][Vectorized] Fix insert bimmap column with nullable column (#9408)
HappenLee [Tue, 17 May 2022 06:42:20 +0000 (14:42 +0800)] 
[Bug][Vectorized] Fix insert bimmap column with nullable column (#9408)

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
35 hours ago[Bug][Vectorized] Fix BE crash with delete condition and enable_storage_vectorization...
HappenLee [Tue, 17 May 2022 06:01:22 +0000 (14:01 +0800)] 
[Bug][Vectorized] Fix BE crash with delete condition and enable_storage_vectorization (#9547)

Co-authored-by: lihaopeng <lihaopeng@baidu.com>
37 hours ago[fix](storage-vectorized) fix VMergeIterator core dump (#9564) opentelemetry_dev
Xin Liao [Tue, 17 May 2022 03:58:59 +0000 (11:58 +0800)] 
[fix](storage-vectorized) fix VMergeIterator core dump (#9564)

It could be re appeared on rowset with many segment, it means segment overlap. Maybe could not reappear it easily.

37 hours ago[feature-wip](hudi) Step1: Support create hudi external table (#9559)
dujl [Tue, 17 May 2022 03:30:23 +0000 (11:30 +0800)] 
[feature-wip](hudi) Step1: Support create hudi external table (#9559)

support create hudi table
support show create table for hudi table

### Design
1. create hudi table without schema(recommanded)
```sql
    CREATE [EXTERNAL] TABLE table_name
    ENGINE = HUDI
    [COMMENT "comment"]
    PROPERTIES (
    "hudi.database" = "hudi_db_in_hive_metastore",
    "hudi.table" = "hudi_table_in_hive_metastore",
    "hudi.hive.metastore.uris" = "thrift://127.0.0.1:9083"
    );
```

2. create hudi table with schema
```sql
    CREATE [EXTERNAL] TABLE table_name
    [(column_definition1[, column_definition2, ...])]
    ENGINE = HUDI
    [COMMENT "comment"]
    PROPERTIES (
    "hudi.database" = "hudi_db_in_hive_metastore",
    "hudi.table" = "hudi_table_in_hive_metastore",
    "hudi.hive.metastore.uris" = "thrift://127.0.0.1:9083"
    );
```
When create hudi table with schema, the columns must exist in corresponding table in hive metastore.

39 hours ago[feature-wip](parquet-vec) Support parquet scanner in vectorized engine (#9433)
yinzhijian [Tue, 17 May 2022 01:37:17 +0000 (09:37 +0800)] 
[feature-wip](parquet-vec) Support parquet scanner in vectorized engine (#9433)

40 hours ago[docs]Modifide flink-doris-connector.md (#9595)
zy-kkk [Tue, 17 May 2022 01:01:14 +0000 (09:01 +0800)] 
[docs]Modifide flink-doris-connector.md (#9595)

2 days ago[chore] Fix compilation errors reported by clang (#9584)
Adonis Ling [Mon, 16 May 2022 14:36:16 +0000 (22:36 +0800)] 
[chore] Fix compilation errors reported by clang (#9584)

2 days ago[fix](planner)VecNotImplException thrown when query need rewrite and some slot cannot...
morrySnow [Mon, 16 May 2022 14:34:02 +0000 (22:34 +0800)] 
[fix](planner)VecNotImplException thrown when query need rewrite and some slot cannot changed to nullable (#9589)

2 days ago[Feature](Nereids) Data structure of comparison predicate (#9506)
EmmyMiao87 [Mon, 16 May 2022 07:01:13 +0000 (15:01 +0800)] 
[Feature](Nereids) Data structure of comparison predicate (#9506)

1. The data structure of the comparison expression
2. Refactored the inheritance and implementation relationship of tree node

```
        +-- ---- ---- ---+- ---- ---- ---- ---+- ---- ----- ---- ----TreeNode-----------------+
        |                |                    |                                               |
                                                                                              |
        |                |                    |                                               |
                                                                                              v
        v                v                    v                                           Abstract Tree Node
    Leaf Node        Unary Node          Binary Node                              +--------          ---------+
        |                |                    |                                   |        (children)         |
                                                                                  |                           |
        v                v                    v                                   v                           v
Leaf Expression   Unary Expression      Binary Expression              +------Expression----+           Plan Node
        |                |                    |                        |                    |
                                                                       |                    |
        |                |                    |                        v                    v
        |                |                    +- ---- ---- -----> Comparison Predicate     Named Expr
                                                                                       +----   -------+
        |                |                                                             v              v
        |                +- -- --- --- --- --- --- --- --- --- --- --- --- --- ---> Alias Expr      Slot
                                                                                                      ^
        |                                                                                             |
        |                                                                                             |
        +---- --- ---- ------ ---- ------- ------ ------- --- ------ ------ ----- ---- ----- ----- ---+
```

3 days ago[fix](function) fix last_value get wrong result when have order by clause (#9247)
zhangstar333 [Sun, 15 May 2022 15:56:01 +0000 (23:56 +0800)] 
[fix](function) fix last_value get wrong result when have order by clause (#9247)

3 days ago[enhancement][betarowset]optimize lz4 compress and decompress speed by reusing contex...
Kang [Sun, 15 May 2022 13:18:32 +0000 (21:18 +0800)] 
[enhancement][betarowset]optimize lz4 compress and decompress speed by reusing context (#9566)

3 days ago[Doc]Add show tables help documentation (#9568)
jiafeng.zhang [Sun, 15 May 2022 02:18:33 +0000 (10:18 +0800)] 
[Doc]Add show tables help documentation (#9568)

3 days ago[doc]fix doc typo in data-model and date data type (#9571)
dataalive [Sun, 15 May 2022 02:17:46 +0000 (10:17 +0800)] 
[doc]fix doc typo in data-model and date data type (#9571)

4 days agoADD: 补充idea开发文档,添加help-resource.zip的生成步骤 (#9561)
Stalary [Sat, 14 May 2022 11:04:11 +0000 (19:04 +0800)] 
ADD: 补充idea开发文档,添加help-resource.zip的生成步骤 (#9561)

4 days ago[bugfix](vectorized) vectorized write: invalid memory access caused by podarray resiz...
jacktengg [Sat, 14 May 2022 11:03:51 +0000 (19:03 +0800)] 
[bugfix](vectorized) vectorized write: invalid memory access caused by podarray resize (#9556)

4 days ago[refactor](es) Clean es tcp scannode and related thrift definitions (#9553)
yiguolei [Sat, 14 May 2022 02:03:55 +0000 (10:03 +0800)] 
[refactor](es) Clean es tcp scannode and related thrift definitions (#9553)

PaloExternalSourcesService is designed for es_scan_node using tcp protocol.
But es tcp protocol need deploy a tcp jar into es code. Both es version and lucene version are upgraded,
and the tcp jar is not maintained any more.

So that I remove all the related code and thrift definitions.

4 days ago[fix](lateral-view) Error view includes lateral view (#9530)
EmmyMiao87 [Sat, 14 May 2022 01:57:08 +0000 (09:57 +0800)] 
[fix](lateral-view) Error view includes lateral view (#9530)

Fixed #9529

When the lateral view based on a inline view which belongs to a view,
Doris could not resolve the column of lateral view in query.
When a query uses a view, it mainly refers to the string representation of the view.
That is, if the view's string representation is wrong, the view is wrong.
The string representation of the inline view lacks the handling of the lateral view.
This leads to query errors when using such views.
This PR mainly fixes the string representation of inline views.

4 days ago[fix](Function) fix case when function return null with abs function (#9493)
zhangstar333 [Sat, 14 May 2022 01:50:45 +0000 (09:50 +0800)] 
[fix](Function) fix case when function return null with abs function (#9493)

4 days ago[feature] add vectorized vjson_scanner (#9311)
carlvinhust2012 [Sat, 14 May 2022 01:50:05 +0000 (09:50 +0800)] 
[feature] add vectorized vjson_scanner (#9311)

This pr is used to add the vectorized vjson_scanner, which can support vectorized json import in stream load flow.

5 days ago[doc] [Improved] The flink connector documentation is perfect (#9528)
LOVEGISER [Fri, 13 May 2022 08:22:54 +0000 (16:22 +0800)] 
[doc] [Improved] The flink connector documentation is perfect (#9528)

Co-authored-by: 王磊 <lei.wang@unidt.com>
5 days ago[Improvement] remove unnecessary memcpy in OlapBlockDataConvertor (#9491)
Gabriel [Fri, 13 May 2022 08:13:07 +0000 (16:13 +0800)] 
[Improvement] remove unnecessary memcpy in OlapBlockDataConvertor (#9491)

* [Improvement] remove unnecessary memcpy in OlapBlockDataConvertor

5 days ago[feature-wip](array-type) array_contains support more nested data types (#9170)
camby [Fri, 13 May 2022 04:42:40 +0000 (12:42 +0800)] 
[feature-wip](array-type) array_contains support more nested data types (#9170)

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
5 days ago[doc]Add ARM architecture compilation tutorial content (#9535)
FreeOnePlus [Fri, 13 May 2022 02:24:19 +0000 (10:24 +0800)] 
[doc]Add ARM architecture compilation tutorial content (#9535)

Co-authored-by: manyi <fop@freeoneplus.com>
5 days agoClean the version.sh file before build, otherwise the version information in the...
Dongyang Li [Fri, 13 May 2022 02:23:44 +0000 (10:23 +0800)] 
Clean the version.sh file before build, otherwise the version information in the binary package produced by this compilation is still the commit id of the last time. (#9534)

Co-authored-by: stephen <hello-stephen@qq.com>
5 days ago[fix][vectorized-storage] did not check column writer's write status
yinzhijian [Fri, 13 May 2022 01:57:33 +0000 (09:57 +0800)] 
[fix][vectorized-storage] did not check column writer's write status

6 days ago[bugfix](load) fix coredump in ordinal index flush (#9518)
yixiutt [Thu, 12 May 2022 13:10:49 +0000 (21:10 +0800)] 
[bugfix](load) fix coredump in ordinal index flush (#9518)

commit #9123 introduce the bug. bitshuffle page return error when
page is full, so scalar column write cannot switch to next page, which make
ordinal index is null when flush.

All page builder should return ok when page full, and column writer procedure
shoud be append_data, check is_page_full, switch to next page

Co-authored-by: yixiutt <yixiu@selectdb.com>
6 days ago[style](java) format fe code with some check rules (#9460)
morrySnow [Thu, 12 May 2022 12:14:38 +0000 (20:14 +0800)] 
[style](java) format fe code with some check rules (#9460)

Issue Number: close #9403

set below rules' severity to error and format code according check info.
a. Merge conflicts unresolved
b. Avoid using corresponding octal or Unicode escape
c. Avoid Escaped Unicode Characters
d. No Line Wrap
e. Package Name
f. Type Name
g. Annotation Location
h. Interface Type Parameter
i. CatchParameterName
j. Pattern Variable Name
k. Record Component Name
l. Record Type Parameter Name
m. Method Type Parameter Name
n. Redundant Import
o. Custom Import Order
p. Unused Imports
q. Avoid Star Import
r. tab character in file
s. Newline At End Of File
t. Trailing whitespace found

6 days ago[regression test]add the regression test for json load (#9517)
carlvinhust2012 [Thu, 12 May 2022 08:08:03 +0000 (16:08 +0800)] 
[regression test]add the regression test for json load (#9517)

Co-authored-by: hucheng01 <hucheng01@baidu.com>
6 days agoIncorrect sequence numbers in revision documents. (#9496)
smallhibiscus [Thu, 12 May 2022 07:44:41 +0000 (15:44 +0800)] 
Incorrect sequence numbers in revision documents. (#9496)

Co-authored-by: smallhibiscus <844981280>
6 days ago[Bug] (load) Broker load kerberos auth fail (#9494)
Hui Tian [Thu, 12 May 2022 07:43:29 +0000 (15:43 +0800)] 
[Bug] (load) Broker load kerberos auth fail (#9494)

6 days ago[fix](storage) fix core for string predicate in storage layer (#9500)
wangbo [Thu, 12 May 2022 07:41:39 +0000 (15:41 +0800)] 
[fix](storage) fix core for string predicate in storage layer (#9500)

Co-authored-by: Wang Bo <wangbo36@meituan.com>
6 days ago[refactor] Check status precise_code instead of construct OLAPInternalError (#9514)
plat1ko [Thu, 12 May 2022 07:39:29 +0000 (15:39 +0800)] 
[refactor] Check status precise_code instead of construct OLAPInternalError (#9514)

* check status precise_code instead of construct OLAPInternalError
* move is_io_error to Status

6 days ago[refactor]Cleanup unused empty files (#9497)
hongbin [Thu, 12 May 2022 06:58:28 +0000 (14:58 +0800)] 
[refactor]Cleanup unused empty files (#9497)

6 days ago[fix](binlog-load) binlog load fails because txn exceeds the default value (#9471)
jiafeng.zhang [Thu, 12 May 2022 05:31:22 +0000 (13:31 +0800)] 
[fix](binlog-load) binlog load fails because txn exceeds the default value (#9471)

binlog load Because txn exceeds the default value, resume is a failure,
and a friendly prompt message is given to the user, instead of prompting success now,
it still fails after a while, and the user will feel inexplicable
Issue Number: close #9468

6 days ago[fix](broker-load) can't load parquet file with column name case sensitive with Doris...
deardeng [Thu, 12 May 2022 05:27:03 +0000 (22:27 -0700)] 
[fix](broker-load) can't load parquet file with column name case sensitive with Doris column (#9358)

6 days ago[chore](fe code style)add suppressions to fe check style (#9429)
morrySnow [Thu, 12 May 2022 04:16:55 +0000 (12:16 +0800)] 
[chore](fe code style)add suppressions to fe check style (#9429)

Current fe check style check all files. But some rules should be only applied on production files.
Add suppressions to suppress some rules on test files.

7 days ago[feature] support row policy filter (#9206)
Stalary [Wed, 11 May 2022 14:11:10 +0000 (22:11 +0800)] 
[feature] support row policy filter (#9206)

7 days ago[fixbug]fix bug for OLAP_SUCCESS with Status (#9427)
pengxiangyu [Wed, 11 May 2022 12:04:06 +0000 (20:04 +0800)] 
[fixbug]fix bug for OLAP_SUCCESS with Status (#9427)

7 days ago[bugfix](vtablet_sink) fix max_pending_bytes for vtablet_sink (#9462)
yixiutt [Wed, 11 May 2022 10:00:56 +0000 (18:00 +0800)] 
[bugfix](vtablet_sink) fix max_pending_bytes for vtablet_sink (#9462)

Co-authored-by: yixiutt <yixiu@selectdb.com>
7 days ago[doc] update fe checkstyle doc (#9373)
morrySnow [Wed, 11 May 2022 07:44:29 +0000 (15:44 +0800)] 
[doc] update fe checkstyle doc (#9373)

7 days ago[refactor](Nereids): cascades refactor (#9470)
jakevin [Wed, 11 May 2022 03:07:58 +0000 (11:07 +0800)] 
[refactor](Nereids): cascades refactor (#9470)

Describe the overview of changes.

- rename GroupExpression
- use `HashSet<GroupExpression> groupExpressions` in `memo`
- add label of `Nereids` for CI
- remove `GroupExpr` from Plan

7 days ago[fix](http) Hardening Recommendations Disable TRACE/TRAC methods (#9479)
jiafeng.zhang [Wed, 11 May 2022 01:41:59 +0000 (09:41 +0800)] 
[fix](http) Hardening Recommendations Disable TRACE/TRAC methods (#9479)

7 days ago[refactor](backend) Refactor the logic of selecting Backend in FE. (#9478)
Mingyu Chen [Wed, 11 May 2022 01:40:57 +0000 (09:40 +0800)] 
[refactor](backend) Refactor the logic of selecting Backend in FE. (#9478)

There are many places in FE where a group of BE nodes needs to be selected according to certain requirements. for example:
1. When creating replicas for a tablet.
2. When selecting a BE to execute Insert.
3. When Stream Load forwards http requests to BE nodes.

These operations all have the same logic. So this CL mainly changes:
1. Create a new `BeSelectionPolicy` class to describe the set of conditions for selecting BE.
2. The logic of selecting BE nodes in `SystemInfoService` has been refactored, and the following two methods are used uniformly:
    1. `selectBackendIdsByPolicy`: Select the required number of BE nodes according to the `BeSelectionPolicy`.
    2. `selectBackendIdsForReplicaCreation`: Select the BE node for the replica creation operation.

Note that there are some changes here:
For the replica creation operation, the round-robin method was used to select BE nodes before,
but now it is changed to `random` selection for the following reasons:
1. Although the previous logic is round-robin, it is actually random.
2. The final diff of the random algorithm will not be greater than 5%, so it can be considered that the random algorithm
     can distribute the data evenly.

7 days ago[regression] add regression test for compaction (#9437)
Gabriel [Wed, 11 May 2022 01:40:21 +0000 (09:40 +0800)] 
[regression] add regression test for compaction (#9437)

Trigger compaction via REST API in this case.

7 days ago[feature](mysql-table) support utf8mb4 for mysql external table (#9402)
xueweizhang [Wed, 11 May 2022 01:39:23 +0000 (09:39 +0800)] 
[feature](mysql-table) support utf8mb4 for mysql external table (#9402)

This patch supports utf8mb4 for mysql external table.

if someone needs a mysql external table with utf8mb4 charset, but only support charset utf8 right now.

When create mysql external table, it can add an optional propertiy "charset" which can set character fom mysql connection,
default value is "utf8". You can set "utf8mb4" instead of "utf8" when you need.

7 days ago[feature] show create materialized view (#9391)
Stalary [Wed, 11 May 2022 01:29:55 +0000 (09:29 +0800)] 
[feature] show create materialized view (#9391)

8 days ago[refactor][style] Use clang-format to sort includes (#9483)
Adonis Ling [Tue, 10 May 2022 13:25:35 +0000 (21:25 +0800)] 
[refactor][style] Use clang-format to sort includes (#9483)

8 days ago[refactor] delete OLAP_LOG_WARNING related macro definition (#9484)
BePPPower [Tue, 10 May 2022 12:53:45 +0000 (20:53 +0800)] 
[refactor] delete OLAP_LOG_WARNING related macro definition (#9484)

Co-authored-by: BePPPower <fangtiewei@selectdb.com>
8 days ago[feature-wip] (memory tracker) (step6, End) Fix some details (#9301)
Xinyi Zou [Tue, 10 May 2022 10:17:09 +0000 (18:17 +0800)] 
[feature-wip] (memory tracker) (step6, End) Fix some details  (#9301)

1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.

8 days ago[Enhancement](Optimizer) Nereids pattern matching base framework (#9474)
924060929 [Tue, 10 May 2022 02:06:04 +0000 (10:06 +0800)] 
[Enhancement](Optimizer) Nereids pattern matching base framework (#9474)

This pr provide a new pattern matching framework for Nereids optimizer.

The new pattern matching framework contains this concepts:

1. `Pattern`/`PatternDescriptor`: the tree node's multiple hierarchy shape, e.g. `logicalJoin(logicalJoin(), any()` pattern describe a plan that root is a `LogicalJoin` and the left child is `LogicalJoin` too.
2. `MatchedAction`: a callback function when the pattern matched, usually you can create new plan to replace the origin matched plan.
3. `MatchingContext`: the param pass through MatchedAction, contains the matched plan root and the PlannerContext.
4. `PatternMatcher`: contains PatternDescriptor and MatchedAction
5. `Rule`: a rewrite rule contains RuleType, PatternPromise, Pattern and transform function(equals to MatchedAction)
6. `RuleFactory`: the factory can help us build Rules easily. RuleFactory extends Patterns interface, and have some predefined pattern descriptors.

for example, Join commutative:
```java
public class JoinCommutative extends OneExplorationRuleFactory {
    @Override
    public Rule<Plan> build() {
        return innerLogicalJoin().thenApply(ctx -> {
            return new LogicalJoin(
                JoinType.INNER_JOIN,
                ctx.root.getOnClause(),
                ctx.root.right(),
                ctx.root.left()
            );
        }).toRule(RuleType.LOGICAL_JOIN_COMMUTATIVE);
    }
}
```

the code above show the three step to create a Rule
1. 'innerLogicalJoin()' declare pattern  is an inner logical join. 'innerLogicalJoin' is a predefined pattern.
2. invoke 'thenApply()' function to combine a MatchedAction, return a new LogicalJoin with exchange children.
3. invoke 'toRule()' function to convert to Rule

You can think the Rule contains three parts:
1. Pattern
2. transform function / MatchedAction
3. RuleType and RulePromise

So
1. `innerLogicalJoin()` create a `PatternDescriptor`, which contains a `Pattern`
2. `PatternDescriptor.then()` convert `PatternDescriptor` to `PatternMatcher,` witch contains Pattern and MatchedAction
3. `PatternMatcher.toRule()` convert `PatternMatcher` to a Rule

This three step inspired by the currying in function programing.

It should be noted, #9446 provide a generic type for TreeNode's children, so we can infer multiple hierarchy type in this pattern matching framework, so you can get the really tree node type without unsafely cast. like this:
```java
logicalJoin(logicalJoin(), any()).then(j -> {
     // j can be inferred type to LogicalJoin<LogicalJoin<Plan, Plan>, Plan>
     // so j.left() can be inferred type to LogicalJoin<Plan, Plan>,
     // so you don't need to cast j.left() from 'Plan' to 'LogicalJoin'
     var node = j.left().left();
})
```

8 days ago[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
hongbin [Tue, 10 May 2022 01:25:59 +0000 (09:25 +0800)] 
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)

* Replace '#ifndef' with '#pragma once'

8 days ago[refactor] modify all OLAP_LOG_WARNING to LOG(WARNING) (#9473)
BePPPower [Tue, 10 May 2022 01:25:25 +0000 (09:25 +0800)] 
[refactor] modify all OLAP_LOG_WARNING to LOG(WARNING) (#9473)

Co-authored-by: BePPPower <fangtiewei@selectdb.com>
8 days ago[docs][typo] Fix some typoes in "update.md" content. (#9455)
liuzhuang2017 [Tue, 10 May 2022 01:02:04 +0000 (09:02 +0800)] 
[docs][typo] Fix some typoes in "update.md" content. (#9455)

Fix some typoes in "update.md" content

8 days ago[Doc] 添加CTAS文档 (#9454)
Stalary [Tue, 10 May 2022 01:01:42 +0000 (09:01 +0800)] 
[Doc] 添加CTAS文档 (#9454)

* ADD: 添加CTAS文档

9 days ago[feature] (vec) instead of converting line to src tuple for stream load in vectorize...
xiepengcheng01 [Mon, 9 May 2022 03:24:07 +0000 (11:24 +0800)] 
[feature] (vec)  instead of converting line to src tuple for stream load in vectorized. (#9314)

Co-authored-by: xiepengcheng01 <xiepengcheng01@xafj-palo-rpm64.xafj.baidu.com>
9 days ago[code style](fe) Include test sources (#9366)
leo65535 [Mon, 9 May 2022 01:40:44 +0000 (09:40 +0800)] 
[code style](fe) Include test sources (#9366)

Include test sources, we also need to check them.

9 days ago[fix](ut) fix DeltaWriter::close_wait parameter mismatch in delta_writer_test (#9457)
Xinyi Zou [Mon, 9 May 2022 01:38:12 +0000 (09:38 +0800)] 
[fix](ut) fix DeltaWriter::close_wait parameter mismatch in delta_writer_test (#9457)

9 days ago[fix] UT MathFunctionTest.round_test fix (#9447)
camby [Mon, 9 May 2022 01:37:27 +0000 (09:37 +0800)] 
[fix] UT MathFunctionTest.round_test fix (#9447)

Function round support two format round(double) and round(double, int), the argument is variadic.
But FunctionBinaryArithmetic not support variadic argument now, make get_function for round(double, int) failed.

reproduce steps:
1. set enable_vectorized_engine=true;
2. try to call round(double, int);
```
> select round(10.12345,2);
ERROR 1105 (HY000): errCode = 2, detailMessage = Function round is not implemented
```

9 days ago[fix](s3) fix s3 Temp file may write failed because of has no space on disk (#9421)
Zhengguo Yang [Mon, 9 May 2022 01:28:43 +0000 (09:28 +0800)] 
[fix](s3) fix s3 Temp file may write failed because of has no space on disk (#9421)

9 days ago[doc] update data-model.md and data-partition.md (#9448)
BePPPower [Mon, 9 May 2022 01:19:09 +0000 (09:19 +0800)] 
[doc] update data-model.md and data-partition.md (#9448)

update data-model.md and data-partition.md

9 days ago[doc] Update DECIMAL.md (#9451)
emerkfu [Mon, 9 May 2022 01:17:24 +0000 (09:17 +0800)] 
[doc] Update DECIMAL.md (#9451)

* Update DECIMAL.md

9 days agoUpdate data-partition.md (#9450)
sparklezzz [Mon, 9 May 2022 01:17:00 +0000 (09:17 +0800)] 
Update data-partition.md (#9450)

Update data-partition.md

9 days ago[Doc] fix doc link suffix .html to .md (#9442)
wudi [Mon, 9 May 2022 01:16:06 +0000 (09:16 +0800)] 
[Doc] fix doc link suffix .html to .md (#9442)

* fix doc link suffix html to md

10 days ago[fix](schema_hash) Fix bug that introduced by removing schema_hash (#9449)
caiconghui [Sun, 8 May 2022 13:03:10 +0000 (21:03 +0800)] 
[fix](schema_hash) Fix bug that introduced by removing schema_hash (#9449)

10 days ago[feature] (sql-digest) support sql digest (#8919)
Henry2SS [Sun, 8 May 2022 09:25:41 +0000 (17:25 +0800)] 
[feature] (sql-digest) support sql digest (#8919)

10 days ago[Enhancement](Optimizer) Optimize nereids tree node structure (#9446)
924060929 [Sun, 8 May 2022 08:56:00 +0000 (16:56 +0800)] 
[Enhancement](Optimizer) Optimize nereids tree node structure (#9446)

This pr optimize nereids tree node structure for generic parameter and Nary abstract tree node.
It can facilitate the use of pattern match framework.

10 days ago[Bug] Missing error tablet list when close_wait return error (#9418)
pengxiangyu [Sat, 7 May 2022 22:45:28 +0000 (06:45 +0800)] 
[Bug] Missing error tablet list when close_wait return error (#9418)

10 days ago[style]add IntelliJ code style xml (#9438)
morrySnow [Sat, 7 May 2022 22:41:46 +0000 (06:41 +0800)] 
[style]add IntelliJ code style xml (#9438)

10 days agoimprove doc spark3.2.0 (#9445)
cxzl25 [Sat, 7 May 2022 22:39:58 +0000 (06:39 +0800)] 
improve doc spark3.2.0 (#9445)

11 days ago[refactor](test) Refactor FE unit test framework that starts a FE server. (#9388)
Shuo Wang [Sat, 7 May 2022 13:28:42 +0000 (21:28 +0800)] 
[refactor](test) Refactor FE unit test framework that starts a FE server. (#9388)

Currently, we use `UtFrameUtils` to start a FE server in the FE unit test.
Each test class has to do some initialization and clean up stuff with the JUnit4
`@BeforeClass` and `@AfterClass` annotation. It's redundant and boring.
Besides, almost all the APIs in `UtFrameUtils` has a `ConnectContext` parameter, which is not easy to use.

This PR proposes to use an inherit-manner, i.e., wrap all the common logic in base class `TestWithFeService`,
leveraging the
JUnit5 `@BeforeAll` and `@AfterAll` annotation to narrow down the setup and cleanup lifecycle to each test class instance.
At the same time, the derived concrete test class could directly use utility methods inherited from the base class,
without calling a util class and passing a `ConnectContext` argument.

`UtFrameUtils` and `DorisAssert`  are marked as deprecated. We could remove these two classes
if this refactor works well for a time.

11 days ago[fix][feature](Function) fix return type && support hll_union_agg/group_concat agg...
zhangstar333 [Sat, 7 May 2022 12:44:04 +0000 (20:44 +0800)] 
[fix][feature](Function) fix return type && support hll_union_agg/group_concat agg to window function (#9119)

11 days ago[fix](rewrite) The where condition cannot be pushed down because there is no derivati...
zhengshiJ [Sat, 7 May 2022 12:41:11 +0000 (20:41 +0800)] 
[fix](rewrite) The where condition cannot be pushed down because there is no derivation (#8980)

Fix a bug.
The where condition cannot be pushed down because there is no derivation

eg:
select * from tb1 left join tb2 on tb1.id = tb2.id where tb2.id = 1;

The correct case is that the condition of "=1" needs to be deduced to tb1.id,
but the current implementation does not do the deduction

11 days ago[refactor] some code cleanup for min/max function. (#8874)
zhannngchen [Sat, 7 May 2022 12:39:44 +0000 (20:39 +0800)] 
[refactor] some code cleanup for min/max function. (#8874)

11 days ago[fix]Stream load 307 temporary redirection authentication information is lost (#9363)
jiafeng.zhang [Sat, 7 May 2022 11:22:45 +0000 (19:22 +0800)] 
[fix]Stream load 307 temporary redirection authentication information is lost (#9363)

11 days ago[doc]Add create java udf document (#9430)
caoliang-web [Sat, 7 May 2022 11:20:56 +0000 (19:20 +0800)] 
[doc]Add create java udf document (#9430)

* Add create java udf document

* Add create java udf document

11 days ago[FE Code Style] Fix wrong checkstyle action configuration (#9370)
leo65535 [Sat, 7 May 2022 11:18:00 +0000 (19:18 +0800)] 
[FE Code Style] Fix wrong checkstyle action configuration (#9370)

11 days ago[Bug][CTAS] create table by partition list (#9412)
Stalary [Sat, 7 May 2022 11:17:39 +0000 (19:17 +0800)] 
[Bug][CTAS] create table by partition list (#9412)

Co-authored-by: Rongqian Li <rongqian_li@idgcapital.com>
11 days ago[doc]fix typo in en faq and format pattern (#9423)
dataalive [Sat, 7 May 2022 11:16:59 +0000 (19:16 +0800)] 
[doc]fix typo in en faq and format pattern (#9423)

11 days ago[DOC]Improve fe idea dev (#9431)
hongdd [Sat, 7 May 2022 11:15:35 +0000 (04:15 -0700)] 
[DOC]Improve fe idea dev (#9431)

11 days ago[Bug] (fix) DeltaWriter::mem_consumption() coredump (#9245)
chenlinzhong [Sat, 7 May 2022 11:13:08 +0000 (19:13 +0800)] 
[Bug] (fix) DeltaWriter::mem_consumption() coredump (#9245)

11 days ago[doc]Add hll usage documentation (#9398)
jiafeng.zhang [Sat, 7 May 2022 10:56:48 +0000 (18:56 +0800)] 
[doc]Add hll usage documentation (#9398)

Add hll usage documentation