incubator-doris-flink-connector.git
6 days agofix broke url and ut (#32) master
cxzl25 [Mon, 9 May 2022 06:31:59 +0000 (14:31 +0800)] 
fix broke url and ut (#32)

3 weeks agofix flink schema and doris schema column mapping (#30)
wudi [Mon, 25 Apr 2022 05:49:30 +0000 (13:49 +0800)] 
fix flink schema and doris schema column mapping (#30)

fix flink schema and doris schema column mapping

3 weeks ago[Feature] Refactoring DorisSouce based on FLIP-27 (#24)
wudi [Thu, 21 Apr 2022 02:40:47 +0000 (10:40 +0800)] 
[Feature] Refactoring DorisSouce based on FLIP-27 (#24)

* Refactoring DorisSouce based on FLIP-27

4 weeks ago[Bug] Fix row type decimal convert bug (#26)
aiwenmo [Fri, 15 Apr 2022 03:01:08 +0000 (11:01 +0800)] 
[Bug] Fix row type decimal convert bug (#26)

* Fix row type decimal convert bug

5 weeks ago[improvement] json load by line (#23)
madong [Fri, 8 Apr 2022 01:29:40 +0000 (09:29 +0800)] 
[improvement] json load by line (#23)

7 weeks agoUpdate README.md (#19)
BAKEZQ [Fri, 25 Mar 2022 04:12:46 +0000 (12:12 +0800)] 
Update README.md (#19)

* Update README.md

2 months ago[refactor]doris sink enable 2pc (#12)
madong [Tue, 8 Mar 2022 01:45:45 +0000 (09:45 +0800)] 
[refactor]doris sink enable 2pc (#12)

* [refactor]doris sink enable 2pc

2 months ago[chore] Change project name (#16)
Mingyu Chen [Sun, 6 Mar 2022 02:53:40 +0000 (10:53 +0800)] 
[chore] Change project name (#16)

Change project name (#16)

2 months ago[chore] change the GAV (#14)
Mingyu Chen [Sun, 6 Mar 2022 02:33:18 +0000 (10:33 +0800)] 
[chore] change the GAV (#14)

1. modify GAV to "flink-doris-connector-{flink.minor.version}_${scala_version}"

2 months ago[improvement] Supports traversal of Doris FE nodes when searching for Doris BE (#11)
Jiangqiao Xu [Sun, 27 Feb 2022 03:08:53 +0000 (11:08 +0800)] 
[improvement] Supports traversal of Doris FE nodes when searching for Doris BE (#11)

2 months ago[chore] modify build.sh and github action for 1.14 (#10)
Mingyu Chen [Fri, 25 Feb 2022 05:24:46 +0000 (13:24 +0800)] 
[chore] modify build.sh and github action for 1.14 (#10)

2 months agoFix notification settings not taking effect (#8)
Kirs [Fri, 25 Feb 2022 05:24:31 +0000 (13:24 +0800)] 
Fix notification settings not taking effect (#8)

2 months ago[fix] When doing checkpoint, write cache data to doris to prevent loss (#4)
liuyaolin [Tue, 22 Feb 2022 11:19:32 +0000 (19:19 +0800)] 
[fix] When doing checkpoint, write cache data to doris to prevent loss (#4)

2 months ago[feature] Flink 1.14 support (#6)
wudi [Mon, 21 Feb 2022 12:31:29 +0000 (20:31 +0800)] 
[feature] Flink 1.14 support (#6)

2 months ago[community] modify NOTICE and CONTRIBUTING doc (#7)
Mingyu Chen [Mon, 21 Feb 2022 12:31:03 +0000 (20:31 +0800)] 
[community] modify NOTICE and CONTRIBUTING doc (#7)

2 months ago[chore] Modify the build.sh script (#5)
Mingyu Chen [Fri, 18 Feb 2022 03:48:12 +0000 (11:48 +0800)] 
[chore] Modify the build.sh script (#5)

Add "--tag" option so that when we can build from a tag.
Because in tag, the flink and scala version is deterministic and does not need to be specified.

2 months ago[chore] add maven-gpg-plugin (#3)
Mingyu Chen [Mon, 14 Feb 2022 08:14:12 +0000 (16:14 +0800)] 
[chore] add maven-gpg-plugin (#3)

2 months ago[chore] change the repo name (#2)
Mingyu Chen [Mon, 14 Feb 2022 04:46:58 +0000 (12:46 +0800)] 
[chore] change the repo name (#2)

2 months ago[chore] add maven-release-plugin plugin (#1)
Mingyu Chen [Mon, 14 Feb 2022 04:42:13 +0000 (12:42 +0800)] 
[chore] add maven-release-plugin plugin (#1)

Try to solve the error when executing "mvn release:prepare"

3 months ago[init] add .asf.yaml
morningman [Fri, 11 Feb 2022 15:37:33 +0000 (23:37 +0800)] 
[init] add .asf.yaml

3 months ago[init] do some init work
morningman [Fri, 11 Feb 2022 15:07:49 +0000 (23:07 +0800)] 
[init] do some init work

3 months ago[init] init commit
morningman [Fri, 11 Feb 2022 14:59:52 +0000 (22:59 +0800)] 
[init] init commit

Move flink-doris-connector from incubator-doris@df2c756

3 months ago[fix](httpv2) make http v2 and v1 interface compatible (#7848)
jiafeng.zhang [Mon, 31 Jan 2022 14:12:34 +0000 (22:12 +0800)] 
[fix](httpv2) make http v2 and v1 interface compatible (#7848)

http v2 TableSchemaAction adds the return value of aggregation_type,
and modifies the corresponding code of Flink/Spark Connector

3 months ago[chore][fix][doc](fe-plugin)(mysqldump) fix build auditlog plugin error (#7804)
Zhengguo Yang [Wed, 26 Jan 2022 01:11:23 +0000 (09:11 +0800)] 
[chore][fix][doc](fe-plugin)(mysqldump) fix build auditlog plugin error (#7804)

1. fix problems when build fe_plugins
2. format
3. add docs about dump data using mysql dump

3 months ago[Feature][flink-connector] support flink delete option (#7457)
wudi [Sun, 23 Jan 2022 12:24:41 +0000 (20:24 +0800)] 
[Feature][flink-connector] support flink  delete option (#7457)

* Flink Connector supports delete option on Unique models
Co-authored-by: wudi <wud3@shuhaisc.com>
3 months agoFlink / Spark connector compilation problem (#7725)
jiafeng.zhang [Fri, 14 Jan 2022 14:14:48 +0000 (22:14 +0800)] 
Flink / Spark connector compilation problem (#7725)

Flink / Spark connector compilation problem

4 months ago[chore][docs] add deploy spark/flink connectors to maven release repo docs (#7616)
Zhengguo Yang [Thu, 6 Jan 2022 15:23:33 +0000 (23:23 +0800)] 
[chore][docs] add deploy spark/flink connectors to maven release repo docs (#7616)

4 months ago[refactor] update parent pom version and optimize build scripts (#7548)
Zhengguo Yang [Wed, 5 Jan 2022 02:45:11 +0000 (10:45 +0800)] 
[refactor] update parent pom  version and optimize build scripts (#7548)

4 months ago[refactor] Standardize the writing of pom files, prepare for deployment to maven...
Zhengguo Yang [Thu, 30 Dec 2021 02:16:37 +0000 (10:16 +0800)] 
[refactor] Standardize the writing of pom files, prepare for deployment to maven (#7477)

4 months ago[improvement](spark-connector)(flink-connector) Modify the max num of batch written...
jiafeng.zhang [Sun, 26 Dec 2021 03:13:47 +0000 (11:13 +0800)] 
[improvement](spark-connector)(flink-connector) Modify the max num of batch written by Spark/Flink connector each time. (#7485)

Increase the default batch size and flush interval

4 months ago[improvement](flink-connector) flush data without multi httpclients (#7329) (#7450)
Heng Zhao [Fri, 24 Dec 2021 13:28:35 +0000 (21:28 +0800)] 
[improvement](flink-connector) flush data without multi httpclients (#7329) (#7450)

reuse http client to flush data

4 months ago[fix](flink-connector) Connector should visit the surviving BE nodes (#7435)
Heng Zhao [Tue, 21 Dec 2021 03:05:42 +0000 (11:05 +0800)] 
[fix](flink-connector) Connector should visit the surviving BE nodes (#7435)

4 months ago[improvement](flink-connector) DataSourceFunction read doris supports parallel (...
wudi [Wed, 15 Dec 2021 08:21:29 +0000 (16:21 +0800)] 
[improvement](flink-connector) DataSourceFunction read doris supports parallel (#7232)

The previous DataSourceFunction inherited from RichSourceFunction.
As a result, no matter how much the parallelism of flink is set, the parallelism of DataSourceFunction is only 1.
Now modify it to RichParallelSourceFunction.

And when flink has multiple degrees of parallelism, assign the doris data to each parallelism.
For example, read dorisPartitions.size = 10, flink.parallelism = 4
The task is split as follows:
task0: dorisPartitions[0],[4],[8]
task1: dorisPartitions[1],[5],[9]
task2: dorisPartitions[2],[6]
task3: dorisPartitions[3],[7]

4 months ago[chore][community](github) Remove travis and add github action (#7380)
Mingyu Chen [Wed, 15 Dec 2021 05:27:37 +0000 (13:27 +0800)] 
[chore][community](github) Remove travis and add github action (#7380)

1. Remove travis
2. Add github action to build extension:
    1. docs
    2. fs_broker
    3. flink/spark/connector

5 months ago[License] Add License header for missing files (#7130)
Mingyu Chen [Tue, 16 Nov 2021 10:37:54 +0000 (18:37 +0800)] 
[License] Add License header for missing files (#7130)

1. Add License header for missing files
2. Modify the spark pom.xml to correct the location of `thrift`

6 months ago[Feature] Support Flink and Spark connector support String type (#7075)
wudi [Sat, 13 Nov 2021 09:10:22 +0000 (17:10 +0800)] 
[Feature] Support Flink and Spark connector support String type (#7075)

Support String type for Flink and Spark connector

6 months ago[Build]Compile and output the jar file, add Spark, Flink version and Scala version...
jiafeng.zhang [Tue, 9 Nov 2021 02:02:08 +0000 (10:02 +0800)] 
[Build]Compile and output the jar file, add Spark, Flink version and Scala version (#7051)

The jar file compiled by Flink and Spark Connector, with the corresponding Flink, Spark version
and Scala version at compile time, so that users can know whether the version number matches when using it.

Example of output file name:doris-spark-1.0.0-spark-3.2.0_2.12.jar

6 months ago[HTTP][API] Add backends info API for spark/flink connector (#6984)
Mingyu Chen [Fri, 5 Nov 2021 01:43:06 +0000 (09:43 +0800)] 
[HTTP][API] Add backends info API for spark/flink connector (#6984)

Doris should provide a http api to return backends list for connectors to submit stream load,
and without privilege checking, which can let common user to use it

6 months agosupport use char like \x01 in flink-doris-sink column & line delimiter (#6937)
wunan1210 [Fri, 29 Oct 2021 05:56:52 +0000 (13:56 +0800)] 
support use char like \x01 in flink-doris-sink column & line delimiter (#6937)

* support use char like \x01 in flink-doris-sink column & line delimiter

* extend imports

* add docs

6 months ago[Flink]Simplify the use of flink connector (#6892)
xiaokangguo [Sat, 23 Oct 2021 10:10:47 +0000 (18:10 +0800)] 
[Flink]Simplify the use of flink connector  (#6892)

1. Simplify the use of flink connector like other stream sink by GenericDorisSinkFunction.
2. Add the use cases of flink connector.

## Use case
```
env.fromElements("{\"longitude\": \"116.405419\", \"city\": \"北京\", \"latitude\": \"39.916927\"}")
     .addSink(
          DorisSink.sink(
             DorisOptions.builder()
                   .setFenodes("FE_IP:8030")
                   .setTableIdentifier("db.table")
                   .setUsername("root")
                   .setPassword("").build()
                ));
```

6 months ago[Flink][Bug] Fix potential NPE when cancel DorisSourceFunction (#6838)
Yun Tang [Sat, 23 Oct 2021 08:45:24 +0000 (16:45 +0800)] 
[Flink][Bug] Fix potential NPE when cancel DorisSourceFunction (#6838)

Fix potential NPE of `scalaValueReader` when cancelling DorisSourceFunction.

7 months ago[Dependency] Upgrade thirdparty libs (#6766)
Zhengguo Yang [Fri, 15 Oct 2021 05:03:04 +0000 (13:03 +0800)] 
[Dependency] Upgrade thirdparty libs (#6766)

Upgrade the following dependecies:

libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0

after this pr compile doris should use build-env:1.4.0

7 months ago[Fix] Flink connector support json import and use httpclient to streamlaod (#6740)
wudi [Tue, 28 Sep 2021 09:37:03 +0000 (04:37 -0500)] 
[Fix] Flink connector support json import and use httpclient to streamlaod (#6740)

* [Bug]:fix when data null , throw NullPointerException

* [Bug]:Distinguish between null and empty string

* [Feature]:flink-connector supports streamload parameters

* [Fix]:code style

* [Fix]: support json format import and use httpclient to streamload

* [Fix]:remove System out

* [Fix]:upgrade httpclient  version

* [Doc]: add json format import doc

Co-authored-by: wudi <wud3@shuhaisc.com>
7 months ago[Flink] Fix bug of flink doris connector (#6655)
xhmz [Fri, 24 Sep 2021 13:38:35 +0000 (21:38 +0800)] 
[Flink] Fix bug of flink doris connector (#6655)

Flink-Doris-Connector do not support flink 1.13, refactor doris sink forma
to not use GenericRowData. But to use RowData::FieldGetter.

8 months ago[FlinkConnector] Make flink datastream source parameterized (#6473)
wunan1210 [Sun, 22 Aug 2021 14:03:32 +0000 (22:03 +0800)] 
[FlinkConnector] Make flink datastream source parameterized (#6473)

make flink datastream source parameterized as List<?> instead of Object.

8 months ago[Doc] flink/spark connector: add sources/javadoc plugins (#6435)
wunan1210 [Mon, 16 Aug 2021 14:41:24 +0000 (22:41 +0800)] 
[Doc] flink/spark connector: add sources/javadoc plugins (#6435)

spark-doris-connector/flink-doris-connect add plugins to generate javadoc and sources jar,
so can be easy to distribute and debug.

9 months ago[Feature]:Flink-connector supports streamload parameters (#6243)
wudi [Mon, 9 Aug 2021 14:12:46 +0000 (22:12 +0800)] 
[Feature]:Flink-connector supports streamload parameters (#6243)

Flink-connector supports streamload parameters
#6199

10 months ago[Bug][Flink] Fix when data null , flink-connector throw NullPointerException (#6165)
wudi [Thu, 8 Jul 2021 01:55:50 +0000 (09:55 +0800)] 
[Bug][Flink] Fix when data null , flink-connector throw NullPointerException (#6165)

10 months ago[FlinkConnector] Support time interval for flink connector (#5934)
wudi [Wed, 30 Jun 2021 01:27:12 +0000 (09:27 +0800)] 
[FlinkConnector] Support time interval for flink connector (#5934)

11 months ago[Log] Fix a mistake in DorisDynamicOutputFormat.java (#5963)
zhangboya1 [Sun, 6 Jun 2021 14:06:57 +0000 (22:06 +0800)] 
[Log] Fix a mistake in DorisDynamicOutputFormat.java (#5963)

Fix a mistake DorisDynamicOutputFormat.java

11 months ago[Bug] Modify spark, flink doris connector to send request to FE, fix the problem...
jiafeng.zhang [Wed, 19 May 2021 01:28:21 +0000 (09:28 +0800)] 
[Bug] Modify spark, flink doris connector to send request to FE, fix the problem of POST method, it should be the same as the method when sending the request (#5788)

Modify spark, flink doris connector to send request to FE, fix the problem of POST method,
it should be the same as the method when sending the request

12 months ago[Feature] Flink Doris Connector (#5372) (#5375)
wudi [Fri, 23 Apr 2021 01:43:48 +0000 (09:43 +0800)] 
[Feature] Flink Doris Connector (#5372) (#5375)