[ASTERIXDB-2726][EXT] Report line number instead of record number in messages of...
authorAli Alsuliman <ali.al.solaiman@gmail.com>
Tue, 12 May 2020 03:36:17 +0000 (20:36 -0700)
committerAli Alsuliman <ali.al.solaiman@gmail.com>
Tue, 12 May 2020 07:18:27 +0000 (07:18 +0000)
commite844fad3435568452d6f738b45570d3bc1295039
tree507db622b7629eea7b57d4ada75fd26c775887f2
parent410b6c8492d4d4052b338fa2f5a558c3e77d829d
[ASTERIXDB-2726][EXT] Report line number instead of record number in messages of parsers

- user model changes: no
- storage format changes: no
- interface changes: yes
   IRecordReader:
      added getLineNumber() to provide line number for parsers and converters.
   IRecordConverter:
      added configure() to pass the line number supplier to the record converter.
   IRecordDataParser:
      pass line number supplier from the Reader to the Parser.

Details:
Report line number instead of record number in messages of parsers.

- added getPreviousStreamName() to allow readers to report errors happening on
   the previous stream when the underlying stream has already switched to a new one.
- changed the test executor to compare actual warnings issues by a test case with
   the expected warnigns properly.

Change-Id: I00508d8eeca4d9bae95f55ab51ecfb0ce2ced6b0
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6245
Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Hussain Towaileb <hussainht@gmail.com>
41 files changed:
asterixdb/asterix-app/data/csv/error1_line_num.csv [new file with mode: 0644]
asterixdb/asterix-app/data/csv/error2_line_num.csv [new file with mode: 0644]
asterixdb/asterix-app/data/json/malformed-json-no-closing.json [new file with mode: 0644]
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/CancellationTestExecutor.java
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/IPollTask.java
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/csv-warnings/query-dataset.002.ddl.sqlpp
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/csv-warnings/query-dataset.014.s3bucket.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/csv-warnings/query-dataset.015.ddl.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/csv-warnings/query-dataset.016.query.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/csv-warnings/query-dataset.017.s3bucket.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/csv-warnings/query-dataset.018.query.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/json-warnings/json-warnings.001.s3bucket.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/json-warnings/json-warnings.002.ddl.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/json-warnings/json-warnings.003.query.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/aws/s3/json-warnings/json-warnings.099.ddl.sqlpp [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/objects/ObjectsQueries.xml
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/aws/s3/csv-warnings/external_dataset.006.adm [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/aws/s3/csv-warnings/external_dataset.007.adm [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/aws/s3/json-warnings/json-warnings.003.adm [new file with mode: 0644]
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_one_partition.xml
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/AsterixInputStream.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordConverter.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordDataParser.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReader.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/converter/CSVToRecordWithMetadataAndPKConverter.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/AwsS3InputStream.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReader.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/AsterixInputStreamReader.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/LocalFSInputStream.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/DelimitedDataParser.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/RecordWithMetadataParser.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DataflowControllerProvider.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ParseUtil.java
hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties
hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/file/FieldCursorForDelimitedDataParser.java