Pindikura Ravindra [Sat, 15 Dec 2018 00:18:01 +0000 (05:48 +0530)] 
ARROW-1807: [Java] consolidate bufs to reduce heap (#3121)
- for fixed-len vectors, alloc a combined arrow buf for
value and validity.
- Remove the read-write locks in AllocationMgr, they
contribute about 150 bytes to the heap, and aren't very useful
since there isn't much contention.
Wes McKinney [Fri, 14 Dec 2018 20:49:12 +0000 (14:49 -0600)] 
ARROW-3762: [C++/Python] Support reading Parquet BYTE_ARRAY columns containing over 2GB of data
This patch ended up being a bit more of a bloodbath than I planned: please accept my apologies.
Associated changes in this patch:
* Split up builder.h/builder.cc into a new arrow/array directory. Public arrow/builder.h API preserved. I think this code is going to keep growing more specialized components, so I think we should get out ahead of it by having a subdirectory to contain files related to implementation details
* Implement ChunkedBinaryBuilder, ChunkedStringBuilder classes, add tests and benchmarks
* Deprecate parquet::arrow methods returning Array
* Allow implicit construction of Datum from its variant types (makes for a lot nicer syntax)
As far as what code to review, focus efforts on
* src/parquet/arrow
* src/arrow/array/builder_binary.h/cc, array-binary-test.cc, builder-benchmark
* src/arrow/compute changes
* Python changes
I'm going to tackle ARROW-2970 which should not be complicated after this patch; I will submit that as a PR after this is reviews and merged.
Author: Wes McKinney <wesm+git@apache.org>
Closes #3171 from wesm/ARROW-3762 and squashes the following commits:
822451280 <Wes McKinney> Fix int conversion warning on Windows
695ffc9df <Wes McKinney> Remove unimplemented and unused ChunkedBinaryBuilder ctor
5a525115c <Wes McKinney> Use strnlen to compute string length. Inline BinaryBuilder::AppendNextOffset
b90eb4b71 <Wes McKinney> Restore sstream include to pretty_print.cc
3669201be <Wes McKinney> Fix deprecated API use
5fdbbb261 <Wes McKinney> Rename columnar/ directory to array/
8ffaec1ef <Wes McKinney> Address preliminary code comments. Check in missing files
81e787c69 <Wes McKinney> Fix up Python bindings, unit test
2efae064c <Wes McKinney> Finish scaffolding. Get fully compiling again and original parquet-arrow test suite passing
3d075e4aa <Wes McKinney> Additional refactoring to make things chunked. Allow implicit construction of arrow::compute::Datum
922811278 <Wes McKinney> More refactoring
716322377 <Wes McKinney> Split up builder.h, builder.cc into smaller headers, compilation units. add failing test case for ARROW-3762. Add ChunkedBinaryBuilder, make BinaryBuilder Append methods inline
Wes McKinney [Fri, 14 Dec 2018 19:05:50 +0000 (13:05 -0600)]
ARROW-3184: [C++] Enable modular builds and installs with ARROW_OPTIONAL_INSTALL option. Remove ARROW_GANDIVA_BUILD_TESTS
Apparently CMake really does not want you to do `make $TARGET install` out of the box; I searched various threads about this and there's no great solutions. For expert users, this provides that option of installing only targets that have been built, while others will be ignored.
Because the possibility of users shooting themselves in the foot is so high with this, it doesn't make sense to enable by default. In the hands of an expert though, this can significantly reduce build times and make it possible to build libraries and unit tests for only a part of the project, then only install those libraries.
This will install all header files regardless of what libraries are built; I didn't see any easy way to work that out since you have to have knowledge of what headers are used by what library.
Resolves ARROW-3994
Author: Wes McKinney <wesm+git@apache.org>
Closes #3172 from wesm/ARROW-3184 and squashes the following commits:
583a916e0 <Wes McKinney> plasma_store_server requires static libraries
3c2a21ea1 <Wes McKinney> Add plasma_store_server to 'plasma' target
85fda6419 <Wes McKinney> Build plasma again in Python build for now
1b3ac57dc <Wes McKinney> Fix multiline comment in CMake
d3ce84c4e <Wes McKinney> More option reorg
b6630605c <Wes McKinney> Reorganize CMake options a bit more logically. Add more explicit warning about ARROW_OPTIONAL_INSTALL
262058b2f <Wes McKinney> Do not build Gandiva JNI bindings by default
918fdb371 <Wes McKinney> Fix ARROW_TEST_INCLUDE_LABELS option to actually work
578bc58f5 <Wes McKinney> Use GLOB instead of GLOB_RECURSE
daaafa214 <Wes McKinney> Misc fixes
a84643d6e <Wes McKinney> Fix header install option
f899bdd99 <Wes McKinney> Work around ARROW-4026 via environment variable for now
001a3ad57 <Wes McKinney> Pass in ARROW_TEST_INCLUDE_LABELS via environment variable
a1df9ab3d <Wes McKinney> Clarify documentation
2eca8a740 <Wes McKinney> Enable modular builds and install with ARROW_OPTIONAL_INSTALL option. Remove ARROW_GANDIVA_BUILD_TESTS. Add helper function for installing header files. Build fewer targets using these options in some Travis CI jobs
Philipp Moritz [Fri, 14 Dec 2018 15:27:08 +0000 (07:27 -0800)] 
ARROW-4015: [Plasma] remove unused interfaces for plasma manager
https://github.com/apache/arrow/issues/3154
This removes unused plasma interfaces Fetch(), Wait(), Transfer() and Info(), which depend on plasma manager which has already been removed from ray.
Author: Philipp Moritz <pcmoritz@gmail.com>
Author: Zhijun Fu <pingfu.fzj@antfin.com>
Author: Robert Nishihara <robertnishihara@gmail.com>
Closes #3167 from zhijunfu/remove-legacy-interfaces and squashes the following commits:
0efb5005f <Philipp Moritz> fix tensorflow op
be92e9085 <Philipp Moritz> fix java client
9da2cd38b <Philipp Moritz> Update _plasma.pyx
16ec63e9a <Robert Nishihara> More updates
e7413f739 <Robert Nishihara> Update _plasma.pyx
21398b5e7 <Zhijun Fu> merge
bcb320400 <Zhijun Fu> address comments
7967aea09 <Philipp Moritz> Merge branch 'master' into remove-legacy-interfaces
583cd97c4 <Zhijun Fu> ARROW-4015: remove unused interfaces for plasma manager
Philipp Moritz [Thu, 13 Dec 2018 23:09:27 +0000 (15:09 -0800)]
ARROW-3958: [Plasma] Reduce number of IPCs
This PR also removes the client unmap, which is not necessary any more since the introduction of malloc (since there is only few memory mapped files and they typically stay around for the lifetime of the application).
The PR also gets rid of a bunch of code that is not needed any more now (the release buffer, yay!).
Benchmarks:
```
import pyarrow.plasma as plasma
client = plasma.connect("/tmp/plasma", "", 0)
# Put performance
def f():
for i in range(10000):
client.put(1)
%timeit f()
# without optimization:
# 1.51 s ± 2.22 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 1.52 s ± 9.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 1.53 s ± 19 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# with optimizations:
# 1.27 s ± 10.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 1.31 s ± 8.18 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 1.31 s ± 17.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# Create/seal performance
def f():
for i in range(10000):
object_id = plasma.ObjectID.from_random()
client.create(object_id, 0)
client.seal(object_id)
%timeit f()
# without optimizations:
# 571 ms ± 2.28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 583 ms ± 22.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 588 ms ± 14.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# with optimizations:
# 531 ms ± 3.24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 541 ms ± 9.99 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 542 ms ± 19.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# Get performance
objects = [client.put(1) for i in range(10000)]
def g():
for i in range(10000):
client.get(objects[i])
%timeit g()
# without optimizations
# 1.11 s ± 6.17 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 1.12 s ± 1.49 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 1.19 s ± 24.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# with optimizations
# 776 ms ± 11.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 792 ms ± 3.06 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 778 ms ± 9.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Author: Philipp Moritz <pcmoritz@gmail.com>
Author: Robert Nishihara <robertnishihara@gmail.com>
Closes #3124 from pcmoritz/plasma-send-fd and squashes the following commits:
f899f459 <Philipp Moritz> Update client.cc
a0384040 <Robert Nishihara> Update _plasma.pyx
af150c14 <Philipp Moritz> comments and fixes
71c4c5c1 <Philipp Moritz> don't close fd twice
0d572823 <Philipp Moritz> linting
f60dcbed <Philipp Moritz> fix tests
502aeda4 <Philipp Moritz> linting
2887b170 <Philipp Moritz> clean up some code
cfff7e32 <Philipp Moritz> lint
e5ccbbac <Philipp Moritz> fixes
5f091993 <Philipp Moritz> introduce method
24beb277 <Philipp Moritz> working version
Antoine Pitrou [Thu, 13 Dec 2018 15:52:17 +0000 (09:52 -0600)] 
ARROW-4019: [C++] Fix Coverity issues
This fixes a number of issues found by Coverity.
Other issues are benign, or need to be tackled separately.
Author: Antoine Pitrou <antoine@python.org>
Closes #3168 from pitrou/ARROW-4019-fix-coverity-issues and squashes the following commits:
6311aa99a <Antoine Pitrou> ARROW-4019: Fix Coverity issues
Hatem Helal [Thu, 13 Dec 2018 14:56:16 +0000 (08:56 -0600)] 
PARQUET-1473: [C++] Add helper function that converts ParquetVersion to human-friendly string
Author: Hatem Helal <Hatem.Helal@mathworks.co.uk>
Author: Hatem Helal <hatem.helal@gmail.com>
Closes #3148 from hatemhelal/parquet-1473-wip and squashes the following commits:
8983ae629 <Hatem Helal> remove default case from switch to make any omission a compile-time error
a87c3ba99 <Hatem Helal> Add helper function parquet::ParquetVersionToString and use it in parquet-reader tool
Tanya Schlusser [Thu, 13 Dec 2018 12:36:21 +0000 (13:36 +0100)] 
ARROW-3866: [Python] Column metadata is not transferred to tables in pyarrow
Use columns' existing metadata to create the new fields in `Table.from_arrays()`.
Also persists the original `nullable` value.
Happy to change things! Thank you for putting a newbie label on it.
Author: Tanya Schlusser <tanya@tickel.net>
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes #3160 from tanyaschlusser/ARROW-3866 and squashes the following commits:
005940ea <Tanya Schlusser> Move the test for preserved metadata to a separate function. Add a test that nullable=False is preserved.
e4256a17 <Krisztián Szűcs> use column.field()
76216eae <Tanya Schlusser> Arrow-3866: keep field matadata for columns passed to pa.Table.from_arrays()
33950a83 <Tanya Schlusser> ARROW-3866: test to confirm column metadata is added when calling pa.Table.from_arrays(column_list)
Yosuke Shiro [Thu, 13 Dec 2018 08:44:17 +0000 (17:44 +0900)] 
ARROW-4005: [Plasma] [GLib] Add gplasma_client_disconnect()
Author: Yosuke Shiro <yosuke.shiro615@gmail.com>
Closes #3163 from shiro615/glib-add-disconnect-for-plasma-glib and squashes the following commits:
3d990034 <Yosuke Shiro> Remove require_gi by using options
a8575acd <Yosuke Shiro> Use bool instead of gboolean
b2c9ccf7 <Yosuke Shiro> Keep disconnected information
789c1dd4 <Yosuke Shiro> Fix test case
5182beb1 <Yosuke Shiro> Add Add gplasma_client_disconnect()
Wes McKinney [Wed, 12 Dec 2018 22:11:11 +0000 (16:11 -0600)]
ARROW-4008: [C++] Restore ARROW_BUILD_UTILITIES to fix integration tests
In recent refactoring, ARROW_BUILD_UTILITIES got accidentally set to OFF
Author: Wes McKinney <wesm+git@apache.org>
Closes #3166 from wesm/ARROW-4008 and squashes the following commits:
105651722 <Wes McKinney> Only add json-integration-test dependency when it is built
96bec050d <Wes McKinney> Actually build utilities
02fd08ff6 <Wes McKinney> Add integration target as dependency of arrow target
8c9fcf809 <Wes McKinney> Do not write integration test files to /tmp
af6a23b98 <Wes McKinney> Add option to write integration test files to somewhere outside of /tmp. Add integration target to C++ build
François Saint-Jacques [Wed, 12 Dec 2018 18:54:45 +0000 (19:54 +0100)] 
ARROW-3470: [C++] Fix row-wise example
- Implement the `ADD_EXAMPLE` cmake function with new ctest label
`example`, also covered by the `runexample` target. This can be
toggled via the `ARROW_BUILD_EXAMPLES` option which is ON by default.
- Implement fully working `row-wise-conversion-example.cc` and add it to
the default build.
- Update documentation to embed (manually) the newly created example.
Author: François Saint-Jacques <fsaintjacques@gmail.com>
Closes #3078 from fsaintjacques/ARROW-3470-out-of-date-example and squashes the following commits:
fab63f6f <François Saint-Jacques> ARROW-3470: Fix status macro
1eba067d <François Saint-Jacques> ARROW-3470: Fix row-wise example
Antoine Pitrou [Wed, 12 Dec 2018 16:58:11 +0000 (10:58 -0600)]
ARROW-3986: [C++] Document memory management and table APIs
Author: Antoine Pitrou <antoine@python.org>
Closes #3159 from pitrou/ARROW-3986-more-prose-documentation and squashes the following commits:
4e8ff421d <Antoine Pitrou> ARROW-3986: Document memory management and table APIs
Wes McKinney [Wed, 12 Dec 2018 15:19:41 +0000 (09:19 -0600)]
ARROW-3988: [C++] Do not build unit tests by default, fix building Gandiva unit tests when ARROW_BUILD_TESTS=OFF
I found while working on this that disabling `ARROW_GANDIVA_BUILD_TESTS` would break the build -- I think this was caused by some other changes I made. We should remove that option and instead use the new modular build targets and invoke unit tests using labels. So we would write
```
ninja gandiva # this will build all libraries and unit tests when ARROW_BUILD_TESTS=ON
ctest -L gandiva
```
Author: Wes McKinney <wesm+git@apache.org>
Closes #3156 from wesm/ARROW-3988 and squashes the following commits:
0420f9ed0 <Wes McKinney> Remove arrow::PrimitiveBuilder from builder.rst for now because of Sphinx warning
f8a33a5aa <Wes McKinney> Fix gandiva test flag
c4893534c <Wes McKinney> Add ARROW_BUILD_TESTS to appveyor-cpp-test-cmake-script.bat
5c6a33271 <Wes McKinney> Do not build unit tests by default, fix building Gandiva unit tests when ARROW_BUILD_TESTS=OFF
Kouhei Sutou [Wed, 12 Dec 2018 14:16:30 +0000 (08:16 -0600)] 
ARROW-4002: [C++][Gandiva] Remove needless CMake version check
I could build Gandiva with CMake 3.7.2 and LLVM 6.0.0 on Debian
stretch. But I disabled Gandiva JNI.
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3161 from kou/cpp-gandiva-remove-cmake-version-check and squashes the following commits:
1506c546c <Kouhei Sutou> Remove needless CMake version check
Kouhei Sutou [Wed, 12 Dec 2018 14:13:36 +0000 (08:13 -0600)]
ARROW-3976: [Ruby] Try to upgrade git to avoid errors caused by Homebrew on older git
It seems that `brew update` can fail if the git version is too old, see https://github.com/Linuxbrew/brew/issues/820. Also adds retry logic
Author: Kouhei Sutou <kou@clear-code.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes #3155 from wesm/ARROW-3976 and squashes the following commits:
0c7964ba3 <Kouhei Sutou> Stop to use old Ruby
7dce4f0a3 <Kouhei Sutou> travis_wait isn't available in custom shell script
6d41e7196 <Kouhei Sutou> Make brew commands more robust
05044892b <Wes McKinney> Incorporate code review
8c0454fd7 <Wes McKinney> Try to upgrade git to avoid errors caused by Homebrew on older git, where --local argument is missing
Kouhei Sutou [Wed, 12 Dec 2018 10:57:12 +0000 (19:57 +0900)]
ARROW-4004: [GLib] Replace GPU with CUDA
This is a follow-up change for #3088.
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3162 from kou/glib-replace-gpu-with-cuda and squashes the following commits:
8891e510 <Kouhei Sutou> Replace GPU with CUDA
Yosuke Shiro [Wed, 12 Dec 2018 08:50:13 +0000 (17:50 +0900)]
ARROW-3913: [Gandiva] [GLib] Add GGandivaLiteralNode
Support GGandivaLiteralNode including the following sub classes.
- GGandivaUint8LiteralNode
- GGandivaUint16LiteralNode
- GGandivaUint32LiteralNode
- GGandivaUint64LiteralNode
- GGandivaInt8LiteralNode
- GGandivaInt16LiteralNode
- GGandivaInt32LiteralNode
- GGandivaInt64LiteralNode
- GGandivaFloatLiteralNode
- GGandivaDoubleLiteralNode
- GGandivaStringLiteralNode
- GGandivaBinaryLiteralNode
Author: Yosuke Shiro <yosuke.shiro615@gmail.com>
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3092 from shiro615/glib-add-ggandiva-literal-node and squashes the following commits:
fb49b256 <Kouhei Sutou> Add a missing ref
91cebe1c <Kouhei Sutou> Break a long line
3ad1e5d8 <Yosuke Shiro> Support BooleanLiteralNode#value
28d301eb <Yosuke Shiro> Fix class orders
7d70c89e <Yosuke Shiro> Remove binary literal property
783a2868 <Yosuke Shiro> Use g_bytes_ref in ggandiva_binary_literal_node_get_value()
4162234d <Yosuke Shiro> Fix class orders
289dfce2 <Yosuke Shiro> Add ggandiva_binary_literal_node_new_bytes()
77f9eb89 <Yosuke Shiro> Remove (transfer full) to use return value.c_str()
e43d525f <Yosuke Shiro> Use static_pointer_cast
62a6dd5c <Yosuke Shiro> Return GBytes in ggandiva_binary_literal_node_get_value()
48d1175d <Yosuke Shiro> Remove unnecessary static_cast
d7ac46b4 <Yosuke Shiro> Remove (nullable) of size of binary literal
8f6643af <Yosuke Shiro> Fix documents
3ded5866 <Yosuke Shiro> Refactor ggandiva_literal_{}_node_new_raw()
a54c6f58 <Yosuke Shiro> Add the original raw value getter
bb2f71be <Yosuke Shiro> Rename Uint to UInt
34422ad1 <Yosuke Shiro> Remove property
7a3fe325 <Yosuke Shiro> Use 'const guint8 *value, gsize size' for binary data
138abbf8 <Yosuke Shiro> Fix a typo
ba501d60 <Yosuke Shiro> Rename is_true to value
a45fa752 <Yosuke Shiro> Use MakeStringLiteral, MakeBinaryLiteral
d8775e4a <Yosuke Shiro> Fix property name in BooleanLiteralNode
62a4eb48 <Yosuke Shiro> Add test case for LiteralNode
83876ccd <Yosuke Shiro> Support GGandivaLiteralNode
kabukawa [Wed, 12 Dec 2018 06:28:54 +0000 (15:28 +0900)] 
ARROW-3996: [C++] Add missing packages on Linux
[C++] Build requirement libraries add to README.md.
* autoconf
* Jemalloc
* boost-regex
Author: kabukawa <kabukawa@gmail.com>
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3157 from kabukawa/apache-arrow-develop and squashes the following commits:
a9f465a3 <Kouhei Sutou> Add autoconf
45568fd1 <kabukawa> Instration requirement add.(modified)
dcee4855 <kabukawa> Instration requirement add.
Chao Sun [Wed, 12 Dec 2018 00:21:35 +0000 (17:21 -0700)] 
ARROW-3960: [Rust] remove extern crate for Rust 2018
This is a trivial change to remove "extern crate" definitions
in lib.rs, to follow the new module system in Rust 2018 edition.
Author: Chao Sun <sunchao@apache.org>
Author: Chao Sun <sunchao@uber.com>
Closes #3125 from sunchao/ARROW-3960 and squashes the following commits:
56a4393 <Chao Sun> Remove star import
0e5d06c <Chao Sun> Fixing json_internal error
53c13a9 <Chao Sun> ARROW-3960: remove extern crate for Rust 2018
Antoine Pitrou [Tue, 11 Dec 2018 14:16:06 +0000 (15:16 +0100)]
ARROW-3986: [C++] Write prose documentation
Author: Antoine Pitrou <antoine@python.org>
Closes #3149 from pitrou/ARROW-3986-cpp-prose-doc and squashes the following commits:
77e37940 <Antoine Pitrou> ARROW-3986: Write prose documentation
Korn, Uwe [Tue, 11 Dec 2018 14:01:03 +0000 (15:01 +0100)] 
ARROW-3995: [CI] Use understandable names on Travis
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com>
Closes #3158 from xhochy/travis-names and squashes the following commits:
f268f276 <Korn, Uwe> ARROW-3995: Use understandable names on Travis
Brian Hulette [Tue, 11 Dec 2018 03:30:34 +0000 (21:30 -0600)] 
ARROW-3993: [JS] CI Jobs Failing
Use `gulp@4.0.0` rather than `gulp@next`
Author: Brian Hulette <hulettbh@gmail.com>
Closes #3153 from TheNeuralBit/gulp-fix and squashes the following commits:
e5d2e74c4 <Brian Hulette> gulp@{next->4.0.0}
Andy Grove [Tue, 11 Dec 2018 03:03:34 +0000 (20:03 -0700)] 
ARROW-3880: [Rust] Implement simple math operations for numeric arrays
Author: Andy Grove <andygrove73@gmail.com>
Closes #3033 from andygrove/ARROW-3880 and squashes the following commits:
17cd418 <Andy Grove> merge from master
afb3518 <Andy Grove> Move min and max to array_ops
0c77c61 <Andy Grove> code cleanup
f8bfb41 <Andy Grove> move comparison ops to array_ops
7a5975e <Andy Grove> Move math ops into new array_ops source file
7946142 <Andy Grove> Address PR feedback
adfe4b0 <Andy Grove> merge from master and fix conflicts
5ed5f6e <Andy Grove> add comparison operations
42c68af <Andy Grove> re-implement with generics
963def6 <Andy Grove> Merge branch 'master' into ARROW-3880
729cd9a <Andy Grove> fix formatting
405c63e <Andy Grove> re-implement using macros
5876fb7 <Andy Grove> save work
a2b87e2 <Andy Grove> merge from master, comment out new methods
2a43b3f <Andy Grove> merge from master
06bbc4a <Andy Grove> improve handling of divide by zero, format for rust nightly
1ea98cf <Andy Grove> Improve error handling
dcad28a <Andy Grove> cargo fmt
12dc05b <Andy Grove> Implement simple math operations for numeric arrays
Wes McKinney [Mon, 10 Dec 2018 23:55:23 +0000 (17:55 -0600)]
ARROW-3248: [C++] Add "arrow" prefix to Arrow core unit tests, use PREFIX instead of file name for csv, io, ipc tests. Modular target cleanup
I added a section to the cpp/README.md about the modular build targets.
Author: Wes McKinney <wesm+git@apache.org>
Closes #3152 from wesm/ARROW-3248 and squashes the following commits:
ba3a3e58c <Wes McKinney> Need to add arrow- prefix to some Travis scripts
1f3daaf78 <Wes McKinney> Rename io/ipc tests/executables. Add appropriate labels/prefixes to all unit tests/benchmarks. Add labels option to ADD_BENCHMARK
Wes McKinney [Mon, 10 Dec 2018 20:02:21 +0000 (14:02 -0600)]
ARROW-3641: [Python] Remove unneeded public keyword from pyarrow public C APIs
According to
https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#c-api-declarations
it is not necessary to use `public` here. If we want to be able to refer to Cython extension types at the C API level (at some point, this may not be a bad idea), then we must use `public` with those.
Author: Wes McKinney <wesm+git@apache.org>
Closes #3147 from wesm/ARROW-3641 and squashes the following commits:
f09902cb4 <Wes McKinney> Remove unneeded public keyword from pyarrow public APIs
Romain Francois [Mon, 10 Dec 2018 15:51:41 +0000 (09:51 -0600)] 
ARROW-3942: [R] Feather api fixes
Some fixes to follow up open #3043, and added the columns argument to `read_feather` that can be:
- character vector
- integer vector : 1-based in R
- NULL: to get all columns (the default)
Also adds `as_tibble` argument to read_feather to switch between data.frame and arrow::Table return value
Author: Romain Francois <romain@purrple.cat>
Closes #3106 from romainfrancois/ARROW-3942/feather and squashes the following commits:
13061af4d <Romain Francois> fixed link in documentation
ce414c153 <Romain Francois> + as_tibble argument to read_feather()
d6c30a38b <Romain Francois> + columns argument to read_feather()
46a6fbb69 <Romain Francois> Update feather factories
Krisztián Szűcs [Mon, 10 Dec 2018 15:42:53 +0000 (16:42 +0100)] 
ARROW-2624: [Python] Random schema generator for Arrow conversion and Parquet testing
- introduced hypothesis to generate pyarrow types, fields and schemas
- test cases to highlight the functionality provided by hypothesis
- hypothesis tests are disabled by default
- represent kev-value metadata as OrderedDict on python side instead of plain dicts (pickling was indeterministic, found this bug by hypo)
- unified multiple metadata conversion paths to a single one (pyarrow_wrap_metadata, pyarrow_unwrap_metadata)
Also resolves: [ARROW-3901: [Python] Make Schema hashable](https://issues.apache.org/jira/browse/ARROW-3901)
Follow-up issue: [ARROW-3903: [Python] Random data generator for ... testing](https://issues.apache.org/jira/browse/ARROW-3903)
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes #3046 from kszucs/ARROW-2624 and squashes the following commits:
3e27ad15 <Krisztián Szűcs> hypo profiles
88b107bb <Krisztián Szűcs> install hypothesis for msvc wheel test
8fb6d0bc <Krisztián Szűcs> make pyarrow_wrap_metadata private
80a276be <Krisztián Szűcs> manylinux
26e6ecd6 <Krisztián Szűcs> manylinux
e385d243 <Krisztián Szűcs> manylinux
b6fe7576 <Krisztián Szűcs> append in unwrap
0e28e5df <Krisztián Szűcs> ci fixes
efeb65ee <Krisztián Szűcs> use conde_env_python.yml in travis
1f7ad6b6 <Krisztián Szűcs> don't validate metadata type pyarrow_wrap_metadata
14e444d9 <Krisztián Szűcs> introduce requirements-test.txt
11b020c0 <Krisztián Szűcs> install hypothesis on appveyor and travis
6bd5b21e <Krisztián Szűcs> license header
a8fae546 <Krisztián Szűcs> remove unbox_metadata
e8c0f3f5 <Krisztián Szűcs> add hypo as test dependency; hashing test
e7bab691 <Krisztián Szűcs> remove box_metadata
f1ae290e <Krisztián Szűcs> hypothesis strategies for pyarrow types; deterministic key-value metadata conversions
Praveen [Mon, 10 Dec 2018 15:39:43 +0000 (16:39 +0100)] 
ARROW-3983: [Gandiva][Crossbow] Link Boost statically in JAR packaging scripts
Use static boost libraries while packaging Gandiva.
Author: Praveen <praveen@dremio.com>
Closes #3145 from praveenbingo/ARROW-3983 and squashes the following commits:
2a704969 <Praveen> ARROW-3983: Use static version of boost.
Paddy Horan [Mon, 10 Dec 2018 15:30:30 +0000 (16:30 +0100)] 
ARROW-3687: [Rust] Anything measuring array slots should be `usize`
Author: Paddy Horan <paddyhoran@hotmail.com>
Closes #3142 from paddyhoran/ARROW-3687 and squashes the following commits:
c0a75e9c <Paddy Horan> Fixed lint issues (outdated stable)
0b39fe87 <Paddy Horan> Updated subtraction to be checked
c4c223c5 <Paddy Horan> Fixing lints for stable fmt
d6aec71c <Paddy Horan> All values measuring array slots changed to `usize`
Pindikura Ravindra [Mon, 10 Dec 2018 15:28:47 +0000 (09:28 -0600)]
ARROW-3977: [Gandiva] fix label during ctest invoc
Author: Pindikura Ravindra <ravindra@dremio.com>
Closes #3139 from pravindra/ci and squashes the following commits:
3372401c3 <Pindikura Ravindra> ARROW-3977: temporary disable valgrind
c51b23aff <Pindikura Ravindra> ARROW-3977: fix label during ctest invoc
Antoine Pitrou [Mon, 10 Dec 2018 15:15:01 +0000 (09:15 -0600)]
ARROW-3980: [C++] Fix CRTP use in json-simple.cc
Nudge the compiler into devirtualizing method calls.
Author: Antoine Pitrou <antoine@python.org>
Closes #3144 from pitrou/ARROW-3980-json-crtp and squashes the following commits:
ef96713a2 <Antoine Pitrou> ARROW-3980: Fix CRTP use in json-simple.cc
Antoine Pitrou [Mon, 10 Dec 2018 15:12:40 +0000 (09:12 -0600)]
ARROW-3727: [Python] Document use of foreign_buffer()
Author: Antoine Pitrou <antoine@python.org>
Closes #3146 from pitrou/ARROW-3727-foreign-buffer-doc and squashes the following commits:
e81a5f0cf <Antoine Pitrou> ARROW-3727: Document use of foreign_buffer()
Wes McKinney [Mon, 10 Dec 2018 13:05:14 +0000 (14:05 +0100)]
ARROW-3792: [C++] Writing a list-type chunked column to Parquet fails if any chunk is 0-length
Thanks to @tanyaschlusser to providing a minimal reproduction to help find the underlying problem
Author: Wes McKinney <wesm+git@apache.org>
Closes #3141 from wesm/ARROW-3792 and squashes the following commits:
1ed82a57 <Wes McKinney> Add test case and fix
Praveen [Mon, 10 Dec 2018 03:45:10 +0000 (21:45 -0600)]
ARROW-3970: [Gandiva][C++] Remove unnecessary boost dependencies.
Removed the dynamic dependencies since we do not need them.
Author: Praveen <praveen@dremio.com>
Closes #3137 from praveenbingo/ARROW-3970 and squashes the following commits:
6e3a6bbdc <Praveen> ARROW-3970: Added more time for a benchmark test.
fbb551645 <Praveen> ARROW-3970: Remove unnecessary boost dynamic dependencies.
Andy Grove [Mon, 10 Dec 2018 03:31:21 +0000 (12:31 +0900)]
ARROW-3885: [Rust] Release prepare step should increment Rust version
Author: Andy Grove <andygrove73@gmail.com>
Closes #3096 from andygrove/ARROW-3885 and squashes the following commits:
7d15ee77 <Andy Grove> add commit step
0d98c2cf <Andy Grove> revert to 0.11.0 ready for next prepare step
a7f60835 <Andy Grove> update release prepare step to increment Rust version
ac6e5fc0 <Andy Grove> Set version to 0.11.0 and update prepare script
b39b7c4b <Andy Grove> Update Rust version to 0.12.0
Yosuke Shiro [Mon, 10 Dec 2018 01:50:03 +0000 (10:50 +0900)]
ARROW-3967: [Gandiva] [C++] Make node.h public
Because some methods in node.h is useful in bindings.
C GLib Gandiva bindings want to use LiteralNode::holder() to access raw literal data.
Author: Yosuke Shiro <yosuke.shiro615@gmail.com>
Closes #3135 from shiro615/make-gandiva-node-header-public and squashes the following commits:
5950c52b <Yosuke Shiro> Make node.h public
c-bata [Mon, 10 Dec 2018 01:37:30 +0000 (10:37 +0900)] 
ARROW-3964: [Go] Refactor examples of csv reader
Example of godoc doesn't include input file(testdata/simple.csv). So it's hard to understand the output. This PR refactors it.
<img width="652" alt="screenshot" src="https://user-images.githubusercontent.com/5564044/
49687567-
0f6fc380-fb48-11e8-895a-
ee0d554ef7c8.png">
https://godoc.org/github.com/apache/arrow/go/arrow/csv
Author: c-bata <contact@c-bata.link>
Closes #3131 from c-bata/refactor-csv-reader-example and squashes the following commits:
eed8e29b <c-bata> Refactor examples of csv reader for Go
Krisztián Szűcs [Sun, 9 Dec 2018 21:22:08 +0000 (22:22 +0100)]
ARROW-3963: [Packaging/Docker] Nightly test for building sphinx documentations
Test is here: https://github.com/kszucs/crossbow/branches/all?utf8=%E2%9C%93&query=docker-docs
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes #3130 from kszucs/ARROW-3963 and squashes the following commits:
0b5be2cc <Krisztián Szűcs> add docker-docs to docker group
1575909e <Krisztián Szűcs> path corrections
51768fc0 <Krisztián Szűcs> use sphinx-build command instead of setup.py
60635acc <Krisztián Szűcs> error msg
a93fcad6 <Krisztián Szűcs> merge _as_type and ensure_type
8d3d58fd <Krisztián Szűcs> nightly test for building cpp and python docs
Andy Grove [Sun, 9 Dec 2018 20:45:15 +0000 (14:45 -0600)]
ARROW-3969: [Rust] Format using stable rustfmt
Author: Andy Grove <andygrove73@gmail.com>
Closes #3138 from andygrove/ARROW-3969 and squashes the following commits:
99b6256b6 <Andy Grove> move rustfmt installation into travis_install_cargo
55ab06fee <Andy Grove> Be more explicit and don't assume order of stable vs nightly builds
5fed7dbc2 <Andy Grove> simplify
cca7da3ab <Andy Grove> oops, wrong command
9b2e5b771 <Andy Grove> Format using stable rustfmt
Antoine Pitrou [Sun, 9 Dec 2018 19:28:34 +0000 (13:28 -0600)]
ARROW-3303: [C++] API for creating arrays from simple JSON string
Author: Antoine Pitrou <antoine@python.org>
Closes #3084 from pitrou/ARROW-3303-json-values and squashes the following commits:
1b9f4b510 <Antoine Pitrou> ARROW-3303: API for creating arrays from simple JSON string
Kengo Seki [Sun, 9 Dec 2018 17:30:20 +0000 (11:30 -0600)] 
ARROW-3940: [Python/Documentation] Add required packages to the development instruction
I mistakenly closed #3102 so I'll submit the revised PR. @wesm would you take a look at this?
Author: Kengo Seki <sekikn@apache.org>
Closes #3126 from sekikn/ARROW-3940-2 and squashes the following commits:
15e369eb0 <Kengo Seki> ARROW-3940: Add required packages to the development instruction
Kengo Seki [Sat, 8 Dec 2018 13:12:42 +0000 (14:12 +0100)]
ARROW-3961: [Python/Documentation] Fix wrong path in the pyarrow README
Author: Kengo Seki <sekikn@apache.org>
Closes #3127 from sekikn/ARROW-3961 and squashes the following commits:
87d03e4a <Kengo Seki> ARROW-3961: Fix wrong path in the pyarrow README
Philipp Moritz [Sat, 8 Dec 2018 02:27:27 +0000 (18:27 -0800)]
ARROW-3950: [Plasma] Make loading the TensorFlow op optional
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes #3117 from pcmoritz/tf-optional-loading and squashes the following commits:
0404e7ede <Philipp Moritz> fix
2b0d25432 <Philipp Moritz> make loading the tensorflow op optional
Yosuke Shiro [Sat, 8 Dec 2018 01:38:03 +0000 (10:38 +0900)]
ARROW-3955: [GLib] Add (transfer full) to free when no longer needed
Author: Yosuke Shiro <yosuke.shiro615@gmail.com>
Closes #3122 from shiro615/glib-fix-memory-leak and squashes the following commits:
0a60f20a <Yosuke Shiro> Add missing Ruby/GObjectIntrospection version check
6b0f7ee2 <Yosuke Shiro> Add (transfer full) to free when no longer needed
Andy Grove [Sat, 8 Dec 2018 00:34:46 +0000 (09:34 +0900)]
ARROW-3952: [Rust] Upgrade to Rust 2018 Edition
See https://hacks.mozilla.org/2018/12/rust-2018-is-here/ for more info.
Changes:
- Add `edition = "2018"` to Cargo.toml
- Change lots of import statements due to changes in module system in Rust 2018
- Remove `mut` from one variable assignment
Author: Andy Grove <andygrove73@gmail.com>
Closes #3119 from andygrove/ARROW-3952 and squashes the following commits:
bfebf9e4 <Andy Grove> cargo fmt
bda17699 <Andy Grove> remove unnecessary extern crate statements
048b00cd <Andy Grove> Upgrade to Rust 2018
f843ad46 <Andy Grove> Save
c60ac679 <Andy Grove> Specify Rust 2018
Kouhei Sutou [Fri, 7 Dec 2018 05:45:10 +0000 (14:45 +0900)]
ARROW-3948: [GLib][CI] Set timeout to Homebrew
We can't get Homebrew log when Travis detects hanged Homebrew process and kill the CI job.
We need to detect hanged Homebrew process and kill the Homebrew process to get Homebrew log.
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3115 from kou/glib-ci-timeout-homebrew and squashes the following commits:
fe23b049 <Kouhei Sutou> Set timeout to Homebrew
Antoine Pitrou [Fri, 7 Dec 2018 03:41:48 +0000 (21:41 -0600)]
ARROW-3930: [C++] Avoid using Mersenne Twister for random test data
Mersenne Twister is much slower than the default C++ random engine (at least on GNU libstdc++), and we're not looking for very good statistical properties either.
Author: Antoine Pitrou <antoine@python.org>
Closes #3074 from pitrou/ARROW-3930-faster-test-random and squashes the following commits:
52993b39a <Antoine Pitrou> ARROW-3930: Avoid using Mersenne Twister for random test data
Kouhei Sutou [Fri, 7 Dec 2018 01:25:34 +0000 (10:25 +0900)]
ARROW-3946: [GLib] Add support for union
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3112 from kou/glib-union-array and squashes the following commits:
b1d8870f <Kouhei Sutou> Fix a typo
c45e7fc0 <Kouhei Sutou> Add support for union
Wes McKinney [Thu, 6 Dec 2018 20:26:05 +0000 (14:26 -0600)]
PARQUET-1471: [C++] TypedStatistics<T>::UpdateSpaced reads out of bounds value when there are more definition levels than spaced values
This bug arose in https://github.com/apache/arrow/pull/3074 as a result of random data changing
Author: Wes McKinney <wesm+git@apache.org>
Closes #3114 from wesm/PARQUET-1471 and squashes the following commits:
4108d8e60 <Wes McKinney> MSVC was including src/parquet/arrow/test-util.h instead of src/arrow/test-util.h
755c9905f <Wes McKinney> Add missing include guard to parquet/arrow/test-util.h
8410efa6a <Wes McKinney> Unit test reproducing valgrind error on optional list
6632f7cd2 <Wes McKinney> Fix out of bounds access when updating column statistics when max_definition_level > 1
Romain Francois [Thu, 6 Dec 2018 20:17:41 +0000 (14:17 -0600)]
ARROW-3941: [R] RecordBatchStreamReader$schema
https://issues.apache.org/jira/browse/ARROW-3941
follow up to #3043 to fix api of classes RecordBatchFileReader, RecordBatchStreamReader, RecordBatchFileWriter, RecordBatchStreamWriter
Author: Romain Francois <romain@purrple.cat>
Closes #3104 from romainfrancois/ARROW-3941/RecordBatchStreamReader and squashes the following commits:
0d2494fc9 <Romain Francois> s/get/read/g
01bd167db <Romain Francois> fix RecordBatch(Stream|File)(Reader|Writer) api
Justin Dunham [Thu, 6 Dec 2018 15:13:45 +0000 (09:13 -0600)] 
ARROW-3945: [Website] Update website for Gandiva donation
Added a blog post for the donation of Gandiva.
Author: Justin Dunham <riboflavin@justindunham.net>
Closes #3110 from riboflavin/master and squashes the following commits:
8efd1a0de <Justin Dunham> Add blog post about Gandiva donation
François Saint-Jacques [Thu, 6 Dec 2018 15:07:16 +0000 (09:07 -0600)]
ARROW-3586: [Python] Add test ensuring no segfault
As reported in ARROW-3586, a conversion to categorical on an empty array
would trigger a segfault. This test ensure it is safe to do.
Author: François Saint-Jacques <fsaintjacques@gmail.com>
Closes #3116 from fsaintjacques/ARROW-3586-segfault and squashes the following commits:
7f779f1bf <François Saint-Jacques> ARROW-3586: Add test ensuring no segfault
Kouhei Sutou [Thu, 6 Dec 2018 05:53:36 +0000 (14:53 +0900)]
ARROW-3921: [GLib][CI] Log Homebrew output
Homebrew stuck problem is still occurred: https://github.com/apache/arrow/pull/3095#issuecomment-
444559301
This change starts logging Homebrew output to debug this case.
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3113 from kou/glib-ci-log-homebrew and squashes the following commits:
60856208 <Kouhei Sutou> Log Homebrew output
Kouhei Sutou [Thu, 6 Dec 2018 03:26:21 +0000 (12:26 +0900)]
ARROW-3209: [C++] Rename libarrow_gpu to libarrow_cuda
Also rename the arrow::gpu namespace to arrow::cuda.
Author: Kouhei Sutou <kou@clear-code.com>
Author: Antoine Pitrou <antoine@python.org>
Closes #3088 from pitrou/ARROW-3209-rename-arrow-gpu-to-cuda and squashes the following commits:
d16eb713 <Kouhei Sutou> Support arrow-cuda.pc in build directory
32381216 <Kouhei Sutou> Fix path
219ffa28 <Kouhei Sutou> Rename GPU to CUDA
f8683cac <Kouhei Sutou> Rename GPU to CUDA
c4f7f381 <Kouhei Sutou> Rename red-arrow-gpu to red-arrow-cuda
267de1b8 <Kouhei Sutou> Rename libarrow-gpu-glib to libarrow-cuda-glib
9687346a <Antoine Pitrou> ARROW-3209: Rename libarrow_gpu to libarrow_cuda
Chao Sun [Thu, 6 Dec 2018 02:10:18 +0000 (11:10 +0900)]
ARROW-3908: [Rust] Update rust dockerfile to use nightly toolchain
This replaces the base docker image for Rust from stable to nightly.
Author: Chao Sun <sunchao@apache.org>
Closes #3101 from sunchao/ARROW-3908 and squashes the following commits:
c340038e <Chao Sun> ARROW-3908: Update rust dockerfile to use nightly toolchain
Uwe L. Korn [Thu, 6 Dec 2018 01:54:57 +0000 (19:54 -0600)] 
ARROW-3834: [Doc] Merge C++ and Python documentation
Author: Uwe L. Korn <uwelk@xhochy.com>
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes #2856 from xhochy/doc-merge and squashes the following commits:
5b687ffa7 <Wes McKinney> Add simple README for the format/ directory
071d16ad7 <Uwe L. Korn> Move format specifications back to /format/
337088bfd <Uwe L. Korn> Review comments
fbe99c96c <Uwe L. Korn> Add more C++ docs
78a5eaf4b <Uwe L. Korn> Fix Python docs build
0b4dd3341 <Uwe L. Korn> Rename doc to docs
918e76244 <Uwe L. Korn> Convert format docs to reST
7aeff65ef <Uwe L. Korn> Add doc generation to docker-compose
185cba8fb <Uwe L. Korn> Add pre-commit check for RAT
671d24466 <Uwe L. Korn> Fix references to format documents
bdd824c87 <Uwe L. Korn> Move doc to top-level
985d42806 <Uwe L. Korn> Move Sphinx doc to top-level directory
f7d5e92d9 <Uwe L. Korn> Build C++ API docs
7850db8a6 <Uwe L. Korn> Add breathe as a requirement
d4cf54259 <Uwe L. Korn> Fix linter issues
fd75660e6 <Korn, Uwe> Fix Sphinx build for sphinx>=1.8
9be6fbe6f <Korn, Uwe> Merge C++ and Python documentation
Wes McKinney [Thu, 6 Dec 2018 00:05:58 +0000 (18:05 -0600)]
ARROW-3291: [C++] Add string_view-based constructor for BufferReader
This makes the API simpler for creating emphemeral IO sources. The caller is responsible for managing memory lifetime in this case
Author: Wes McKinney <wesm+git@apache.org>
Closes #3093 from wesm/ARROW-3291 and squashes the following commits:
125d37546 <Wes McKinney> Add string_view-based constructor for BufferReader, use in various unit tests
Bryan Cutler [Wed, 5 Dec 2018 18:26:26 +0000 (10:26 -0800)] 
ARROW-3272: [Java][Docs] Add documentation about Java code style
The Java code style is based off of the Google style guide with some small differences. This adds documentation with a link to the Google style guide and a list of what is different.
Author: Bryan Cutler <cutlerb@gmail.com>
Closes #3090 from BryanCutler/java-checkstyle-docs-ARROW-3272 and squashes the following commits:
73a1206 <Bryan Cutler> change wording
bad8c59 <Bryan Cutler> add docs about Java code style
Sebastien Binet [Wed, 5 Dec 2018 18:06:43 +0000 (19:06 +0100)] 
ARROW-3929: [Go] improve CSV reader memory usage
This CL enables the encoding/csv reader to reuse the memory used by
records, from row to row, and thus reduce memory pressure on Go's GC.
```
$> benchstat old.txt new.txt
name old time/op new time/op delta
Read/rows=10_cols=1_chunks=10-8 39.4µs ±18% 42.6µs ±24% ~ (p=0.218 n=10+10)
Read/rows=10_cols=10_chunks=10-8 293µs ±23% 280µs ±24% ~ (p=0.400 n=10+9)
Read/rows=10_cols=100_chunks=10-8 2.72ms ±24% 2.56ms ±20% ~ (p=0.353 n=10+10)
Read/rows=10_cols=1000_chunks=10-8 24.3ms ± 2% 24.0ms ± 3% ~ (p=0.059 n=8+9)
Read/rows=100_cols=1_chunks=10-8 74.9µs ±11% 62.1µs ±19% -17.21% (p=0.004 n=10+10)
Read/rows=100_cols=10_chunks=10-8 559µs ±21% 474µs ±21% -15.12% (p=0.009 n=10+10)
Read/rows=100_cols=100_chunks=10-8 5.53ms ±21% 4.36ms ±16% -21.27% (p=0.000 n=10+9)
Read/rows=100_cols=1000_chunks=10-8 41.9ms ± 3% 42.2ms ±13% ~ (p=0.684 n=10+10)
Read/rows=1000_cols=1_chunks=10-8 421µs ±13% 320µs ±10% -23.98% (p=0.000 n=10+10)
Read/rows=1000_cols=10_chunks=10-8 3.24ms ±24% 2.63ms ±15% -18.77% (p=0.007 n=10+10)
Read/rows=1000_cols=100_chunks=10-8 33.0ms ±17% 27.0ms ±19% -18.09% (p=0.001 n=10+10)
Read/rows=1000_cols=1000_chunks=10-8 219ms ± 1% 211ms ± 2% -3.81% (p=0.000 n=9+10)
Read/rows=10000_cols=1_chunks=10-8 3.66ms ±11% 2.91ms ±10% -20.27% (p=0.000 n=10+10)
Read/rows=10000_cols=10_chunks=10-8 31.8ms ±16% 25.6ms ±15% -19.66% (p=0.000 n=10+10)
Read/rows=10000_cols=100_chunks=10-8 192ms ± 1% 182ms ± 1% -5.19% (p=0.000 n=10+10)
Read/rows=10000_cols=1000_chunks=10-8 1.99s ± 1% 1.93s ± 2% -3.26% (p=0.000 n=9+9)
Read/rows=100000_cols=1_chunks=10-8 32.9ms ± 4% 26.1ms ± 4% -20.75% (p=0.000 n=10+10)
Read/rows=100000_cols=10_chunks=10-8 203ms ± 1% 198ms ± 7% ~ (p=0.123 n=10+10)
Read/rows=100000_cols=100_chunks=10-8 2.00s ± 1% 1.92s ± 1% -4.24% (p=0.000 n=10+8)
Read/rows=100000_cols=1000_chunks=10-8 22.7s ± 2% 22.0s ± 2% -3.31% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
Read/rows=10_cols=1_chunks=10-8 32.7kB ± 0% 32.2kB ± 0% -1.32% (p=0.000 n=10+10)
Read/rows=10_cols=10_chunks=10-8 281kB ± 0% 277kB ± 0% -1.54% (p=0.000 n=10+10)
Read/rows=10_cols=100_chunks=10-8 2.77MB ± 0% 2.73MB ± 0% -1.58% (p=0.000 n=10+10)
Read/rows=10_cols=1000_chunks=10-8 27.8MB ± 0% 27.3MB ± 0% -1.59% (p=0.000 n=9+9)
Read/rows=100_cols=1_chunks=10-8 44.0kB ± 0% 39.3kB ± 0% -10.80% (p=0.000 n=10+10)
Read/rows=100_cols=10_chunks=10-8 381kB ± 0% 333kB ± 0% -12.48% (p=0.000 n=10+10)
Read/rows=100_cols=100_chunks=10-8 3.78MB ± 0% 3.29MB ± 0% -12.75% (p=0.000 n=10+10)
Read/rows=100_cols=1000_chunks=10-8 37.9MB ± 0% 33.1MB ± 0% -12.83% (p=0.000 n=10+9)
Read/rows=1000_cols=1_chunks=10-8 200kB ± 0% 152kB ± 0% -23.99% (p=0.000 n=10+10)
Read/rows=1000_cols=10_chunks=10-8 1.84MB ± 0% 1.36MB ± 0% -26.08% (p=0.000 n=10+9)
Read/rows=1000_cols=100_chunks=10-8 18.4MB ± 0% 13.5MB ± 0% -26.44% (p=0.000 n=9+10)
Read/rows=1000_cols=1000_chunks=10-8 184MB ± 0% 135MB ± 0% -26.62% (p=0.000 n=10+10)
Read/rows=10000_cols=1_chunks=10-8 1.65MB ± 0% 1.17MB ± 0% -29.02% (p=0.000 n=10+10)
Read/rows=10000_cols=10_chunks=10-8 15.7MB ± 0% 10.9MB ± 0% -30.65% (p=0.000 n=10+10)
Read/rows=10000_cols=100_chunks=10-8 156MB ± 0% 108MB ± 0% -31.12% (p=0.000 n=10+8)
Read/rows=10000_cols=1000_chunks=10-8 1.58GB ± 0% 1.09GB ± 0% -31.06% (p=0.000 n=10+10)
Read/rows=100000_cols=1_chunks=10-8 20.1MB ± 0% 15.3MB ± 0% -23.93% (p=0.000 n=10+9)
Read/rows=100000_cols=10_chunks=10-8 197MB ± 0% 149MB ± 0% -24.39% (p=0.000 n=10+8)
Read/rows=100000_cols=100_chunks=10-8 1.96GB ± 0% 1.47GB ± 0% -24.86% (p=0.000 n=10+10)
Read/rows=100000_cols=1000_chunks=10-8 19.7GB ± 0% 14.7GB ± 0% -25.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Read/rows=10_cols=1_chunks=10-8 319 ± 0% 310 ± 0% -2.82% (p=0.000 n=10+10)
Read/rows=10_cols=10_chunks=10-8 2.63k ± 0% 2.62k ± 0% -0.34% (p=0.000 n=10+10)
Read/rows=10_cols=100_chunks=10-8 25.7k ± 0% 25.7k ± 0% -0.04% (p=0.000 n=10+10)
Read/rows=10_cols=1000_chunks=10-8 256k ± 0% 256k ± 0% -0.00% (p=0.000 n=10+10)
Read/rows=100_cols=1_chunks=10-8 524 ± 0% 425 ± 0% -18.89% (p=0.000 n=10+10)
Read/rows=100_cols=10_chunks=10-8 3.02k ± 0% 2.92k ± 0% -3.27% (p=0.000 n=10+10)
Read/rows=100_cols=100_chunks=10-8 28.0k ± 0% 27.9k ± 0% -0.35% (p=0.000 n=10+10)
Read/rows=100_cols=1000_chunks=10-8 277k ± 0% 277k ± 0% -0.04% (p=0.000 n=10+10)
Read/rows=1000_cols=1_chunks=10-8 2.43k ± 0% 1.44k ± 0% -41.04% (p=0.000 n=10+10)
Read/rows=1000_cols=10_chunks=10-8 5.92k ± 0% 4.92k ± 0% -16.87% (p=0.000 n=10+10)
Read/rows=1000_cols=100_chunks=10-8 40.8k ± 0% 39.8k ± 0% -2.45% (p=0.000 n=10+10)
Read/rows=1000_cols=1000_chunks=10-8 389k ± 0% 388k ± 0% -0.26% (p=0.000 n=10+10)
Read/rows=10000_cols=1_chunks=10-8 20.6k ± 0% 10.6k ± 0% -48.58% (p=0.000 n=10+10)
Read/rows=10000_cols=10_chunks=10-8 25.4k ± 0% 15.4k ± 0% -39.33% (p=0.000 n=10+10)
Read/rows=10000_cols=100_chunks=10-8 73.8k ± 0% 63.8k ± 0% -13.56% (p=0.000 n=10+10)
Read/rows=10000_cols=1000_chunks=10-8 557k ± 0% 547k ± 0% -1.79% (p=0.000 n=10+10)
Read/rows=100000_cols=1_chunks=10-8 201k ± 0% 101k ± 0% -49.78% (p=0.000 n=10+10)
Read/rows=100000_cols=10_chunks=10-8 208k ± 0% 108k ± 0% -48.02% (p=0.000 n=10+10)
Read/rows=100000_cols=100_chunks=10-8 282k ± 0% 182k ± 0% -35.49% (p=0.000 n=10+10)
Read/rows=100000_cols=1000_chunks=10-8 1.02M ± 0% 0.92M ± 0% -9.83% (p=0.000 n=10+10)
```
Author: Sebastien Binet <binet@cern.ch>
Closes #3073 from sbinet/issue-3929 and squashes the following commits:
67a3272c <Sebastien Binet> ARROW-3929: improve CSV reader memory usage
8eb60c52 <Sebastien Binet> ARROW-3681: Add benchmarks for CSV reader
Sebastien Binet [Wed, 5 Dec 2018 18:00:21 +0000 (19:00 +0100)]
ARROW-3681: [Go] Add benchmarks for CSV reader
This CL adds a set of benchmarks for the CSV reader type.
E.g.:
```
$> go test -run=NONE -bench=Read/rows=.*_cols=.*_chunks=-1 -benchmem
goos: linux
goarch: amd64
pkg: github.com/apache/arrow/go/arrow/csv
BenchmarkRead/rows=10_cols=1_chunks=-1-8 200000 10219 ns/op 9560 B/op 73 allocs/op
BenchmarkRead/rows=10_cols=10_chunks=-1-8 30000 75434 ns/op 47264 B/op 368 allocs/op
BenchmarkRead/rows=10_cols=100_chunks=-1-8 3000 489027 ns/op 426960 B/op 3255 allocs/op
BenchmarkRead/rows=10_cols=1000_chunks=-1-8 200 5400913 ns/op 4308912 B/op 32072 allocs/op
BenchmarkRead/rows=100_cols=1_chunks=-1-8 50000 45297 ns/op 30552 B/op 268 allocs/op
BenchmarkRead/rows=100_cols=10_chunks=-1-8 5000 333999 ns/op 195520 B/op 661 allocs/op
BenchmarkRead/rows=100_cols=100_chunks=-1-8 500 2660322 ns/op 1869777 B/op 4538 allocs/op
BenchmarkRead/rows=100_cols=1000_chunks=-1-8 50
25683147 ns/op
18805425 B/op 43256 allocs/op
BenchmarkRead/rows=1000_cols=1_chunks=-1-8 5000 423213 ns/op 218968 B/op 2086 allocs/op
BenchmarkRead/rows=1000_cols=10_chunks=-1-8 500 2420959 ns/op 1591808 B/op 2614 allocs/op
BenchmarkRead/rows=1000_cols=100_chunks=-1-8 50
21765485 ns/op
15474384 B/op 7841 allocs/op
BenchmarkRead/rows=1000_cols=1000_chunks=-1-8 5
222083917 ns/op
154949808 B/op 60060 allocs/op
BenchmarkRead/rows=10000_cols=1_chunks=-1-8 500 3938427 ns/op 3083224 B/op 20123 allocs/op
BenchmarkRead/rows=10000_cols=10_chunks=-1-8 50
22066971 ns/op
20298368 B/op 20903 allocs/op
BenchmarkRead/rows=10000_cols=100_chunks=-1-8 5
209542066 ns/op
193038672 B/op 28651 allocs/op
BenchmarkRead/rows=10000_cols=1000_chunks=-1-8 1
2696959353 ns/op
1939814576 B/op 106070 allocs/op
BenchmarkRead/rows=100000_cols=1_chunks=-1-8 30
35208837 ns/op
31869150 B/op 200155 allocs/op
BenchmarkRead/rows=100000_cols=10_chunks=-1-8 5
219030269 ns/op
183553152 B/op 201125 allocs/op
BenchmarkRead/rows=100000_cols=100_chunks=-1-8 1
2421018029 ns/op
1692336464 B/op 210762 allocs/op
BenchmarkRead/rows=100000_cols=1000_chunks=-1-8 1
28196721844 ns/op
16891740336 B/op 307082 allocs/op
PASS
ok github.com/apache/arrow/go/arrow/csv 107.802s
```
Author: Sebastien Binet <binet@cern.ch>
Closes #3071 from sbinet/issue-3681 and squashes the following commits:
8eb60c52 <Sebastien Binet> ARROW-3681: Add benchmarks for CSV reader
Wes McKinney [Wed, 5 Dec 2018 17:33:22 +0000 (11:33 -0600)]
ARROW-3441: [Gandiva] Use common unit test creation facilities, do not produce multiple executables for the same unit tests
This also adds a "gandiva" CMake target so you can run `ninja gandiva`. Additionally, all unit tests now have a `gandiva-` prefix.
Author: Wes McKinney <wesm+git@apache.org>
Closes #3091 from wesm/ARROW-3441 and squashes the following commits:
d6f4703de <Wes McKinney> Add option to ADD_ARROW_TESTS to override global NO_TESTS option
4f006feb3 <Wes McKinney> Add prefix for precompiled tests, add to gandiva target
5bdfd3090 <Wes McKinney> Add ADD_GANDIVA_TEST function, remove GandivaBuildUtils, cleaning. Add gandiva custom target
Romain Francois [Wed, 5 Dec 2018 17:07:34 +0000 (11:07 -0600)]
ARROW-3871: [R] Replace usages of C++ GetValuesSafely with new methods on ArrayData
To get rid of `GetValuesSafely` I've had to add variants of `ArrayData.GetValues<>` that take and `offset` as a parameter, this is useful e.g. for boolean arrays or string arrays.
Author: Romain Francois <romain@purrple.cat>
Closes #3103 from romainfrancois/ARROW-3871/GetValuesSafely and squashes the following commits:
bcb108242 <Romain Francois> s/offset/absolute_offset/
acd96e3eb <Romain Francois> s/index_/index/
a1ec32f2f <Romain Francois> + ArrayData->GetValues<>(i, offset_)
ad17f4a57 <Romain Francois> using ArrayData.GetValues<>
Wes McKinney [Wed, 5 Dec 2018 16:55:45 +0000 (10:55 -0600)]
ARROW-3894: [C++] Ensure that IPC file is properly initialized even if no record batches are written
Without invoking `Start()`, the file cannot be read
Author: Wes McKinney <wesm+git@apache.org>
Closes #3095 from wesm/ARROW-3894 and squashes the following commits:
5fc65282d <Wes McKinney> Also compare schemas
1074a690a <Wes McKinney> Ensure that IPC file is properly initialized even if no record batches are written
Wes McKinney [Wed, 5 Dec 2018 16:48:44 +0000 (10:48 -0600)]
ARROW-3318: [C++] Push down read-all-batches operation on RecordBatchReader into C++
Author: Wes McKinney <wesm+git@apache.org>
Closes #3097 from wesm/ARROW-3318 and squashes the following commits:
28c31bc4e <Wes McKinney> Remove unneeded this->
03216b352 <Wes McKinney> Push down read-all operation on RecordBatchReader into C++
Chao Sun [Wed, 5 Dec 2018 16:47:37 +0000 (10:47 -0600)]
ARROW-3939: [Rust] Remove macro definition for ListArrayBuilder
Currently we only implemented a few value builder types for `ListArrayBuilder` via macros. This removes the macro definitions and use generic definition instead. As result, list builders with arbitrary value builder types can be defined. One more nice thing is that with Rust's type inference we don't need to explicitly specify the generic type parameters in most situations.
Author: Chao Sun <sunchao@apache.org>
Closes #3100 from sunchao/ARROW-3939 and squashes the following commits:
72bff6e2f <Chao Sun> ARROW-3939: Remove macro definition for ListArrayBuilder
Andy Grove [Wed, 5 Dec 2018 16:42:18 +0000 (10:42 -0600)]
ARROW-3883: [Rust] Update README
Author: Andy Grove <andygrove73@gmail.com>
Closes #3105 from andygrove/ARROW-3883 and squashes the following commits:
e01383548 <Andy Grove> update README
philjdf [Wed, 5 Dec 2018 10:59:21 +0000 (11:59 +0100)] 
ARROW-3936: [C++] Add _O_NOINHERIT to the file open flags on Windows
Unlike Linux, Windows doesn't let you delete files that are currently opened by another process. So if you create a child process while a Parquet file is open, with the current code the file handle is inherited to the child process, and the parent process can't then delete the file after closing it without the child process terminating first.
By default, Win32 file handles are not inheritable (likely because of the aforementioned problems). Except for _wsopen_s, which tries to maintain POSIX compatibility.
This is a serious problem for us.
We would argue that specifying _O_NOINHERIT by default in the _MSC_VER path is a sensible approach and would likely be the correct behaviour as it matches the main Win32 API.
However, it could be that some developers rely on the current inheritable behaviour. In which case, the Arrow public API should take a boolean argument on whether the created file descriptor should be inheritable. But this would break API backward compatibility (unless a new overloaded method is introduced).
Is forking and inheriting Arrow internal file descriptor something that Arrow actually means to support?
What do we think of the proposed fix?
Author: philjdf <git@pfelton.co.uk>
Author: Phil Felton <
39185762+philjdf@users.noreply.github.com>
Closes #3085 from philjdf/InheritableHandlesFix and squashes the following commits:
6b4be85d <Phil Felton> Fix linting issue
a9dda7fb <philjdf> Do not share handles on Windows
Andy Grove [Wed, 5 Dec 2018 08:37:34 +0000 (17:37 +0900)]
ARROW-3937: [Rust] Fix Rust nightly build (formatting rules changed)
Fix formatting using rustc 1.32.0-nightly (
b3af09205 2018-12-04) .. looks like formatting rules changed recently
Author: Andy Grove <andygrove73@gmail.com>
Closes #3098 from andygrove/ARROW-3937 and squashes the following commits:
89362b06 <Andy Grove> Fix formatting using rustc 1.32.0-nightly (
b3af09205 2018-12-04)
Brian Hulette [Wed, 5 Dec 2018 02:57:32 +0000 (18:57 -0800)]
ARROW-2993: [JS] Document minimum supported NodeJS version
Adds an engine requirement to package.json, and sets engine-strict in project-level npm config
This makes it so that running `npm install` in arrow/js yields the following error:
```
$ node -v
v9.11.2
$ npm install
npm ERR! code ENOTSUP
npm ERR! notsup Unsupported engine for apache-arrow@0.3.0: wanted: {"node":">=10.0"} (current: {"node":"9.11.2","npm":"5.6.0"})
npm ERR! notsup Not compatible with your version of node/npm: apache-arrow@0.3.0
npm ERR! notsup Not compatible with your version of node/npm: apache-arrow@0.3.0
npm ERR! notsup Required: {"node":">=10.0"}
npm ERR! notsup Actual: {"npm":"5.6.0","node":"9.11.2"}
npm ERR! A complete log of this run can be found in:
npm ERR! /home/hulettbh/.npm/_logs/2018-12-04T16_42_12_911Z-debug.log
```
Users should get a similar error when they install an arrow package if they have engine-strict set in their own npm config.
Author: Brian Hulette <hulettbh@gmail.com>
Closes #3087 from TheNeuralBit/node-version and squashes the following commits:
92a74b5b <Brian Hulette> Add engine requirement to package.json
Tanya Schlusser [Wed, 5 Dec 2018 00:18:50 +0000 (18:18 -0600)]
ARROW-3629: [Python] Add write_to_dataset to Python Sphinx API listing
Not sure whether this was all that was wanted. The new Sphinx theme is working, by the way. More comments in the JIRA. Screenshot with `write_to_dataset` highlighted:
<img width="663" alt="screen shot 2018-12-04 at 11 43 07 am" src="https://user-images.githubusercontent.com/7432951/
49462054-
c40d8c00-f7ba-11e8-8bdd-
aad09f01b6a5.png">
Author: Tanya Schlusser <tanya@tickel.net>
Closes #3089 from tanyaschlusser/ARROW-3629 and squashes the following commits:
499a4b25b <Tanya Schlusser> ARROW-3629: Add write_to_dataset to Python Sphinx API listing
Philipp Moritz [Tue, 4 Dec 2018 15:37:39 +0000 (09:37 -0600)]
ARROW-3934: [Gandiva] Only add precompiled tests if ARROW_GANDIVA_BUILD_TESTS
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes #3082 from pcmoritz/gandiva-precompiled-tests and squashes the following commits:
f47a35695 <Philipp Moritz> only add precompiled tests if ARROW_GANDIVA_BUILD_TESTS
sarutak [Tue, 4 Dec 2018 15:32:17 +0000 (09:32 -0600)] 
ARROW-3931: [C++] Make possible to build regardless of LANG
At the time of building C++ libs, CompilerInfo.cmake checks the version of compiler to be used.
How to check is string matching of output of gcc -v or like clang -v.
When LANG is not related to English, build will fail because string match fails.
The following is the case of ja_JP.UTF-8 (Japanese).
```
CMake Error at cmake_modules/CompilerInfo.cmake:92 (message):
Unknown compiler. Version info:
組み込み spec を使用しています。
COLLECT_GCC=/usr/bin/c++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
ターゲット: x86_64-redhat-linux
configure 設定: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla
--enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto
--enable-plugin --enable-initfini-array --disable-libgcj
--with-isl=/builddir/build/BUILD/gcc-4.8.5-
20150702/obj-x86_64-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.5-
20150702/obj-x86_64-redhat-linux/cloog-install
--enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64
--build=x86_64-redhat-linux
スレッドモデル: posix
gcc バージョン 4.8.5
20150623 (Red Hat 4.8.5-11) (GCC)
```
I fixed this issue by modifying CompilerInfo.cmake.
Author: sarutak <sarutak@oss.nttdata.co.jp>
Author: Wes McKinney <wesm+git@apache.org>
Closes #3077 from sarutak/fix-CompilerInfo and squashes the following commits:
0de995ef6 <Wes McKinney> Create compiler version command as list to appease MSVC
199117d00 <sarutak> Fixed to set LANG=C only when Unix or Apple
87d3d1715 <sarutak> Fixed CompilerInfo to make possible to build regardless of LANG
Brian Hulette [Tue, 4 Dec 2018 15:21:31 +0000 (07:21 -0800)]
ARROW-3892: [JS] Remove any dependency on compromised NPM flatmap-stream package
Bump npm-run-all to 4.1.5 which removes event-stream/flatmap-stream
Author: Brian Hulette <hulettbh@gmail.com>
Closes #3083 from TheNeuralBit/event-stream-fix and squashes the following commits:
a37a2131 <Brian Hulette> Bump npm-run-all to 4.1.5
Brian Hulette [Tue, 4 Dec 2018 03:12:34 +0000 (19:12 -0800)]
ARROW-2909: [JS] Add convenience function for creating a table from a list of vectors
Simplifies the creation of a `Table` from JS Arrays:
```js
const LENGTH = 20000;
const idVec = Arrow.vector.IntVector.from(
Uint32Array.from({length: LENGTH}, () => Math.round(Math.random() * 2E9))
);
const latVec = Arrow.vector.FloatVector.from(
Float32Array.from({length: LENGTH}, () => Number((Math.random() * 180 - 90).toFixed(1)))
);
const lngVec = Arrow.vector.FloatVector.from(
Float32Array.from({length: LENGTH}, () => Number((Math.random() * 360 - 180).toFixed(1)))
);
const table = Arrow.Table.fromVectors([idVec, latVec, lngVec], ['id', 'lat', 'lng'])
onsole.log(table.schema.fields.map((f) => f.name));
// [ 'id', 'lat', 'lng' ]
console.log(table.schema.fields.map((f) => f.type));
// [ Uint32 [Int] { TType: 2, children: undefined, isSigned: false, bitWidth: 32 },
// Float32 [Float] { TType: 3, children: undefined, precision: 1 },
// Float32 [Float] { TType: 3, children: undefined, precision: 1 } ]
```
Author: Brian Hulette <hulettbh@gmail.com>
Closes #2322 from TheNeuralBit/from-vectors and squashes the following commits:
12b4c286 <Brian Hulette> Update "Table from JS Arrays" example
cfc4948a <Brian Hulette> Use Table.fromVectors in table-tests
f6551a3e <Brian Hulette> Add convenience function for constructing a Table from a list of Vectors
Krisztián Szűcs [Tue, 4 Dec 2018 02:52:59 +0000 (20:52 -0600)]
ARROW-3914: [C++/Python/Packaging] Docker-compose setup for Alpine linux
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes #3059 from kszucs/ARROW-3914 and squashes the following commits:
1a9224e87 <Krisztián Szűcs> rat
18a0f64fb <Krisztián Szűcs> crossbow test groups
00211ff6b <Krisztián Szűcs> missing conda_env_unix.yml
8f80fbbed <Krisztián Szűcs> lib vs lib64
698d5d5e4 <Krisztián Szűcs> don't build parquet in the python image
04851f2c6 <Krisztián Szűcs> don't install 3rdparty deps
584072d50 <Krisztián Szűcs> missing line break
5fe018dc1 <Krisztián Szűcs> install additional deps from apk
536f477c2 <Krisztián Szűcs> python 3.7 is not shipped with apk
8a29e1480 <Krisztián Szűcs> alpine dockerfiles for cpp and python implementations
Suvayu Ali [Tue, 4 Dec 2018 01:20:14 +0000 (19:20 -0600)] 
ARROW-3874: [C++] Add LLVM_DIR to find_package in FindLLVM.cmake
Identical to the cmake patch from [ARROW-3874](https://issues.apache.org/jira/browse/ARROW-3874).
Author: Suvayu Ali <suvayu@users.noreply.github.com>
Closes #3072 from suvayu/patch-1 and squashes the following commits:
f7e1eabda <Suvayu Ali> FindLLVM.cmake: Add LLVM_DIR to find_package
Antoine Pitrou [Tue, 4 Dec 2018 01:19:10 +0000 (19:19 -0600)]
ARROW-3906: [C++] Break out builder.cc into multiple compilation units
This improves readability and can also make incremental compilation faster.
Author: Antoine Pitrou <antoine@python.org>
Closes #3076 from pitrou/ARROW-3906-break-out-builder-cc and squashes the following commits:
3cf77e4d2 <Antoine Pitrou> ARROW-3906: Break out builder.cc into multiple compilation units
Wes McKinney [Tue, 4 Dec 2018 01:11:28 +0000 (19:11 -0600)]
PARQUET-1469: [C++] Fix data corruption bug in parquet::internal::DefinitionLevelsToBitmap that was triggered through random data
I also refactored some code to aid with debugging
Author: Wes McKinney <wesm+git@apache.org>
Closes #3080 from wesm/PARQUET-1469 and squashes the following commits:
5863562d0 <Wes McKinney> Fix bug in parquet::internal::DefinitionLevelsToBitmap that was triggered by the PRNG change. Some refactoring to assist with debugging
Uwe L. Korn [Tue, 4 Dec 2018 01:05:19 +0000 (19:05 -0600)]
ARROW-3884: [Python] Add LLVM6 to manylinux1 base image
I had to remove the `.dockerignore` file to make `quay.io` work again. I have a successful local build with Gandiva in the wheels but there are still some build systems we need to fix to activate Gandiva directly in the wheels.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes #3079 from xhochy/ARROW-3884 and squashes the following commits:
6697a92a5 <Uwe L. Korn> ARROW-3884: Add LLVM6 to manylinux1 base image
suquark [Mon, 3 Dec 2018 22:34:04 +0000 (14:34 -0800)] 
ARROW-2759: [Plasma] Export plasma notification socket
https://issues.apache.org/jira/browse/ARROW-2759
Author: suquark <suquark@gmail.com>
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes #3008 from suquark/plasma and squashes the following commits:
0bc89eba5 <Philipp Moritz> Update _plasma.pyx
a6d598159 <suquark> fix a bug. style
f037c319e <suquark> fix imports
19dae23d0 <suquark> use compat
942a62f84 <suquark> fix test
37b656013 <suquark> fix test
19f6b9852 <suquark> py2 compatibility
8e636ebaf <suquark> py2 compatibility
bbf07b94d <suquark> lint
6005b34a5 <suquark> lint
2719d0dc1 <suquark> lint
048f0c2ab <suquark> Export plasma notification socket.
Philipp Moritz [Mon, 3 Dec 2018 22:28:49 +0000 (14:28 -0800)]
ARROW-3199: [Plasma] File descriptor send and receive retries
An additional piece of eyes would be appreciated for this. It seems to solve our issue reported in the JIRA, but I'm not sure what the semantics of partial reads/writes is here (in particular, how are partial read/writes handled for ancillary data like file descriptors?).
found by cc @stephanie-wang
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes #2551 from pcmoritz/plasma-retries and squashes the following commits:
c7ca3b700 <Philipp Moritz> fix
91061adc5 <Philipp Moritz> linting
fda4dd27d <Philipp Moritz> move retry code
5dbabd79a <Philipp Moritz> fix travis
bc876dc29 <Philipp Moritz> fix compile errors
e1580fe81 <Philipp Moritz> plasma file descriptor send and receive retries
Romain Francois [Mon, 3 Dec 2018 21:21:23 +0000 (15:21 -0600)]
ARROW-3842: [R] RecordBatchStreamWriter api
This ended up being more than anticipated. In this PR, trying to get closer to the pyarrow api.
- Using active bindings instead of methods for things that are exposed as properties in python
- Using a factory function with the same name as the class name, e.g.. `arrow::RecordBatchFileWriter` gets a `RecordBatchFileWriter` function
This is fairly benign even though it's a big pr, it's mostly mechanical.
@javierluraschi this probably affects code that e.g. uses `$schema()` and now should use `$schema` etc ...
- `read_record_batch` now always need a `schema`
- `write_table` and `write_record_batch` are gone, and replaced with `write_arrow` which uses either the streaming format or the binary file format depending on the `stream` parameter:
- `arrow::ipc::RecordBatchStreamWriter` : streaming format
- `arrow::ipc::RecordBatchFileWriter` : binary file format
- `character` : file format
- `fs_path` : file format
- `raw` : streaming format
it's always possible to construct a RecordBatchStreamWriter or a RecordBatchFileWriter from an InputStream of your choice if these defaults don't work, but I believe they make sense.
Author: Romain Francois <romain@purrple.cat>
Closes #3043 from romainfrancois/ARROW-3842 and squashes the following commits:
01517e753 <Romain Francois> handle C_VISIBILITY at configure time.
e9889114d <Romain Francois> more converting public to active
d11c175c6 <Romain Francois> doc fixes
eed49297e <Romain Francois> doc fixes
ac0528d3c <Romain Francois> methods -> active
a7d73066b <Romain Francois> DataType$id and FixedWidthDataType$bit_width become active (as in pyarrow)
f62a22d31 <Romain Francois> Move more things to active rather than methods, to match pyarrow
590002079 <Romain Francois> + Table$columns(), RecordBatch$columns()
cc9669519 <Romain Francois> num_rows, num_columns, and schema become active rather than methods (as in pyarrow api)
77916c5d7 <Romain Francois> -doc directory
6f5db727c <Romain Francois> document read_arrow with read_table
f16f1267a <Romain Francois> update documentation for read_table
2dcca0345 <Romain Francois> write_arrow replaces write_table and handles Table, RecordBatch and data.frame, thanks to the unexported dispatcher to_arrow
1ea3b558f <Romain Francois> + method $batches() in RecordBatch(Stream|File)Reader
79192cb4b <Romain Francois> rename BufferOutputStream$Finish to $getvalue, as in pyarrow
f1af42e84 <Romain Francois> using lower case methods, like in pyarrow
740352d35 <Romain Francois> refactoring
Antoine Pitrou [Mon, 3 Dec 2018 13:50:13 +0000 (07:50 -0600)]
ARROW-3853: [C++] Implement string to timestamp cast
Author: Antoine Pitrou <antoine@python.org>
Closes #3044 from pitrou/ARROW-3853-cast-string-to-timestamp and squashes the following commits:
00bc36067 <Antoine Pitrou> ARROW-3853: Implement string to timestamp cast
Wes McKinney [Mon, 3 Dec 2018 03:44:36 +0000 (21:44 -0600)]
PARQUET-1467: [C++] Remove defunct ChunkedAllocator code
It does not seem that memory allocation on the dictionary encoding path requires something so elaborate right now
Author: Wes McKinney <wesm+git@apache.org>
Closes #3069 from wesm/PARQUET-1467 and squashes the following commits:
f37ed0756 <Wes McKinney> Remove defunct memory allocator code
Wes McKinney [Sun, 2 Dec 2018 23:05:30 +0000 (17:05 -0600)]
ARROW-3922: [C++] Micro-optimizations to BitUtil::GetBit
The results are fairly noisy so I couldn't really measure any improvement, but it seems to be at most 0.5-1%. I changed the NaiveBitmapReader to use `BitUtil::GetBit` so it is an apples-to-apples comparison. On my laptop (with CPU throttling disabled) the difference is within 1 standard deviation of the mean, so not statistically significant. Since the generated assembly is smaller, I would say it's a reasonable change
after
```
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------------------------------
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12255 us 12255 us 111 155.636MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12367 us 12367 us 111 154.228MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12243 us 12243 us 111 155.786MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12235 us 12236 us 111 155.885MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12426 us 12426 us 111 153.491MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12372 us 12372 us 111 154.164MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12283 us 12283 us 111 155.285MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12340 us 12340 us 111 154.567MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12389 us 12390 us 111 153.946MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12489 us 12489 us 111 152.722MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10_mean 12340 us 12340 us 111 154.571MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10_median 12353 us 12354 us 111 154.397MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10_stddev 85 us 85 us 111 1085.01kB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10717 us 10717 us 132 177.969MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10656 us 10657 us 132 178.982MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10835 us 10836 us 132 176.028MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10735 us 10735 us 132 177.672MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10700 us 10700 us 132 178.253MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 11481 us 11481 us 132 166.131MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10850 us 10850 us 132 175.797MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10591 us 10591 us 132 180.095MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10684 us 10684 us 132 178.523MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10705 us 10705 us 132 178.167MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10_mean 10795 us 10796 us 132 176.762MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10_median 10711 us 10711 us 132 178.068MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10_stddev 253 us 253 us 132 3.94562MB/s
```
previous
```
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------------------------------
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12349 us 12348 us 107 154.464MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12289 us 12288 us 107 155.225MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12368 us 12366 us 107 154.235MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12340 us 12339 us 107 154.578MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12446 us 12445 us 107 153.268MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12450 us 12449 us 107 153.211MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12377 us 12376 us 107 154.112MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12455 us 12454 us 107 153.157MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12431 us 12429 us 107 153.454MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10 12382 us 12381 us 107 154.052MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10_mean 12389 us 12388 us 107 153.976MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10_median 12380 us 12379 us 107 154.082MB/s
BM_NaiveBitmapReader/1000000/min_time:1.000/repeats:10_stddev 55 us 55 us 107 706.181kB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10898 us 10897 us 130 175.038MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10916 us 10915 us 130 174.739MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10923 us 10922 us 130 174.64MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 11002 us 11000 us 130 173.39MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10875 us 10874 us 130 175.403MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 11165 us 11163 us 130 170.859MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 11018 us 11016 us 130 173.141MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10948 us 10947 us 130 174.24MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 11079 us 11078 us 130 172.171MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10 10915 us 10914 us 130 174.766MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10_mean 10974 us 10973 us 130 173.839MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10_median 10935 us 10934 us 130 174.44MB/s
BM_BitmapReader/1000000/min_time:1.000/repeats:10_stddev 92 us 92 us 130 1.44155MB/s
```
Author: Wes McKinney <wesm+git@apache.org>
Closes #3067 from wesm/ARROW-3922 and squashes the following commits:
9c3fe8cf3 <Wes McKinney> clang-format
15f6a7b66 <Wes McKinney> Use c++11 syntax with gbenchmark
eb3a1a893 <Wes McKinney> A little bit less noisy benchmarks
4f56f2b10 <Wes McKinney> Micro-optimizations to BitUtil::GetBit
Sebastien Binet [Sun, 2 Dec 2018 17:35:53 +0000 (11:35 -0600)]
ARROW-3684: [Go] Add chunking ability to CSV reader
Author: Sebastien Binet <binet@cern.ch>
Closes #3019 from sbinet/issue-3684 and squashes the following commits:
64e88515e <Sebastien Binet> ARROW-3684: Add chunking ability to CSV reader
Wes McKinney [Sun, 2 Dec 2018 16:41:24 +0000 (10:41 -0600)]
ARROW-3890: [Python] Handle NumPy binary arrays with UTF-8 validation when converting to StringArray
I'm not sure if all compilers will be smart enough to do loop unswitching here. If it ends up being a bottleneck I suggest rewriting in a follow up patch.
The BinaryArray overflow issue (ChunkedArray is not being produced) is still present here. We will need to address that in ARROW-2970
This patch also includes symbol export macros particular to the arrow_python shared library. These are needed so that global data members in arrow.dll can be accessed in arrow_python.dll
Author: Wes McKinney <wesm+git@apache.org>
Closes #3063 from wesm/ARROW-3890 and squashes the following commits:
dac4995f9 <Wes McKinney> Windows needs arrow.lib in addition to arrow_python.lib now because of the new export flags
91dbea8bb <Wes McKinney> Add libarrow_python-specific visibility macros so that global data members from arrow.dll can be accessed correctly in arrow_python.dll
062c3836f <Wes McKinney> Clarify comment
cfbd30bbe <Wes McKinney> Handle case where user passes UTF-8 encoded numpy.str_ type array to pyarrow.array with type=pyarrow.string()
Kouhei Sutou [Sun, 2 Dec 2018 10:39:18 +0000 (19:39 +0900)]
ARROW-3912: [Plasma][GLib] Add support for creating and referring objects
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3056 from kou/glib-plasma-create-get and squashes the following commits:
e6bc59ff <Kouhei Sutou> Add missing status check for Disconnect()
623ce411 <Kouhei Sutou> Add missing return on error
d90d8960 <Kouhei Sutou> Add missing status check for Release()
1d5bf107 <Kouhei Sutou> Add missing include
7ac27db6 <Kouhei Sutou> Support old GObject Introspection
84df1849 <Kouhei Sutou> Support old GObject Introspection
26b2926e <Kouhei Sutou> Add support for creating and referring objects
Krisztián Szűcs [Sun, 2 Dec 2018 04:25:09 +0000 (22:25 -0600)]
ARROW-3925: [Python] Add autoconf to conda install instructions
autoconf is required by the jemalloc build
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Author: fjetter <fjetter@users.noreply.github.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes #2836 from fjetter/patch-1 and squashes the following commits:
9fba4ebfa <Wes McKinney> Create conda_env_unix.yml and use that in the Python documentation
dcbff0ffc <Krisztián Szűcs> don't escape within code-block
cfe3d3fab <Krisztián Szűcs> move autoconf to conda_env_cpp; update development documentation
7414fad1b <fjetter> Add glog to conda install instruction
768303ff3 <fjetter> Add autoconf to conda install instructions
Wes McKinney [Sun, 2 Dec 2018 04:20:53 +0000 (22:20 -0600)]
ARROW-3870: [C++] Add Peek to InputStream abstract interface
Default implementation returns a 0-length `arrow::util::string_view`
Author: Wes McKinney <wesm+git@apache.org>
Closes #3064 from wesm/ARROW-3870 and squashes the following commits:
6d9b411a0 <Wes McKinney> Fix another usage
b6173295a <Wes McKinney> Comparing std::string with arrow::util::string_view seems not to work with MSVC
30271b111 <Wes McKinney> Add InputStream::Peek base API and default implementation returning 0-length string
Wes McKinney [Sun, 2 Dec 2018 03:52:41 +0000 (21:52 -0600)]
ARROW-3898: [Example] parquet-arrow example has compilation errors
When compiling example, reader-writer.cc, it shows following compilation errors:
no member named 'cout' in namespace 'std'
PARQUET_THROW_NOT_OK(arrow::PrettyPrint(*array, 4, &std::cout));
in multiple places.
Fix:
Add the #include<iostream>
Test:
make passed
Author: Wes McKinney <wesm+git@apache.org>
Author: wangmiao1981 <wm624@hotmail.com>
Closes #3049 from wangmiao1981/example and squashes the following commits:
ecca664a2 <Wes McKinney> Build all examples in examples/parquet when PARQUET_BUILD_EXAMPLES=ON
64d51b214 <wangmiao1981> add include file
Kouhei Sutou [Sun, 2 Dec 2018 01:41:40 +0000 (19:41 -0600)]
ARROW-3924: [Packaging][Plasma] Add support for Plasma deb/rpm packages
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3068 from kou/linux-packages-plasma and squashes the following commits:
2cb7bad24 <Kouhei Sutou> Add support for Plasma deb/rpm packages
Robert Nishihara [Sat, 1 Dec 2018 08:26:51 +0000 (00:26 -0800)] 
ARROW-3920: [plasma] Fix reference counting in custom tensorflow plasma operator.
There is an issue here where `Release` was never being called in the plasma TF operator.
Note that I also changed the release delay in the plasma operator to 0.
Author: Robert Nishihara <robertnishihara@gmail.com>
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes #3061 from robertnishihara/extrareleaseinplasmaop and squashes the following commits:
c10956692 <Philipp Moritz> add include guards
f89d5df8c <Philipp Moritz> lint
4836342e0 <Philipp Moritz> unregister memory
e3b3864ef <Robert Nishihara> Linting
b948ce0f4 <Robert Nishihara> Add test.
75f2bd99c <Robert Nishihara> Remove logging statement.
f04a7d26d <Robert Nishihara> Fix
574c03532 <Robert Nishihara> Fix ndarray/tensor confusion in plasma op.
06985cd1c <Robert Nishihara> Have plasma op deserialize as numpy array.
a2a9c36b3 <Robert Nishihara> Add release call into wrapped_callback.
0db9154bd <Robert Nishihara> Change release delay to 0.
f4340946c <Robert Nishihara> Add Release call in plasma op.
Wes McKinney [Fri, 30 Nov 2018 21:48:27 +0000 (15:48 -0600)] 
[Website] Update website for 0.11.1 bugfix release (#3060)
Change-Id: I5eddfb4c51ffd3ea5ad7f98a10c98addbcf3c883
Chao Sun [Fri, 30 Nov 2018 18:39:44 +0000 (12:39 -0600)]
ARROW-3878: [Rust] Improve primitive types
Author: Chao Sun <sunchao@apache.org>
Closes #3031 from sunchao/types and squashes the following commits:
68ecfef96 <Chao Sun> Address comments
5b3f3720e <Chao Sun> Fix benchmark
c977ae86c <Chao Sun> Revert travis change
0f711a5b0 <Chao Sun> Revert numeric_from_nonnull
c280e83f2 <Chao Sun> Change one more place to use array shortname
a2943ba6c <Chao Sun> Rename T::T to T::Native
d1a69fd83 <Chao Sun> ARROW-3878: Improve primitive types
Antoine Pitrou [Fri, 30 Nov 2018 18:29:17 +0000 (12:29 -0600)]
ARROW-3893: [C++] Improve adaptive int builder performance
The strategy is two-pronged:
- scalar append to int builder appends to a small uint64_t scratch space, deferring width detection and conversion until the scratch space is full
- bulk int width detection is much improved
Author: Antoine Pitrou <antoine@python.org>
Closes #3040 from pitrou/ARROW-3893-adaptive-int-builder-perf and squashes the following commits:
61a101b12 <Antoine Pitrou> ARROW-3893: Improve adaptive int builder performance
Bradley Skaggs [Fri, 30 Nov 2018 15:21:38 +0000 (09:21 -0600)] 
ARROW-3202: [C++] Fix compilation on Alpine Linux by using ARROW_WITH_BACKTRACE define
Should fix #2818.
Author: Bradley Skaggs <bskaggs@acm.org>
Closes #3041 from bskaggs/master and squashes the following commits:
680b9644f <Bradley Skaggs> change include guard to ARROW_WITH_BACKTRACE
Kouhei Sutou [Fri, 30 Nov 2018 14:35:25 +0000 (08:35 -0600)]
ARROW-3886: [C++] Add support for decompressed buffer size check for Snappy
Author: Kouhei Sutou <kou@clear-code.com>
Closes #3057 from kou/cpp-decompress-check and squashes the following commits:
463f6467a <Kouhei Sutou> Add support for decompressed buffer size check for Snappy
Yosuke Shiro [Fri, 30 Nov 2018 01:34:07 +0000 (10:34 +0900)]
ARROW-3905: [Ruby] Add StructDataType#[]
Author: Yosuke Shiro <yosuke.shiro615@gmail.com>
Closes #3053 from shiro615/ruby-add-convenient-method-to-struct-data-type and squashes the following commits:
fdaf7b8b <Yosuke Shiro> Rename SchemaContainable to FieldContainable
9d24d267 <Yosuke Shiro> Use alias method for #
aef063fc <Yosuke Shiro> Add StructDataType#
François Saint-Jacques [Thu, 29 Nov 2018 20:51:13 +0000 (14:51 -0600)]
ARROW-3740: [C++] Builder should not downsize
Downsizing is a tricky business, it is safer to invoke Reset() then Reserve().
Author: François Saint-Jacques <fsaintjacques@gmail.com>
Closes #3055 from fsaintjacques/ARROW-3740-builder-resize-shrink and squashes the following commits:
4af5f1661 <François Saint-Jacques> Arrow-3740: Builder should not downsize
projects / arrow.git / log
