Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add integration testing reader for Column #325

Merged
merged 19 commits into from
Nov 29, 2023
Merged

feat: Add integration testing reader for Column #325

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
bc8f990
first stabs at buffer converters
paleolimbot Nov 27, 2023
19f684c
bitmap setter
paleolimbot Nov 27, 2023
e710119
building
paleolimbot Nov 27, 2023
893d5ba
more cases
paleolimbot Nov 28, 2023
88f9931
one basic column test
paleolimbot Nov 28, 2023
31100a5
move length/offset validation to array view
paleolimbot Nov 28, 2023
7953e9b
tack on roundtrip for column json
paleolimbot Nov 28, 2023
8964f2c
with integer tests
paleolimbot Nov 28, 2023
fd07808
better error for signed integer in uint64 column
paleolimbot Nov 28, 2023
00dea85
fix string/binary
paleolimbot Nov 28, 2023
46e7c5f
fixed size binary
paleolimbot Nov 28, 2023
07a1fa5
floatingpoint
paleolimbot Nov 28, 2023
1122875
review comments
paleolimbot Nov 29, 2023
a96f9bd
separate fixed size and non-fixed size
paleolimbot Nov 29, 2023
0b79088
use json::exception
paleolimbot Nov 29, 2023
2e8b76f
fix python
paleolimbot Nov 29, 2023
d96bc28
remove testing file
paleolimbot Nov 29, 2023
8350dd0
ignore testing file
paleolimbot Nov 29, 2023
e1afd57
test union roundtrips
paleolimbot Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion extensions/nanoarrow_device/src/nanoarrow/nanoarrow_device.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,7 +408,7 @@ static ArrowErrorCode ArrowDeviceArrayViewCopyInternal(struct ArrowDevice* devic
dst->offset = src->offset;
dst->null_count = src->null_count;

for (int i = 0; i < 3; i++) {
for (int i = 0; i < NANOARROW_MAX_FIXED_BUFFERS; i++) {
if (src->layout.buffer_type[i] == NANOARROW_BUFFER_TYPE_NONE) {
break;
}
Expand Down
4 changes: 2 additions & 2 deletions extensions/nanoarrow_ipc/src/nanoarrow/nanoarrow_ipc_decoder.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1155,7 +1155,7 @@ static void ArrowIpcDecoderInitFields(struct ArrowIpcField* fields,
field->array = array;
field->buffer_offset = *n_buffers;

for (int i = 0; i < 3; i++) {
for (int i = 0; i < NANOARROW_MAX_FIXED_BUFFERS; i++) {
*n_buffers += array_view->layout.buffer_type[i] != NANOARROW_BUFFER_TYPE_NONE;
}

Expand Down Expand Up @@ -1524,7 +1524,7 @@ static int ArrowIpcDecoderWalkSetArrayView(struct ArrowIpcArraySetter* setter,
array_view->null_count = ns(FieldNode_null_count(field));
setter->field_i += 1;

for (int64_t i = 0; i < 3; i++) {
for (int i = 0; i < NANOARROW_MAX_FIXED_BUFFERS; i++) {
if (array_view->layout.buffer_type[i] == NANOARROW_BUFFER_TYPE_NONE) {
break;
}
Expand Down
5 changes: 5 additions & 0 deletions python/bootstrap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ def generate_nanoarrow_pxd(self, file_in, file_out):
# Strip comments
content = self.re_comment.sub("", content)

# Replace NANOARROW_MAX_FIXED_BUFFERS with its value
content = self.re_max_buffers.sub("3", content)

# Find types and function definitions
types = self._find_types(content)
func_defs = self._find_func_defs(content)
Expand All @@ -59,6 +62,7 @@ def generate_nanoarrow_pxd(self, file_in, file_out):
output.write(b"\n")
output.write(b" ctypedef int ArrowErrorCode\n")
output.write(b" cdef int NANOARROW_OK\n")
output.write(b" cdef int NANOARROW_MAX_FIXED_BUFFERS\n")
output.write(b"\n")

for type in types_cython:
Expand All @@ -71,6 +75,7 @@ def generate_nanoarrow_pxd(self, file_in, file_out):

def _define_regexes(self):
self.re_comment = re.compile(r"\s*//[^\n]*")
self.re_max_buffers = re.compile(r"NANOARROW_MAX_FIXED_BUFFERS")
self.re_type = re.compile(
r"(?P<type>struct|union|enum) (?P<name>Arrow[^ ]+) {(?P<body>[^}]*)}"
)
Expand Down
Loading
Loading