How to append a String or bool to UnionBuilder?

[surister]

  let mut builder = UnionBuilder::new_dense();
  builder.append::<Int32Type>("a", 1).unwrap();
  builder.append::<Float64Type>("b", 3.0).unwrap();
  builder.append::<Int32Type>("a", 4).unwrap();
  builder.append::<?>("c", "some_string".to_string()).unwrap();
                   ^

What DataType do I specify, couldn't find anything suitable that implements ArrowPrimitiveType, or what is the right idiom?

[tustvold]

At the moment UnionBuilder only supports primitive types, you will need to construct the union manually to use more complex types.

[surister]

I hear you, in my use case I'm transforming mongodb documents (rows) to arrow,
something along the lines of:

let mut builder = UnionBuilder::new_dense(); 
let documents: Vec<Document> = get_documents()

for doc in documents {
    for value in doc {
          match value {
              Bson::String(s) => builder.append<Utf8Type>("a", value),
              Bson::I32(s) => builder.append<Int32Type>("b", value),
              _ => ...
          }
    }
}

so I don't see a way of avoiding the bad performance.

[tustvold]

You could do something along the lines of

let type_ids = vec![];
let offsets = vec![];
let mut integers = Int32Builder::new();
let mut strings = StringArrayBuilder::new();
documents.iter().flatten().for_each(|value| match value {
    Bson::I32(s) =>{
        type_ids.push(0);
        offsets.append(integers.len() as _);
        integers.append_value(s);
    }
    Bson::String(s) =>{
        type_ids.push(1);
        offsets.append(strings.len() as _);
        strings.append_value(s);
    },
});

assert!(integers.len() < i32::MAX as _);
assert!(strings.len() < i32::MAX as _);
let array = UnionArray::new(fields, type_ids.into(), Some(offsets), vec![Arc::new(integers.finish()) as _, Arc::new(strings.finish()) as _])

This would avoid some of the overheads

[surister]

This would put everything into the same UnionArray, I'm aiming to have one UnionArray per Bson field, the use case is that I don't really know how many fields I get in the bson and they might have different datatypes for the same field, e.g:

let docs = vec![
  doc! { "i32": 42, "string": 12345, "some_field": vec![1, 2, 3] },
  doc! { "i32": 24, "string": "world", "some_field": vec![4, 5] },
  doc! { "string": vec!["hello"; "world"]], "some_field": "hello" },
                                                           ^^^^^
  doc! { "i32_list": [1, 2, 3] }
];

The idea of using UnionArrays per field is to later create a RecordBatch and start cleaning things up, like exploding some_field to some_field_arr_str and some_field_str until I have a proper table with only one datatype per field to be later put into a sane database, pretty much a NoSql to Sql kind of thing.

With your approach I'd have to initialize an ArrayBuilder per possible type per field right?

[tustvold]

With your approach I'd have to initialize an ArrayBuilder per possible type per field right?

Yes

I'd personally recommend cleaning up the data in the row-oriented representation, and then converting the cleaned data to arrow. UnionArray won't give you much performance uplift, if any, as it breaks vectorised execution. They're also a pain to work with, and not very well supported by arrow-rs or the ecosystem at large

[surister]

cleaning up the data in the row-oriented representation

I kind of thought about this but wasn't unsure as my knowledge of Arrow is still sparse, in my current implementation I transform the rows into data:

enum CValue {
    String(str),
    I32(i32),
    I64(i64),
    ...
}

let data: HashMap<String, Vec<CValue>

And then I normalize the hashmap and end up with clean vectors of only one CValue primitive type, maybe its at this point where I should make it to arrow as it'd be fairly easy and later do all the filters/casts.