feat: null support for plain encoding

[rok]

Subtask of #16.

Create proposal for nullability implementation with a mini batch binary mask.

[westonpace]

@wjones127 to answer your question regarding structs & nullability. I agree that we should not try and store nulls for structs. A null struct should become a struct of nulls. This matches our current I/O most closely and so it is the easier change. If we have a need for it we can add that in the future.

[dnsco]

Weighing in as a user, being able to write queries in duckdb as "where some_struct is not null" is really nice. It is different than "where some_struct.some_field is not null".

I have no idea about the implementation implications on your side, but want to throw out that the former is really really nice.

[mkleinbort]

Very keen on this feature, we'd like to move to lance but this is a deal breaker for our data infrastructure

[westonpace]

Very keen on this feature, we'd like to move to lance but this is a deal breaker for our data infrastructure

It's in progress. Right now the plan is for it to be delivered as part of #1929 . I've been focused on lancedb/lancedb#926 recently but should be finishing up this week and getting some time to spend on Lance v2 again.